Can LLMs Take Retrieved Information with a Grain of Salt?

大语言模型能否对检索到的信息保持审慎态度？

Abstract: Large language models have demonstrated impressive retrieval-augmented capabilities. However, a crucial area remains underexplored: their ability to appropriately adapt responses to the certainty of the retrieved information. It is a limitation with real consequences in high-stakes domains like medicine and finance.

摘要： 大语言模型（LLMs）已经展现出令人印象深刻的检索增强能力。然而，一个关键领域仍未得到充分探索：模型根据检索信息的确定性来适当地调整回答的能力。在医学和金融等高风险领域，这一局限性会带来实际的后果。

We evaluate eight LLMs on their context-certainty obedience, measuring how well they adjust responses to match expressed context certainty. Our analysis reveals systematic limitations: LLMs struggle to recall prior knowledge after observing an uncertain context, misinterpret expressed certainties, and overtrust complex contexts.

我们评估了八种大语言模型在“上下文确定性服从度”方面的表现，衡量它们在多大程度上能根据所表达的上下文确定性来调整回答。我们的分析揭示了系统性的局限性：大语言模型在观察到不确定的上下文后，难以调用先验知识，容易误解所表达的确定性，并且会过度信任复杂的上下文。

To address these, we propose an interaction strategy combining prior reminders, certainty recalibration, and context simplification. This approach reduces obedience errors by 25% on average, without modifying model weights, demonstrating the efficacy of interaction design in enhancing LLM reliability.

为了解决这些问题，我们提出了一种结合了先验提醒、确定性重新校准和上下文简化的交互策略。该方法在不修改模型权重的情况下，平均将服从性错误降低了 25%，证明了交互设计在提升大语言模型可靠性方面的有效性。

Our contributions include a principled evaluation metric, empirical insights into LLMs’ uncertainty handling, and a portable strategy to improve context-certainty obedience across diverse LLMs.

我们的贡献包括一套原则性的评估指标、关于大语言模型处理不确定性的实证见解，以及一种可移植的策略，旨在改善不同大语言模型在上下文确定性方面的服从表现。