Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

同一问题，不同答案：超越准确率评估大语言模型的可靠性

Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways. 大语言模型（LLMs）在基准测试中通常能取得很高的准确率，但当同一个问题以不同但等价的方式表述时，模型应用这些知识的可靠性仍不明确。

In this work, we study how model answers change under meaning-preserving paraphrases across factual question answering and mathematical reasoning tasks. 在这项工作中，我们研究了在事实问答和数学推理任务中，模型答案如何随着保持语义的改写而发生变化。

Across four benchmarks and 13 models, we find that model outputs frequently depend on the exact wording of the prompt. 通过对四个基准测试和 13 个模型的研究，我们发现模型输出往往取决于提示词的具体措辞。

While overall accuracy typically changes only modestly across paraphrases, instance-level behavior is far less stable: for many questions, models alternate between correct and incorrect answers depending on phrasing, with mismatch rates reaching more than 23%. 虽然总体准确率在不同改写版本间通常变化不大，但实例层面的表现却远不稳定：对于许多问题，模型会根据措辞在正确和错误答案之间切换，不匹配率高达 23% 以上。

Conditioning on questions that are answered correctly in their original form reveals even larger failures measured by answer flip rates, showing that single-prompt correctness is often a poor indicator of reliability. 以原始形式回答正确的问题为条件进行分析，通过答案翻转率衡量，可以发现更大的失败，这表明单一提示词的正确性往往不能很好地反映模型的可靠性。

At the same time, we find that models often produce a correct answer for at least one paraphrase of a question, suggesting that the underlying knowledge is present but inconsistently retrieved. 与此同时，我们发现模型通常至少能对问题的一种改写版本给出正确答案，这表明模型具备底层知识，但检索过程并不稳定。

Building on this observation, we show that a simple self-paraphrasing strategy can partially recover this latent knowledge and improve performance at inference time. 基于这一观察，我们证明了一种简单的自我改写策略可以在推理阶段部分恢复这些潜在知识，并提升模型性能。

Together, these findings suggest that standard accuracy metrics can mask substantial instability, and that evaluating consistency across equivalent inputs provides a clearer picture of LLM reliability. 总之，这些发现表明标准的准确率指标可能会掩盖实质性的不稳定性，而评估等价输入之间的一致性，能更清晰地反映大语言模型的可靠性。