Improved Confidence Estimates for Black-Box Large Language Models

黑盒大语言模型的置信度估计改进

Abstract: Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data.

摘要： 不确定性量化（UQ）对于大语言模型（LLM）的安全部署至关重要。现有的方法，从口头置信度（verbalized confidence）到需要多次生成的方案，通常都是零样本（zero-shot）的，且无需标注数据即可生成量化不确定性的分数。

Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores.

然而，在实际应用中，人们在部署前必须始终在目标数据集上评估其性能。在这项工作中，我们证明了通过利用这些数据集，我们的方法能够持续优于现有的评分方法。

Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.

具体而言，我们构建了简单的分类器，通过将这些分数以及相似查询的正确性作为特征，来预测 LLM 响应的正确性。我们的方法产生的计算开销极小，使其成为现实应用中增强 LLM 不确定性量化的一种廉价且直接的方案。