Improving LLMs via Validator-to-Generator Alignment

通过验证器与生成器对齐改进大语言模型

Abstract: Large language models are inconsistent: varying prompts or including unrelated information can lead to unexpected changes in model outputs. The generator-validator (G-V) gap is one manifestation of this phenomenon, where LLMs generate responses that they then deem as invalid if re-queried to validate them.

摘要： 大语言模型（LLM）存在不一致性：改变提示词或包含无关信息可能导致模型输出发生意外变化。“生成器-验证器”（G-V）差距是这一现象的表现之一，即大语言模型生成的回答，在被重新询问进行验证时，却会被模型自身判定为无效。

In this work, we introduce a new formulation of G-V consistency that involves a principled correction for utterance frequency. Specifically, generators often assign low likelihood to valid strings simply because those strings are a priori unlikely, which makes naive notions of G-V consistency unworkable.

在这项工作中，我们引入了一种新的 G-V 一致性表述，其中包含针对话语频率的原则性校正。具体而言，生成器往往会给有效的字符串分配较低的概率，仅仅是因为这些字符串在先验上是不太可能的，这使得朴素的 G-V 一致性概念无法实施。

We show that under a natural model of rational agents answering questions with multiple answers, consistency of the validator with a frequency-corrected generator score emerges naturally. Our method, \emph{\FCPAname} (\FCPA), is a training objective implementing frequency-corrected G-V consistency for real-world LLMs.

我们证明，在理性智能体回答多答案问题的自然模型下，验证器与经过频率校正的生成器分数之间的一致性会自然产生。我们的方法 \emph{\FCPAname} (\FCPA) 是一种训练目标，旨在为现实世界的大语言模型实现经过频率校正的 G-V 一致性。

Our experimental results show that training with \FCPA{} substantially improves both G-V consistency and generator performance over prior methods, with gains of up to $+27$pp in Pearson correlation on IFEval and HumanEval, while preserving validator quality across all evaluated tasks.

实验结果表明，使用 \FCPA 进行训练在 G-V 一致性和生成器性能方面均显著优于现有方法，在 IFEval 和 HumanEval 数据集上的皮尔逊相关系数（Pearson correlation）提升高达 $+27$ 个百分点，同时在所有评估任务中均保持了验证器的质量。