A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

基于共识的大语言模型相对偏好评估框架

Abstract: Traditional benchmarks for LLMs primarily rely on static datasets and objective scoring metrics, which often fail to capture differences in response quality when multiple answers are acceptable. In such settings, correctness alone is insufficient to distinguish between responses that vary in clarity, completeness, and usefulness.

摘要： 传统的大语言模型（LLM）基准测试主要依赖于静态数据集和客观评分指标，当存在多个可接受的答案时，这些方法往往无法捕捉到响应质量的差异。在这种情况下，仅凭正确性不足以区分在清晰度、完整性和实用性方面存在差异的响应。

This paper introduces a consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness. Instead of evaluating outputs against a fixed ground truth, we assess how a panel of diverse LLMs ranks anonymized candidate responses to the same prompt. This approach treats aggregate inter-model agreement as a proxy for perceived response quality under blind conditions.

本文引入了一种基于共识的评估框架，旨在衡量模型生成响应之间的相对偏好，而非绝对正确性。我们不再根据固定的标准答案来评估输出，而是评估一组多样化的大语言模型如何对同一提示词的匿名候选响应进行排序。这种方法将模型间的总体共识视为盲测条件下感知响应质量的代理指标。

We conduct a controlled study using five state-of-the-art LLMs across multiple domains, including programming, general knowledge, safety, logical reasoning, and mathematics. Each model generates responses and independently ranks peer outputs through a structured voting process. Scores are aggregated into a Relative Intelligence Index (RII), representing how frequently a model’s responses are preferred by other models.

我们使用五个最先进的大语言模型在编程、常识、安全、逻辑推理和数学等多个领域进行了对照研究。每个模型通过结构化的投票过程生成响应并独立地对同行的输出进行排名。分数被汇总为“相对智能指数”（Relative Intelligence Index, RII），代表一个模型的响应被其他模型偏好的频率。

Our findings reveal consistent preference patterns across domains, with certain models more frequently ranked highly by their peers. However, we emphasize that these results reflect inter-model preference alignment rather than objective correctness or human judgment. This framework provides a scalable, model-driven method for comparative evaluation, offering an alternative perspective on response quality in scenarios where multiple valid answers exist.

研究结果显示，各领域中存在一致的偏好模式，某些模型更频繁地被同行评为高分。然而，我们强调这些结果反映的是模型间的偏好对齐，而非客观正确性或人类判断。该框架为比较评估提供了一种可扩展的、模型驱动的方法，为存在多个有效答案的场景下的响应质量评估提供了新的视角。

While not directly aligned with human evaluation, prior work suggests that aggregated model preferences can partially correlate with human judgments, motivating this as a proxy signal.

虽然这并不直接等同于人类评估，但先前的研究表明，汇总后的模型偏好可以与人类判断部分相关，这使其成为一种有效的代理信号。