Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Chain-of-Models：用于偏差鲁棒性大模型评估的跨模型审计
Abstract: LLMs increasingly serve as automated judges, but their judgments remain vulnerable to cognitive biases. Existing mitigations mostly rely on prompt-driven debiasing, which is brittle across bias types, or human evaluation, which does not scale. We study \emph{Chain-of-Models} (CoM), an automated audit pipeline in which a second model inspects the first model’s reasoning trace before producing the final judgment.
摘要： 大语言模型（LLM）正越来越多地充当自动化评估员，但其判断结果仍易受到认知偏差的影响。现有的缓解措施大多依赖于基于提示词（Prompt）的去偏技术，这种方法在面对不同类型的偏差时表现脆弱；或者依赖于人工评估，但人工评估难以扩展。我们研究了“模型链”（Chain-of-Models, CoM），这是一种自动化审计流程，其中第二个模型在生成最终判断之前，会先检查第一个模型的推理轨迹。
The key design question is whether the auditor should be the same model, a same-family model, or a different-family model. Across 9 models from 6 families, 4 cognitive biases, and 4 factual datasets, we find that auditor identity matters in two ways. First, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases, yet is a weak auditor for Qwen2.5-72B’s biased traces.
核心设计问题在于：审计模型应该是同一个模型、同一系列模型，还是不同系列的模型？通过对来自 6 个系列的 9 个模型、4 种认知偏差和 4 个事实数据集的测试，我们发现审计模型的身份在两个方面至关重要。首先，模型独立的抗偏差能力并不能预测其审计有效性：Kimi-K2.5 在多种偏差上表现出最强的独立抗性，但在审计 Qwen2.5-72B 的偏差推理轨迹时，却是一个表现较弱的审计员。
Second, the best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction, while GLM-5 is strongest on sycophancy. We operationalize these findings with a per-bias auditor selection rule that, given the bias type, scores candidates along functional diversity, per-bias standalone resistance, and calibrated audit effectiveness.
其次，最佳审计员的选择具有偏差特异性：GPT-4o 在从众效应（bandwagon）、权威偏差（authority）和干扰偏差（distraction）方面表现最强，而 GLM-5 在迎合偏差（sycophancy）方面表现最强。我们将这些发现转化为一种针对特定偏差的审计员选择规则：根据偏差类型，从功能多样性、特定偏差的独立抗性以及校准后的审计有效性三个维度对候选模型进行评分。
Under a calibration/test split, the selector reaches the highest accuracy across the four biased slices ($0.884$ vs.\ $0.824$ for the strongest single fixed auditor and $0.805$ for the no-audit baseline). We release data, configurations, and an LLM-agent skill at this https URL.
在校准/测试集划分下，该选择器在四个偏差切片上达到了最高准确率（0.884，相比之下，最强的单一固定审计员为 0.824，无审计基准为 0.805）。我们已在指定网址发布了相关数据、配置以及一个 LLM 智能体技能。