Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

Computer Science > Computation and Language arXiv:2610.19095 (cs) [Submitted on 13 Oct 2026] Title: Similar Models, Broken Odds: Where Luce’s Axiom Fails in Chatbot Arena Authors: Soovadeep Bakshi, Apurba Bose, Aishwarya Bose, Sandip Nandi.

计算机科学 > 计算与语言 arXiv:2610.19095 (cs) [提交于 2026 年 10 月 13 日] 标题：相似的模型，破碎的几率：卢斯公理在聊天机器人竞技场中的失效之处 作者：Soovadeep Bakshi, Apurba Bose, Aishwarya Bose, Sandip Nandi。

Abstract: Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce’s choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply.

摘要：聊天机器人竞技场（Chatbot Arena）使用 Bradley-Terry 模型对语言模型进行排名，该模型为每个模型赋予一个分数，因此假设了卢斯选择公理（Luce’s choice axiom）。选择理论预测了该公理失效的情形：近乎复制的模型在被区分时，其结果比分数所暗示的更为果断。

On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments, prompt subsets and topics, and timing checks.

在 180 万次竞技场投票中，同一模型不同版本之间的投票果断程度是 Bradley-Terry 模型预测的 1.37 倍，同一代不同规模模型之间为 1.26 倍，同一实验室不同模型之间为 1.12 倍；一年后的版本发布得出了相似的乘数，且该效应在不同的平局处理方式、提示词子集与主题以及时间检查中依然存在。

How alike two models’ answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established.

两个模型回答的措辞和格式的相似程度与该效应相关。一个包含四到六个参数的修正模型在预测留出投票（held-out votes）方面的表现优于 Bradley-Terry 模型以及拥有数百个参数的模型，这主要是通过对平局进行建模实现的；在预先设定的测试中，将其冻结并应用于新版本时，其增益集中在平局处理上，而更好的正面交锋几率并未建立。

The leaderboard order barely moves. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a prompt the shift is smaller and clear for two of four judges.

排行榜的顺序几乎没有变动。在使用语言模型裁判和三选一菜单的情况下，当跨提示词汇总选择时，近乎复制的模型主要从其孪生模型处获取投票（对数几率偏移 +0.10，95% 置信区间 0.05 至 0.16）；在单个提示词内，这种偏移较小，且在四名裁判中的两名身上表现明显。