JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation

JudgeArena：用于可复现 LLM-Judge 评估的统一框架

LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol. This fragmentation makes it difficult to study how design choices—the benchmark, the judge model, the prompt, the inference backend—affect the conclusions we draw about model quality.

“LLM 作为裁判”（LLM-as-a-judge）的评估方式已成为语言模型排名的主流范式，但目前的生态系统仍然十分碎片化：大多数基准测试都自带代码库、硬编码特定的闭源模型作为裁判，且仅支持单一的评估协议。这种碎片化使得研究设计选择（如基准测试、裁判模型、提示词、推理后端）如何影响模型质量评估结论变得十分困难。

We introduce JudgeArena, an open-source framework that unifies major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility. It enables systematic studies of judge choices, as any model accessible via vLLM, this http URL, or OpenRouter can serve as both candidate and judge.

我们推出了 JudgeArena，这是一个开源框架，它将主要的 LLM 裁判基准测试（AlpacaEval、Arena-Hard、MT-Bench 和 m-Arena-Hard）统一在一个接口下，支持可替换的裁判模型，并提供全面的元数据记录，以提高报告的透明度和可复现性。它支持对裁判选择进行系统性研究，因为任何可通过 vLLM、HTTP URL 或 OpenRouter 访问的模型都可以同时作为候选模型和裁判模型。

Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models. Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.

此外，JudgeArena 随附了针对开源模型的调优裁判配置，这些配置在英语和多语言环境下的人类偏好数据集上经过验证，其表现可媲美甚至超越闭源模型裁判，从而减少了对不透明闭源模型的依赖。最后，通过将现有的人类标注与目标模型的 LLM 裁判评估相结合，JudgeArena 能够以高精度模拟 LMArena Elo 分数，为大规模人类标注活动提供了一种实用、开放且低成本的替代方案。