Codifying the Judge: Scalable Evaluation via Program Distillation

编纂评判者：通过程序蒸馏实现可扩展评估

Abstract: LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions — limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs.

摘要： “大模型作为评判者”（LLM-as-a-judge）已成为自动化评估的标准，但它存在成本高昂、延迟显著以及决策过程不透明等问题，这些局限性削弱了其可扩展性和可靠性。我们提出了一种简单且高效的替代方案：程序蒸馏（program distillation）。我们不再在评估时调用大模型，而是将其决策逻辑蒸馏为一组程序，直接对候选结果进行评分。这些程序化评判者提供了透明度，易于检查或编辑，并消除了每个样本的 API 调用成本。

Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier.

基于这一理念，我们推出了 PAJAMA 系统。该系统能够合成程序作为评判者，将它们的决策汇总为联合结论，并引入了一种回退机制，将低置信度的案例选择性地升级给大模型处理。在五个数据集和四个模型系列上的实验表明，程序化评判者可以达到与 13B 参数规模的大模型评判者相当的性能。当使用程序输出作为路由信号时，PAJAMA 不仅提高了准确率和吞吐量，还推动了帕累托前沿（Pareto frontier）的进展。

Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs’ verdicts outperforms one trained on a proprietary LLM’s labels at two orders of magnitude lower API cost.

除了评估之外，程序化评判者还能产生廉价且有效的奖励信号：在 RewardBench 测试中，一个通过程序决策蒸馏出的奖励模型，其表现优于使用专有大模型标签训练的模型，且 API 调用成本降低了两个数量级。