Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Reinforcement learning with verifiable rewards (RLVR) is credited with gains beyond the domain its rewards check, such as higher MMLU accuracy after training on mathematics. Such accuracies are usually read out of the model’s text by an answer extractor. A model that learns to write its answer where the extractor looks can therefore score higher without answering better.

强化学习与可验证奖励（RLVR）被认为在奖励检查领域之外也能带来收益，例如在数学训练后提高 MMLU 准确率。此类准确率通常由答案提取器从模型文本中读取。因此，一个学会将答案写在提取器查找位置的模型，无需提升回答质量即可获得更高分数。

We measure how much of the gain reported by extractors copied from public code arises this way. We generate outputs from a base model and its RL model and score them twice: with those extractors, and with one extractor of our own that reads a stated answer whether it is in the answer box \boxed{} or in the text.

我们衡量了从公开代码中复制的提取器所报告的收益中有多少源于这种方式。我们生成了基础模型及其 RL 模型的输出，并进行了两次评分：一次使用这些提取器，另一次使用我们自己的提取器，后者无论答案是在 \boxed{} 框内还是在文本中，都能读取到陈述的答案。

On four released 7B SimpleRL base-RL pairs, 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours: it is a gain in answer format, not in answers. The base models already state most of their answers in text, and RL moves them into the box, where these extractors look first.

在四个已发布的 7B SimpleRL 基础-RL 模型对上，这些提取器报告的域外多项选择收益中，有 82% 在使用我们的提取器后消失了：这属于答案格式的收益，而非答案本身的提升。基础模型已经在文本中陈述了大部分答案，而 RL 将它们移到了这些提取器优先查找的框内。

Independent LLM readers, in place of our extractor, put the figure at 77%. With a prompt that itself asks for the box, our extractor puts it at 79%. On a 32B pair trained with another recipe, almost none of the reported gain is format gain: its base model boxes its answers when asked, and its RL model rarely does when not.

使用独立的 LLM 阅读器代替我们的提取器，该数字为 77%。在使用要求使用框的提示词时，我们的提取器得出的数字为 79%。在采用另一种方案训练的 32B 模型对上，几乎没有报告的收益属于格式收益：其基础模型在被要求时会将答案放入框中，而其 RL 模型在未被要求时很少这样做。

A reported gain is partly a property of the extractor that read it, and re-scoring stored outputs with a second extractor shows how much, without re-running RL.

报告的收益部分取决于读取它的提取器，而使用第二个提取器对存储的输出进行重新评分，无需重新运行 RL 即可显示出具体程度。