Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

探究推理性能的起源：强化学习（RL）与监督微调（SFT）模型在数学问题解决中的表征质量对比

Abstract: Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear.

摘要： 越来越多的研究表明，通过强化学习（RL）训练的大型推理模型在数学推理任务上的表现优于监督微调（SFT）模型；然而，这种优势背后的机制基础尚不明确。

We therefore ask, what internal representational differences enable RL models’ superior performance? Our work presents two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations.

因此，我们提出疑问：什么样的内部表征差异使得 RL 模型具备更优越的性能？我们的研究提出了两条趋同的证据：首先，基于各层隐藏状态训练的线性探测（linear probes）显示，与 SFT 模型相比，RL 模型在预测答案正确性方面往往能达到更高的准确率，这表明其表征具有更好的线性可分性和结构化特征。

Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers. Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems.

其次，均值消融研究表明，RL 模型发展出了一种分层架构，其中深层变得愈发关键，而 SFT 模型则将重要性均匀地分布在各层中。综上所述，这些发现证明了 RL 训练从根本上重构了模型表征和处理推理问题的方式。

Finally, we analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation. While we observe higher variability in some RL-tuned models than in their SFT counterparts, we see strong consistency in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone.

最后，我们分析了在不同问题下重复采样时的 Token 数量变异性，以评估自适应计算分配。虽然我们观察到一些 RL 调优模型比 SFT 模型表现出更高的变异性，但在其他模型中我们看到了很强的一致性，这表明 Token 分配可能更多地取决于整体训练流程，而非仅仅取决于 RL 与 SFT 的区别。

We believe this token-allocation variability reveals the spread of plausible on-policy reasoning, highlighting which models exhibit stable policies versus those that are under-determined, potentially non-identifiable solution behaviour.

我们认为，这种 Token 分配的变异性揭示了合理的策略内（on-policy）推理的分布范围，突显了哪些模型表现出稳定的策略，而哪些模型则表现出欠定（under-determined）且可能无法识别的求解行为。