Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data

Computer Science > Machine Learning arXiv:2610.18878 (cs) [Submitted on 1 Oct 2026] Title: Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data Authors: Eric O. Korman.

计算机科学 > 机器学习 arXiv:2610.18878 (cs) [提交于 2026 年 10 月 1 日] 标题：用于高效后训练和验证的逆向推理模型：基于仅含答案的数据，作者：Eric O. Korman。

Abstract: Post-training reasoning LLMs often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt-answer tuples and lacks the groundtruth chain-of-thought (CoT) sequence. On the other hand, reinforcement learning (with verifiable rewards, for example) can operate without CoT data but it is much more compute-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non-zero policy gradient.

摘要：对推理大语言模型（LLM）进行后训练通常需要在昂贵的监督成本和计算效率低下之间做出权衡：监督微调通常需要推理轨迹，这在数据利用上效率低下，因为在许多应用中，人们只有“提示-答案”元组，而缺乏真实的思维链（CoT）序列。另一方面，强化学习（例如带有可验证奖励的强化学习）可以在没有 CoT 数据的情况下运行，但其计算效率要低得多，因为它在每个优化步骤中需要多次自回归补全，且无法保证其中任何一次能产生非零的策略梯度。

In this work we explore a middleground, a self-distillation technique that uses answer-only data, treating CoT as latent variables. This relies on training a partner “inverse reasoning LLM” that generates the original’s reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, with all generation happening offline and up front, making the approach particularly suited to continual post-training.

在这项工作中，我们探索了一种折中方案，即一种使用仅含答案数据的自蒸馏技术，将思维链视为潜在变量。这依赖于训练一个合作伙伴“逆向推理大语言模型”，该模型根据答案生成原始模型的推理轨迹。逆向模型只需训练一次，即可在后续输入的仅含答案的数据批次中重复使用，所有生成过程均在离线状态下预先完成，这使得该方法特别适用于持续后训练。

Using Qwen3-0.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts and, on Countdown, outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio (with smaller gains on GSM8K); rejection sampling by construction has no signal on problems the model never solves on its own.

在使用 Qwen3-0.6B 模型进行 Countdown 和 GSM8K 任务的测试中，我们发现，在 GRPO 训练达到饱和后，通过逆向模型进行的一轮离线蒸馏，在困难提示上的 pass@1 指标大约翻了一番；在 Countdown 任务中，该方法在所有混合比例下的各项长尾指标上均优于拒绝采样自蒸馏（在 GSM8K 上的增益较小）；而拒绝采样从机制上讲，对于模型自身无法解决的问题无法提供任何信号。

Continued GRPO on the fresh data retains an edge at hard pass@1 but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness. When used for best-of-$n$ selection at test time, it raises selection accuracy on Countdown from 0.753 (majority voting) to 0.969.

在新鲜数据上继续进行 GRPO 训练虽然在困难问题的 pass@1 指标上保持了优势，但在 pass@8 和 pass@32 指标上却落后于蒸馏方法，且需要在每一步重新生成 rollout。此外，我们还发现逆向模型具有第二个应用场景，即作为准确的正确性验证器。当在测试时用于“best-of-n”选择时，它将 Countdown 任务的选择准确率从 0.753（多数投票法）提高到了 0.969。