MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

Computer Science > Artificial Intelligence arXiv:2610.12659 (cs) [Submitted on 8 Oct 2026] Title: MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis Authors: Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi, David Acuna.

计算机科学 > 人工智能 arXiv:2610.12659 (cs) [提交于 2026 年 10 月 8 日] 标题：MagicBeanStalk：通过递归数据合成逃离 RLVR 零优势陷阱 作者：Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi, David Acuna。

Abstract: Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget.

摘要：RLVR 的数据中心化扩展主要集中在收集或生成可验证的训练问题上。然而，语料库的大小本身并不能决定数据集是否为目标策略提供了有用的学习信号。为了指导有效 RLVR 数据的合成，我们通过实验将数据集的有效性表征为问题、目标策略和训练展开预算的函数。

Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems.

在这种表征下，我们发现一个从广泛使用的 RLVR 数据集中汇集的大型开源数学语料库几乎完全陷入了“零优势陷阱”：对于固定的目标策略和展开预算，大约 88% 的问题要么在每次展开中都被解决，要么在每次展开中都失败。过滤虽然能集中剩余部分，但它只能进行选择，从而丢弃了难题。

Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this.

因此，有效的合成问题应该需要解决那些难题所需的推理操作和组合。然而，它们的难度必须经过校准，以便使一些展开成功，一些失败，并且至少有一个正确的解决方案符合训练预算。直接提示教师去改变难题并不能实现这一点。

Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds’ reasoning blueprints.

在这些经验原则的指导下，我们引入了 MagicBeanStalk，这是一种自顶向下的分解和自底向上的递归数据合成算法，它将种子问题扩展为一系列实质上全新的、经过策略和预算校准的训练问题，同时保留了种子的推理蓝图。

Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves by only +1.1 and its best filtered real data subset by +5.8.

在 8 个数学基准测试中，在 16K token 的预算下，对 MagicBeanStalk 问题进行持续 RLVR 训练，使 gpt-oss-120b 从 61.2 提升至 69.8 (+8.6)，DeepSeek v4Flash 从 72.4 提升至 77.9 (+5.4)。使用完整的 78K 源语料库进行训练仅提升了 +1.1，而其最佳过滤后的真实数据子集仅提升了 +5.8。

In finance, we synthesize over 14K verifiable problems from only 340 solutions and improve over base by 6.9 and 2.4 points on SecQue and FinanceBench, respectively.

在金融领域，我们仅从 340 个解决方案中合成了超过 14K 个可验证问题，并在 SecQue 和 FinanceBench 上分别比基准提高了 6.9 和 2.4 个百分点。