Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
Computer Science > Computation and Language arXiv:2610.40726 (cs) [Submitted on 29 Oct 2026] Title: Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping Authors: Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao.
计算机科学 > 计算与语言 arXiv:2610.40726 (cs) [提交于 2026 年 10 月 29 日] 标题：Li$^{2}$SAO：通过组相对行动者-评论家引导提升轻量级单次展开异步强化学习优化。作者：Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao。
Abstract: Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks.
摘要：异步强化学习 (RL) 正成为大型语言模型 (LLM) 后训练的主流范式，因为它通过在展开 (rollouts) 到达时更新模型来提高训练效率。近期研究指出，GRPO 中的组采样并不自然地适用于异步强化学习训练，因为展开长度遵循长尾分布，并提出了单次展开异步优化 (SAO) 以提高稳定性和离策略行为，特别是在智能体任务中。
However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained, LiSAO lags behind GRPO substantially in early training steps.
然而，SAO 依赖于通过扩展价值预训练获得的闭源价值模型。在本文中，我们在单轮推理和多轮智能体任务上分析了 GRPO 和 LiSAO（我们对 SAO 的轻量级复现，不含预训练价值模型）。我们发现 GRPO 在推理任务上优于 LiSAO，而在智能体任务上则相反。由于 LiSAO 的价值模型尚未得到充分训练，LiSAO 在训练初期阶段明显落后于 GRPO。
To address this issue, we propose Li$^{2}$SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling.
为了解决这个问题，我们提出了 Li$^{2}$SAO，它通过组相对行动者-评论家引导来提升轻量级单次展开异步优化：当评论家处于冷启动状态时，使用组相对优势训练行动者，并使用组中心回报训练评论家，之后训练切换为单次展开 GAE。实验表明，Li$^{2}$SAO 在训练初期赶上或超过了 GRPO，并进一步提高了后期的性能上限。