Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

标注即演练：面向视频多模态大模型的高效可扩展强化学习

Abstract: Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation.

摘要： 多模态大模型（MLLMs）已成为统一视频感知的主流范式。然而，在大型多任务数据集上进行后训练仍然充满挑战，因为现有的强化学习方法即便在昂贵的思维链（CoT）生成辅助下，其采样策略组中高质量的演练（rollouts）依然稀缺。

In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target.

本文研究了视频多模态大模型强化学习后训练的样本效率与可扩展性，并提出了 OraRL。我们发现标注存在一个被忽视的作用：除了用于评估演练得分外，每个标注都可以作为“预言家演练”（oracle rollout）进入其策略组，成为直接的正向优化目标。

Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage.

然而，直接整合预言家数据并非易事：高奖励的预言家数据会提高组基准线，并导致原本正向的策略优势发生反转，我们将这种失败称为“优势反转”（advantage inversion）。OraRL 的核心是一个解耦的优势估计器：策略演练确定一个无预言家的基准线，而预言家与策略之间的差距则同时调节方向增益和一个独立的解耦预言家优势。

Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts.

符号平衡剪枝（Sign-balanced pruning）提升了效率：通过仅保留预言家数据和每种符号中最强的演练，OraRL 的步进时间仅为 SFT 的 2.2 倍，不到带 CoT 的 GRPO 所需 4.9 倍的一半。OraRL 能够随模型规模和数据量扩展，在 0.8B 到 9B 的模型规模上均超越了其基座模型，并在高达 10 万条提示词的规模下超越了 GRPO。

Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.

在无需思维链的情况下，Video-ORA-9B 的解码时间从 4,780 毫秒缩短至 130 毫秒。与此前各领域的最佳模型相比，它将时间 mIoU 从 62.5 提升至 66.0，跟踪 AO 从 73.0 提升至 78.2，分割指标从 64.3 提升至 70.4，三个基准测试的空间智能宏平均分从 51.0 提升至 56.1；在 VSI-Bench 上，其得分为 73.1，而 GPT-5 为 55.0，Gemini-3-Pro 为 55.1。