Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

通过速度匹配扩展扩散模型的强化学习

Abstract: Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing methods largely inherit policy-gradient machinery from large language models. 摘要： 奖励微调正成为使扩散模型适应人类偏好和特定任务目标的重要工具，但现有方法大多沿用了大型语言模型中的策略梯度机制。

Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples. As a result, current approaches either construct trajectory likelihoods from stochastic denoising transitions or approximate endpoint likelihoods with evidence lower bound, introducing additional computation and algorithmic complexity. 与自回归模型不同，扩散模型无法为生成的样本提供易于处理的似然估计。因此，当前的方法要么通过随机去噪转换构建轨迹似然，要么利用证据下界（ELBO）来近似端点似然，这引入了额外的计算和算法复杂度。

We demonstrate that this likelihood-based machinery is not necessary for effective diffusion reward fine-tuning. We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field. 我们证明，这种基于似然的机制对于有效的扩散模型奖励微调并非必要。我们提出了基于奖励的速度匹配（RVM），这是一种简单的、无需轨迹的更新方法，直接作用于速度场。

RVM reinforces directions associated with high-reward generations, suppresses those with low reward, and involves an optional anchor term controlling drift from a reference velocity. Notably, it provides a general framework that recovers recent fine-tuning methods, including RAM and DiffusionNFT, as special cases. RVM 强化与高奖励生成相关的方向，抑制低奖励的方向，并包含一个可选的锚定项，用于控制相对于参考速度的漂移。值得注意的是，它提供了一个通用框架，可以将包括 RAM 和 DiffusionNFT 在内的近期微调方法作为其特例涵盖在内。

Across various large-scale diffusion models reward fine-tuning tasks, RVM is competitive with or outperforms trajectory-based policy-gradient methods under substantially reduced training cost. We further find that, once the velocity update is simplified, the particular loss variant matters less than reward and anchor design. 在各种大规模扩散模型的奖励微调任务中，RVM 在大幅降低训练成本的情况下，表现出与基于轨迹的策略梯度方法相当甚至更优的性能。我们进一步发现，一旦速度更新被简化，具体的损失函数变体的重要性远不如奖励和锚定设计。

For video generation, standard preference rewards can favor visually clean but nearly static outputs; introducing a new dynamic-tracking reward that substantially improve motions while improving overall VBench performance. These results suggest that scalable reward fine-tuning for diffusion models is better posed in the native velocity representation than as likelihood-based policy optimization. 在视频生成方面，标准的偏好奖励往往倾向于视觉清晰但几乎静止的输出；我们引入了一种新的动态跟踪奖励，在提升 VBench 整体性能的同时，显著改善了运动效果。这些结果表明，对于扩散模型而言，在原生速度表示下进行可扩展的奖励微调，比将其视为基于似然的策略优化更为有效。