DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

DEEPO：用于缓解多模态大模型幻觉的双熵增强策略优化

Abstract: Reinforcement learning (RL) is widely used to sharpen reasoning in multimodal large language models (MLLMs), yet its effect on hallucination is uneven. We trace this to two weak points in the \emph{correction chain} from reward to parameter update. At the rollout level, hard queries---those with high semantic entropy---frequently produce unanimously wrong sample groups, collapsing the group-relative advantage to zero exactly where hallucination risk is highest. At the optimization level, confident-but-wrong tokens are gradient-invisible: a categorical policy’s expected score-gradient norm vanishes as its distribution sharpens, so the predictions that most need correction receive the weakest updates.

摘要： 强化学习（RL）被广泛用于提升多模态大模型（MLLMs）的推理能力，但其在缓解幻觉方面的效果却参差不齐。我们将此归因于从奖励到参数更新的“修正链”中存在的两个薄弱环节。在采样（Rollout）层面，高语义熵的困难查询经常导致生成的样本组全部错误，使得组内相对优势（group-relative advantage）在幻觉风险最高的地方归零。在优化层面，那些“自信但错误”的 Token 具有梯度不可见性：当分类策略的分布变得尖锐时，其期望得分梯度范数会消失，导致最需要修正的预测反而获得最微弱的更新。

We propose Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage enhancement combining signal variance regularization with gradient preconditioning: semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance, while advantage-sign-aware Renyi preconditioning counteracts logit-level saturation so correction reaches confident errors in the operational confidence regime. Both branches improve over GRPO individually; their interaction is statistically significant on VideoMMMU---the most complex long-horizon task in our evaluation suite (+4.0$, 95% CI [1.1, 6.9])---and additive elsewhere. DEEPO reduces hallucination while preserving accuracy and training stability.

我们提出了双熵增强策略优化（DEEPO），这是一种结合了信号方差正则化与梯度预处理的双阶段增强方法：语义熵触发的专家前缀（expert prefixes）在高不确定性查询中注入基于事实的续写，从而提供直接监督并恢复优势方差；同时，利用优势符号感知的 Renyi 预处理来抵消 Logit 层的饱和，确保修正能够触达处于高置信度区间的错误预测。这两个分支在单独使用时均优于 GRPO；在我们的评估套件中最复杂的长序列任务 VideoMMMU 上，两者的交互作用具有统计学显著性（+4.0，95% 置信区间 [1.1, 6.9]），在其他任务中也表现出叠加增益。DEEPO 在减少幻觉的同时，保持了准确性和训练稳定性。