Intrinsic Mutual Information as a Modulator for Preference Optimization

内在互信息作为偏好优化的调节器

Abstract: Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead.

摘要： 离线偏好优化方法（如直接偏好优化，DPO）在将大语言模型（LLMs）与人类价值观对齐方面具有显著优势。然而，使用这些方法实现最佳性能通常需要额外的超参数调整，从而导致大量的时间开销。

Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization.

尽管先前的工作已经提出了一系列改进方案，但这些方法的有效性仍然有限，且未能完全消除对超参数调整的依赖。在这项工作中，我们提出了 RMiPO，这是一个用于离线偏好优化的轻量级且高效的框架。

RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at this https URL.

RMiPO 利用内在的响应级互信息进行偏好优化，并结合超参数调节，以极低的额外计算成本动态解耦偏好贡献。广泛的实验结果表明，RMiPO 在实现比现有方法更优性能的同时，将训练开销降低了 15% 以上。我们的代码可在该链接获取。