RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation
RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation
RIMS：面向小规模大语言模型检索增强生成的平滑多对聚合偏好优化
Small-scale language models (SLMs) are attractive for retrieval-augmented generation (RAG) in resource-constrained settings, but their limited capacity makes them highly sensitive to noisy or spurious retrieved evidence. Existing preference-based methods such as RoseRAG select only the hardest single preference pair via hard argmin/argmax, discarding the remaining signal; others treat multiple pairs as independent binary comparisons, resulting in low data utilization.
小规模语言模型（SLMs）在资源受限的检索增强生成（RAG）场景中极具吸引力，但其有限的容量使其对噪声或虚假的检索证据高度敏感。现有的基于偏好的方法（如 RoseRAG）仅通过硬 argmin/argmax 选择最困难的单个偏好对，从而丢弃了其余的信号；而其他方法则将多个偏好对视为独立的二元比较，导致数据利用率低下。
We propose RIMS, a three-stage preference optimization framework comprising (1) synthetic chain-of-thought preference data generation via rejection sampling using the target SLM itself without relying on proprietary models, (2) a differentiable soft aggregation mechanism that replaces hard selection with a smooth operator, preserving gradient signal from all preference pairs while retaining the discriminative structure of margin-aware selection, and (3) preference optimization with the smoothed objective applied to multiple alignment algorithms.
我们提出了 RIMS，这是一个包含三个阶段的偏好优化框架：(1) 利用目标 SLM 本身通过拒绝采样生成合成思维链（CoT）偏好数据，无需依赖专有模型；(2) 一种可微分的软聚合机制，用平滑算子替代硬选择，在保留边际感知选择的判别结构的同时，保留来自所有偏好对的梯度信号；(3) 将平滑目标应用于多种对齐算法的偏好优化。
We theoretically show that the smoothed approximation admits a controllable error bound and that smooth aggregation yields provably tighter gradient alignment to the oracle objective than hard selection. Experiments on four multi-hop question answering benchmarks show that our approach outperforms state-of-the-art baselines across multiple SLM backbones, achieving consistent gains in Exact Match and F1 under noisy retrieval conditions.
我们在理论上证明了该平滑近似具有可控的误差界限，并且平滑聚合在梯度对齐方面比硬选择更能证明其与预言机（oracle）目标的一致性。在四个多跳问答基准测试上的实验表明，我们的方法在多个 SLM 主干网络上均优于最先进的基线模型，在噪声检索条件下实现了精确匹配（Exact Match）和 F1 分数的持续提升。