Privacy Without Regret: Differentially Private Inference-Time Alignment
Privacy Without Regret: Differentially Private Inference-Time Alignment
无遗憾隐私：差分隐私推理时对齐
Abstract: Best-of-N (BoN) sampling is the simplest and most widely deployed inference-time alignment strategy, but it suffers from two distinct problems: reward hacking, in which the selected response exploits errors in the proxy reward model, and the absence of any privacy protection for the sensitive human preference data used to train that reward model.
摘要： “Best-of-N (BoN)”采样是目前最简单且应用最广泛的推理时对齐策略，但它存在两个显著问题：一是奖励欺骗（reward hacking），即选定的回复利用了代理奖励模型中的错误；二是缺乏对训练该奖励模型所使用的敏感人类偏好数据的隐私保护。
We show that a single intervention—adding calibrated noise to reward scores before selection—resolves both. Our first result, Private Best-of-N (PrivBoN), establishes that Gumbel noise at an appropriate scale simultaneously provides $\epsilon$-differential privacy and implements KL-regularized alignment.
我们证明，只需进行一项干预——在选择前向奖励分数中添加校准噪声——即可同时解决这两个问题。我们的第一个成果“私有Best-of-N (PrivBoN)”表明，在适当尺度下添加Gumbel噪声，既能提供$\epsilon$-差分隐私，又能实现KL正则化对齐。
Whenever the privacy budget exceeds a critical threshold $\epsilon^*$, the privacy-mandated noise is the regret-optimal regularization, and privacy imposes zero additional alignment cost—matching the information-theoretic skyline of Huang et al. (2025).
当隐私预算超过临界阈值 $\epsilon^*$ 时，隐私所需的噪声即为遗憾最优（regret-optimal）的正则化手段，且隐私保护不会带来额外的对齐成本，这与Huang等人（2025）提出的信息论天际线（skyline）相吻合。
Because $\epsilon^*$ depends on an unknown coverage coefficient, we introduce Private Inference-Time Pessimism (PrivITP), which combines $\chi^2$-regularized rejection sampling with a two-phase Gaussian mechanism. PrivITP achieves ex-post $(\epsilon,\delta)$-DP with a privacy cost independent of the number of responses $n$, cleanly decouples the regularization parameter from the privacy parameter, and attains the skyline up to a noise-inflation term.
由于 $\epsilon^*$ 取决于一个未知的覆盖系数，我们引入了“私有推理时悲观主义 (PrivITP)”，它结合了$\chi^2$-正则化拒绝采样与两阶段高斯机制。PrivITP实现了事后 $(\epsilon,\delta)$-差分隐私，其隐私成本与回复数量 $n$ 无关，清晰地将正则化参数与隐私参数解耦，并在仅存在噪声膨胀项的情况下达到了天际线水平。
Experiments across several language models, datasets, and reward models confirm our results: PrivBoN and PrivITP are scaling-monotonic (unlike BoN, which degrades past a critical $n$), and PrivITP matches or outperforms PrivBoN at equivalent privacy levels, with the largest gains in the strong-privacy regime.
在多种语言模型、数据集和奖励模型上的实验证实了我们的结论：PrivBoN和PrivITP具有缩放单调性（与BoN不同，BoN在超过临界值 $n$ 后性能会下降），且在同等隐私水平下，PrivITP的表现优于或等同于PrivBoN，在强隐私保护场景下增益最为显著。