Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives.

将大型语言模型（LLM）与有用性、无害性和诚实性等多个目标对齐，通常依赖于针对每个目标的偏好判断，这使得标注成本随着目标数量的增加而线性增长。

Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data.

主动偏好学习可以通过选择那些预期标签最能改变策略的比较来降低这一成本。现有的补救措施要么在多目标设置中缺乏目标感知的查询选择机制，要么将冲突的偏好视为有害，从而将其从训练数据中排除。

We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy’s own implicit gradient, without relying on an external reward model.

我们引入了 OGC-DPO（目标条件梯度覆盖 DPO），这是一个主动学习框架，它将每个（响应对，目标）组合视为一个单独的查询，并通过策略自身的隐式梯度来衡量其信息量，而无需依赖外部奖励模型。

OGC-DPO then selects the queries that most expand each objective’s gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost.

随后，OGC-DPO 会选择那些最能扩展每个目标梯度覆盖范围的查询。在不同数量目标下的实证实验表明，在相同的标注成本下，OGC-DPO 比现有的主动偏好学习方法更有效。

Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.

进一步的分析表明，信息丰富的冲突并非一概有害，反而能提供有用的策略训练信号。