Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback

查看原文 / View Original 编辑 / Edit

Computer Science > Machine Learning arXiv:2610.26914 (cs) [Submitted on 17 Oct 2026] Title: Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback Authors: Akansha Kalra, Ellen R. Novoseller, Vernon J. Lawhern, Daniel S. Brown.

计算机科学 > 机器学习 arXiv:2610.26914 (cs) [提交于 2026 年 10 月 17 日] 标题：基于匿名人群反馈的可引导多元策略对齐作者：Akansha Kalra, Ellen R. Novoseller, Vernon J. Lawhern, Daniel S. Brown。

Abstract: Pluralistic alignment requires AI systems to learn from diverse populations whose preferences may reflect heterogeneous or even conflicting objectives, and to personalize their behavior to individual users. Existing approaches in pluralistic alignment often assume access to annotator identities and rely on powerful pretrained representations to model individual preference variations.

摘要：多元对齐要求人工智能系统从不同群体中学习，这些群体的偏好可能反映出异质甚至相互冲突的目标，并要求系统针对个人用户个性化其行为。现有的多元对齐方法通常假设可以获取标注者身份，并依赖强大的预训练表示来建模个人偏好的差异。

We propose Preference-PCA Pluralistic Learning (P3L), a novel framework for pluralistic alignment that jointly learns trajectory embeddings and the shared basis of preference directions via PCA decomposition from heterogeneous crowd feedback without requiring annotator identities. At test time, our approach enables rapid and accurate personalized reward learning and downstream policy optimization.

我们提出了偏好主成分分析多元学习（P3L），这是一种用于多元对齐的新型框架。该框架无需标注者身份，通过对异质人群反馈进行主成分分析（PCA）分解，联合学习轨迹嵌入和偏好方向的共享基。在测试阶段，我们的方法能够实现快速且准确的个性化奖励学习及下游策略优化。

Experiments across multiple embodied control tasks demonstrate that P3L enables steerable personal alignment that generalizes beyond the offline dataset to out-of-distribution new users and attains competitive RL and test performance, while substantially reducing computational cost and supporting variable adaptation query budgets.

在多个具身控制任务上的实验表明，P3L 能够实现可引导的个人对齐，使其泛化能力超越离线数据集，适用于分布外的新用户，并获得具有竞争力的强化学习和测试性能，同时大幅降低了计算成本并支持可变的适应性查询预算。