Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

通过世界模型从人类偏好与理由中学习安全智能体行为

Abstract: We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input.

摘要： 我们探讨了在环境动态未知且缺乏合适奖励函数的情况下，如何安全地训练智能体策略并部署高效且安全的策略。在安全关键型环境的背景下，我们认为传统的强化学习方法并不切实际，因此转向利用人类反馈资源。

We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice.

我们引入了 DROPJ，这是一种以人为中心的方法，旨在实现安全训练与部署。首先，我们利用先前的真实世界轨迹数据集学习一个世界模型（即学习型模拟器）。随后，人类在这一模拟器中进行操作，以提取若干具有信息量的模拟轨迹。我们从中采样模拟轨迹片段对，并引导人类对这些片段表达偏好，同时要求其提供选择的理由（即解释）。

We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment.

接着，我们利用这些带有理由的偏好训练一个奖励模型，并将其与世界模型结合，通过模型预测控制（MPC）直接部署智能体。通过真实用户实验，我们发现，由用户生成具有信息量的模拟轨迹，相较于其他策略能显著降低训练过程中的计算成本，并能提升部署阶段的性能。

In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.

在学习型模拟器训练的背景下，我们证明了使用偏好反馈而非其他类型的反馈，能够大幅提升部署时的性能。此外，我们还证明了伴随偏好提供的安全理由，可以在部署过程中显著增强安全性，或优先考虑用户所指定的安全相关维度。