Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agents

神经符号强化学习中的动作前提管理：具身智能体的三种部署策略

Humans carry behaviour knowledge of how to act in familiar situations into every new task rather than relearning it from scratch. There is no reason a Reinforcement Learning (RL) agent shouldn’t do the same: known behaviour patterns need not be learned, only applied. 人类在面对新任务时，会将熟悉情境下的行为知识迁移过去，而不是从零开始重新学习。没有理由要求强化学习（RL）智能体不能这样做：已知的行为模式无需重新学习，只需直接应用即可。

Neuro-symbolic RL bridges prior knowledge and RL by injecting symbolic knowledge alongside a learned policy. The point at which this knowledge is integrated is critical: a poor choice can produce, for instance, hallucinated preconditions, which surface as safety and reliability problems in agents acting in changing environments. 神经符号强化学习通过在学习到的策略之外注入符号知识，架起了先验知识与强化学习之间的桥梁。知识集成的切入点至关重要：如果选择不当，可能会产生例如“幻觉前提”等问题，这在动态环境中运行的智能体身上会表现为安全性和可靠性问题。

We formalise this behavioural knowledge as a precondition Bayesian network (BN) over the agent’s structural actions - the actions whose legality depends on preconditions, such as picking up a key, grasping a block, toggling a door, or dropping an object. 我们将这种行为知识形式化为一种作用于智能体“结构化动作”的前提贝叶斯网络（BN）。所谓结构化动作，是指那些合法性依赖于前提条件的动作，例如拾取钥匙、抓取积木、开关门或放下物体。

The BN restricts when these actions may fire, and we inject it into the RL loop at three placements: (1) a symbolic verifier, consulted only at inference, that fires a structural action once its preconditions hold; (2) a symbolic enforcer, active during both training and inference, that governs structural-action use throughout learning; and (3) a symbolic learner, which folds the knowledge into the network and learns the restriction and use of structural actions itself. 该贝叶斯网络限制了这些动作的触发时机，我们将它注入到强化学习循环中的三个位置：（1）符号验证器（Symbolic Verifier）：仅在推理阶段调用，当动作前提满足时触发结构化动作；（2）符号执行器（Symbolic Enforcer）：在训练和推理阶段均保持活跃，在整个学习过程中管控结构化动作的使用；（3）符号学习器（Symbolic Learner）：将知识融入神经网络，并自主学习结构化动作的限制与使用。

To test the three variants we run experiments on two benchmarks with opposite regimes: one built on long, ordered planning chains, the other on continuous manipulation. We compare against strong baselines on solution quality, sample efficiency, and traceability. 为了测试这三种变体，我们在两种截然不同的基准测试上进行了实验：一种基于长序列的有序规划链，另一种基于连续操作。我们从解决方案质量、样本效率和可追溯性等方面与强基线模型进行了对比。

The payoff is substantial. On MiniGrid, all three placements improve the solution quality over the PPO+RND baseline, the symbolic enforcer leading at 98.2% against the baseline’s 88.8%. On Fetch, … 实验结果非常显著。在 MiniGrid 测试中，所有三种部署策略在解决方案质量上均优于 PPO+RND 基线，其中符号执行器表现最佳，达到了 98.2%，而基线仅为 88.8%。在 Fetch 测试中，……