SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning

洞察未来：深度强化学习的前瞻性解释方法

Deep reinforcement learning (DRL) agents achieve strong performance in complex environments, yet their decision-making processes remain difficult to interpret. 深度强化学习（DRL）智能体在复杂环境中表现出色，但其决策过程仍然难以解释。

We introduce SPOT (Sampling Policy Observation Tree), a novel model-agnostic, sampling-based framework for interpreting DRL policies. 我们引入了 SPOT（采样策略观察树），这是一种新颖的、与模型无关的、基于采样的 DRL 策略解释框架。

Given access to the policy and an environment simulator, SPOT constructs an interpretable finite-horizon tree by sampling actions and recursively simulating the resulting successor states. 在能够访问策略和环境模拟器的情况下，SPOT 通过采样动作并递归模拟产生的后续状态，构建出一棵可解释的有限视界树。

The tree provides an empirical representation of the policy’s action preferences and their possible downstream evolution. 该树提供了策略动作偏好及其可能后续演变的经验性表示。

We provide formal guarantees establishing SPOT’s asymptotic recovery of the policy’s unique most probable action and characterizing its disagreement behavior under high-entropy policies. 我们提供了形式化保证，确立了 SPOT 在渐近意义上恢复策略唯一最可能动作的能力，并刻画了其在高熵策略下的分歧行为。

We demonstrate SPOT in the SUMO-RL traffic-signal control domain. 我们在 SUMO-RL 交通信号控制领域演示了 SPOT 的应用。

The case study illustrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods. 案例研究表明，其基于树的表示方法可用于检查策略偏好、比较不同的未来轨迹，并揭示那些通过单时间步特征归因方法无法观察到的后续行为。