When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

何时通信：多智能体强化学习中基于信念分布与 KL 散度的原则性门控机制

Abstract: Effective communication in multi-agent reinforcement learning requires agents to decide not only what to communicate, but when? Existing approaches either communicate at every timestep or learn a binary gate through REINFORCE policy gradients, a high-variance signal that produces unstable and uninterpretable gating behavior. I propose a principled alternative: agents communicate only when the KL divergence between their learned belief distributions exceeds a fixed threshold. Each agent maintains a belief distribution over a latent world state computed as a softmax over its LSTM hidden state, and communicates only when belief disagreement is large enough to justify information exchange.

摘要： 在多智能体强化学习中，有效的通信不仅要求智能体决定“通信什么”，还要求决定“何时通信”。现有的方法要么在每个时间步都进行通信，要么通过 REINFORCE 策略梯度学习一个二元门控机制，但这种高方差信号会导致不稳定且难以解释的门控行为。我提出了一种原则性的替代方案：仅当智能体之间学习到的信念分布（belief distributions）之间的 KL 散度超过固定阈值时，智能体才进行通信。每个智能体通过对其 LSTM 隐藏状态进行 Softmax 计算，来维护一个关于潜在世界状态的信念分布，并仅在信念分歧大到足以证明信息交换的必要性时才进行通信。

I evaluate this approach on the Predator-Prey benchmark from IC3Net across two environment sizes with 5 seeds each, and on MPE simple_spread, comparing against IC3Net, CommNet, and an independent controller. On PP 10×10, IC3Net outperforms KL-belief at all thresholds. On the harder PP 20×20, a threshold ablation over $\varepsilon \in {0.1, 0.3, 0.5, 1.0}$ reveals an inverted U-shape: $\varepsilon=0.5$ achieves 73.84 average steps and 42% success rate versus IC3Net’s 75.31 steps and 31%, a gap of 1.47 steps and 11 percentage points with tighter seed variance.

我在 IC3Net 的“捕食者-猎物”（Predator-Prey）基准测试（包含两种环境规模，每种规模使用 5 个随机种子）以及 MPE simple_spread 环境中评估了该方法，并将其与 IC3Net、CommNet 和独立控制器进行了对比。在 10×10 的 PP 环境中，IC3Net 在所有阈值下均优于 KL-belief 方法。在难度更高的 20×20 PP 环境中，对 $\varepsilon \in {0.1, 0.3, 0.5, 1.0}$ 进行的阈值消融实验揭示了一个倒 U 型曲线：当 $\varepsilon=0.5$ 时，平均步数为 73.84 步，成功率为 42%，而 IC3Net 为 75.31 步和 31%。这表明该方法在更紧凑的种子方差下，实现了 1.47 步的提升和 11 个百分点的成功率增长。

On MPE, the belief head improves mean reward by 12 points and reduces variance by 26× even when gating is inactive, suggesting two orthogonal contributions: principled gating when beliefs can converge, and improved latent representations that benefit coordination regardless.

在 MPE 环境中，即使在门控未激活的情况下，信念头（belief head）也将平均奖励提高了 12 分，并将方差降低了 26 倍。这表明该方法具有两个正交的贡献：一是在信念能够收敛时提供原则性的门控机制，二是改善了潜在表征，从而在任何情况下都有助于提升协作效果。