Latent Communication Between Language Model Agents: Channels, Alignment, and the Limits of Text

语言模型智能体之间的潜在通信：通道、对齐与文本的局限性

Abstract: Multi-agent systems (MAS) are utilized in many contexts and many professions. Those MAS rely on inter-agent communication, usually implemented by clear-text message passing. We hypothesize that Large Language Models may have a world model at their disposal that exceeds expressibility in text when complex concepts need to be communicated. Our aim is to approach a proof of this hypothesis with structured experiments.

摘要： 多智能体系统（MAS）已被应用于许多场景和行业。这些系统依赖于智能体间的通信，通常通过明文消息传递来实现。我们假设，当需要交流复杂概念时，大型语言模型可能拥有超出文本表达能力的内部世界模型。我们的目标是通过结构化实验来验证这一假设。

In this work, we show that LLM agents communicating via text lose information, which we quantify via Sparse Autoencoder (SAE) feature analysis. We construct three communication channels and measure concept-discriminating information in each. We first show that the SAE-sparse channel retains a 99.4% probe accuracy at 28-fold compression over the dense-latent channel vs 80.4% for the text channel.

在这项工作中，我们证明了通过文本进行通信的 LLM 智能体会丢失信息，并通过稀疏自编码器（SAE）特征分析对这种损失进行了量化。我们构建了三个通信通道，并测量了每个通道中的概念区分信息。我们首先展示了 SAE 稀疏通道在 28 倍压缩下仍保留了 99.4% 的探测准确率（相对于稠密潜在通道），而文本通道仅为 80.4%。

We then proceed to examine the same for cross-architecture communication by using sparse latent space alignment. We find for Procrustes alignment a 92% top-1 retrieval between Llama and Mistral. Using a text round-trip, we perform feature survival analysis to find that text serialization destroys 88% of SAE features, replacing them with a different feature set. We attribute the loss to identity replacement, not attenuation.

随后，我们通过稀疏潜在空间对齐，研究了跨架构通信的情况。我们发现，使用 Procrustes 对齐时，Llama 和 Mistral 之间的 Top-1 检索准确率达到 92%。通过文本往返测试，我们进行了特征存活分析，发现文本序列化破坏了 88% 的 SAE 特征，并将其替换为另一组特征。我们将这种损失归因于身份替换，而非衰减。

By our analysis, we were able to attribute a 3-10pp performance penalty to the linear Procrustes alignment, improving with nonlinear alignment methods. In a task-level evaluation we find that the latent channel matches the text channel on cross-lingual concept tasks but never exceeds it. Text augmentation with latent features provides no benefit, leading us to negative conclusions for the initial hypothesis: lost features mostly or completely encode surface form, not task-relevant semantics.

根据我们的分析，线性 Procrustes 对齐导致了 3-10 个百分点的性能损失，而使用非线性对齐方法可以改善这一情况。在任务级评估中，我们发现潜在通道在跨语言概念任务上的表现与文本通道相当，但从未超越它。使用潜在特征进行文本增强并没有带来任何益处，这使我们对最初的假设得出了否定结论：丢失的特征大多或完全编码了表面形式，而非任务相关的语义。

To pinpoint the practical advantage of latent communication over a text channel, deeper tasks eliciting complex concepts and an corresponding analysis framework are needed.

为了明确潜在通信相对于文本通道的实际优势，需要更深层次的任务来引出复杂概念，并建立相应的分析框架。