X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding

Abstract: This paper investigates collaborative speculative decoding (CoSD), a distributed large language model (LLM) inference framework in which an on-device small language model (SLM) drafts candidate tokens and a server LLM verifies them.

摘要： 本文研究了协作推测解码（CoSD），这是一种分布式大语言模型（LLM）推理框架，其中端侧小语言模型（SLM）负责起草候选词元（token），而服务器端 LLM 负责验证这些词元。

Existing CoSD methods assume a shared vocabulary between the SLM and the LLM and incur substantial communication load because residual resampling requires token distribution exchange between the user device and the edge server.

现有的 CoSD 方法假设 SLM 和 LLM 之间共享词表，且由于残差重采样（residual resampling）需要用户设备与边缘服务器之间进行词元分布交换，因此会产生巨大的通信负载。

To address these limitations, we propose cross-vocabulary CoSD (X-CoSD), a lossless and communication-efficient CoSD framework for heterogeneous SLM-LLM vocabularies.

为了解决这些局限性，我们提出了跨词表 CoSD（X-CoSD），这是一种针对异构 SLM-LLM 词表的无损且通信高效的 CoSD 框架。

X-CoSD is built on hybrid resampling (HR), which splits residual resampling across the common-vocabulary region on the device and the LLM-only region on the server, so that distribution transmission is required only for the common-vocabulary region.

X-CoSD 基于混合重采样（HR）构建，该技术将残差重采样拆分为设备上的公共词表区域和服务器上的 LLM 独有区域，从而使得仅需在公共词表区域进行分布传输。

We further propose X-CoSD-E, an enhanced variant based on server resampling with device verification (SR-DV), in which the server sends only replacement candidates sampled from the server LLM and their corresponding probabilities for local verification at the device.

我们进一步提出了 X-CoSD-E，这是一种基于“服务器重采样与设备验证”（SR-DV）的增强变体。在该变体中，服务器仅发送从服务器 LLM 中采样出的替换候选词及其对应的概率，供设备进行本地验证。

We prove that both X-CoSD and X-CoSD-E preserve the server LLM distribution, and experiments show that they significantly improve token generation speed while maintaining generation quality comparable to that of the server LLM.

我们证明了 X-CoSD 和 X-CoSD-E 均能保持服务器 LLM 的分布一致性，实验结果表明，它们在显著提高词元生成速度的同时，保持了与服务器 LLM 相当的生成质量。