Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Osprey：与目标无关的预训练技术，打造推测解码中更强大的草稿模型

Speculative decoding is critical for accelerating LLM inference. However, the speedup is fragile: drafters are typically trained against a narrow distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining.

推测解码（Speculative decoding）对于加速大语言模型（LLM）推理至关重要。然而，这种加速效果非常脆弱：草稿模型（Drafter）通常针对单一目标模型的狭窄分布进行训练，一旦工作负载发生变化，其接受率就会大幅下降。这与现代大语言模型的发展趋势形成了鲜明对比——现代大语言模型之所以被重视，恰恰是因为它们通过大规模预训练获得了广泛的泛化能力。

We argue that the natural remedy, pretraining, has been hard to apply to drafters because existing recipes are target-specific: the drafter consumes the target’s hidden states and is distilled on the target’s logits, so pretraining must be repeated for each target. We introduce Osprey, which instead bootstraps drafters from off-the-shelf pretrained small language models, treating broad pretraining as a reusable, target-agnostic asset and reducing per-target work to a lightweight adaptation step.

我们认为，解决这一问题的自然方法——预训练——一直难以应用于草稿模型，因为现有的方案都是针对特定目标的：草稿模型需要消耗目标模型的隐藏状态，并根据目标模型的 Logits 进行蒸馏，因此必须针对每个目标模型重复进行预训练。我们引入了 Osprey，它转而利用现成的预训练小语言模型来引导草稿模型，将广泛的预训练视为一种可重用的、与目标无关的资产，并将针对每个目标的工作量减少为轻量级的适配步骤。

Realizing this requires overcoming two challenges: small LMs are far deeper than a latency-bound drafter can afford, and their pretrained computation must remain intact while the drafter learns to ingest target hidden states and emit tokens in the target’s vocabulary. Osprey addresses both by pruning to a shallow backbone, restoring its language-modeling capability with target-agnostic next-token pretraining, and adapting it to each target through vocabulary alignment, zero-initialized QKV expansion, and distillation from the target model’s output distribution.

实现这一目标需要克服两个挑战：小语言模型的深度远超延迟敏感型草稿模型的承受能力；同时，在草稿模型学习摄取目标隐藏状态并输出目标词表 Token 的过程中，必须保持其预训练的计算能力不变。Osprey 通过以下方式解决了这两个问题：将其剪枝为浅层骨干网络，通过与目标无关的“下一词预测”预训练恢复其语言建模能力，并通过词表对齐、零初始化 QKV 扩展以及从目标模型输出分布进行蒸馏，将其适配到每个目标模型上。

Empirically, a single pretrained Osprey backbone transfers across targets and improves mean acceptance length by 16.1% for Qwen3-8B, 21.2% for Llama-3.3-70B-Instruct, and 22.7% for the 229B MiniMax-M2.5 (with 17.5% higher tokens per second), with the largest gains on out-of-domain and multilingual data. Our code is available at this https URL.

实验证明，单一的预训练 Osprey 骨干网络可以在不同目标模型间迁移，并将平均接受长度分别提升了：Qwen3-8B 提升 16.1%，Llama-3.3-70B-Instruct 提升 21.2%，229B 的 MiniMax-M2.5 提升 22.7%（每秒 Token 数提升 17.5%），其中在域外数据和多语言数据上的增益最为显著。我们的代码已在链接中提供。