Up to 3.2x Faster Inference with LFM2.5-DSpark

使用 LFM2.5-DSpark 实现最高 3.2 倍的推理加速

Today, we release DSpark draft model checkpoints for three models from our LFM2.5 family: LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and LFM2.5-8B-A1B. These add a speculative decoding path that trades a minimal memory increase for a large decoding speedup without changing output quality: 今天，我们发布了 LFM2.5 系列中三款模型的 DSpark 草稿模型检查点：LFM2.5-1.2B-Instruct、LFM2.5-2.6B 和 LFM2.5-8B-A1B。它们增加了一条投机解码路径，以极小的内存增加换取了巨大的解码速度提升，且不会改变输出质量：

Faster inference: up to 3.18 throughput improvement on a GPU and up to 2.87x on-device.
更快的推理： GPU 吞吐量提升最高可达 3.18 倍，端侧设备提升最高可达 2.87 倍。
Toward on-device agentic inference: cuts function-calling latency by 57% on average for LFM2.5-2.6B.
迈向端侧智能体推理： 对于 LFM2.5-2.6B 模型，函数调用延迟平均降低了 57%。
Day-one support for llama.cpp and SGLang: LFM-compatible DSpark integration is open-sourced upstream.
首日支持 llama.cpp 和 SGLang： 兼容 LFM 的 DSpark 集成已在上游开源。

How does DSpark work

DSpark 的工作原理

The decode phase in LLM inference is traditionally memory-bound. Most latency comes from streaming weights from DRAM into SRAM, not from intense computation. Speculative decoding addresses this by using a lightweight draft model to produce candidate tokens, then having the target model verify them all in a single forward pass, sharing the cost of loading the weights across all tokens we verify. 大语言模型推理中的解码阶段通常受限于内存带宽。大部分延迟来自于将权重从 DRAM 传输到 SRAM 的过程，而非高强度的计算。投机解码通过使用轻量级草稿模型生成候选 Token，然后由目标模型在单次前向传播中验证所有候选 Token 来解决这一问题，从而将加载权重的成本分摊到所有被验证的 Token 上。

Over the years, multiple approaches of speculation have been proposed, with the most prominent being EAGLE-3, DFlash, and, most recently, DSpark, which combines three components: 多年来，人们提出了多种投机方法，其中最著名的是 EAGLE-3、DFlash，以及最近的 DSpark。DSpark 结合了三个组件：

DFlash-style parallel backbone conditioned on the target model’s context features, producing hidden states for all draft tokens in a single forward pass. DFlash 风格的并行主干网络：以目标模型的上下文特征为条件，在单次前向传播中为所有草稿 Token 生成隐藏状态。
A lightweight sequential head, modeled as a Markov chain between neighboring tokens, that adds inter-token dependency, raising the acceptance rate at later positions. 轻量级序列头：建模为相邻 Token 之间的马尔可夫链，增加了 Token 间的依赖关系，从而提高了后续位置的接受率。
A confidence-scheduled verifier that predicts each token’s survival probability and prunes low-confidence suffixes when verification would cost more than it saves. 置信度调度验证器：预测每个 Token 的存活概率，并在验证成本高于收益时剪除低置信度的后缀。

Training and Architecture

训练与架构

We follow the DSpark recipe with a larger and more diverse data mix covering SFT, chat, code, and function-calling data. Based on our ablations, the first versions of the draft models are simplified attention-only draft models, with 5 layers and a block of 9. For each draft model, we ran 15 epochs on the entire dataset and selected the epoch with the highest acceptance rate rather than the lowest loss. The resulting draft models are relatively small, with each around ~300M parameters. 我们遵循 DSpark 的方案，使用了涵盖 SFT（监督微调）、聊天、代码和函数调用数据在内更大且更多样化的数据集。根据我们的消融实验，草稿模型的首个版本是简化的纯注意力模型，包含 5 层和 9 个块。对于每个草稿模型，我们在整个数据集上运行了 15 个 Epoch，并选择了接受率最高（而非损失最低）的 Epoch。最终得到的草稿模型相对较小，每个模型约有 3 亿参数。

Quality parity

质量对齐

Under greedy decoding, a draft token is only accepted if it matches the target model’s distribution. On rejection, the target model’s own token takes its place. The emitted sequence is therefore identical to baseline greedy by construction, so benchmark accuracy (pass@1 or exact match) is unchanged. 在贪婪解码下，只有当草稿 Token 与目标模型的分布匹配时才会被接受。如果被拒绝，则由目标模型自身的 Token 取代。因此，生成的序列在结构上与基准贪婪解码完全相同，所以基准测试准确率（pass@1 或精确匹配）保持不变。

Inference Speed Up on CPU and GPU

CPU 和 GPU 上的推理加速

Our DSpark draft models for LFM2.5 ship with day-one support for llama.cpp (implementation builds on top of the official codebase, which we run with experimental metal kernels) and SGLang (implementation builds on the official SGLang implementation of DSpark). We measure on-device throughput with llama.cpp and Metal on an M4 Max MacBook Pro using FP16 GGUF weights and up to 256 output tokens. We measure GPU throughput with SGLang on a single H100 80 GB in BF16. Both configurations use a DSpark block size of 9, a batch size of 1, and a temperature of 0. 我们为 LFM2.5 提供的 DSpark 草稿模型在发布首日即支持 llama.cpp（实现基于官方代码库，我们使用实验性的 Metal 内核运行）和 SGLang（实现基于官方 SGLang 的 DSpark 实现）。我们使用 M4 Max MacBook Pro 上的 llama.cpp 和 Metal，在 FP16 GGUF 权重和最多 256 个输出 Token 的条件下测量了端侧吞吐量。我们使用单张 H100 80GB GPU 在 BF16 下通过 SGLang 测量了 GPU 吞吐量。两种配置均使用 9 的 DSpark 块大小、1 的批处理大小和 0 的温度。

(Note: The original article includes detailed performance tables for MATH500, HumanEval, MBPP, GSM8K, and MT-Bench datasets, showing significant throughput gains across all models.) （注：原文包含针对 MATH500、HumanEval、MBPP、GSM8K 和 MT-Bench 数据集的详细性能表格，显示所有模型在吞吐量上均有显著提升。）

How to use LFM2.5-DSpark

如何使用 LFM2.5-DSpark

Running the DSpark draft models with SGLang requires an SGLang build with DSpark support for LFM2. 使用 SGLang 运行 DSpark 草稿模型需要构建支持 LFM2 DSpark 的 SGLang 版本。