Accelerating vision-language models with LFM2.5-VL-DSpark
Accelerating vision-language models with LFM2.5-VL-DSpark
使用 LFM2.5-VL-DSpark 加速视觉语言模型
Today, we release an experimental DSpark draft model for our vision-language model (VLM) LFM2.5-VL-3B. As with our recently released LFM2.5-DSpark drafter models, it adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. 今天，我们为视觉语言模型 (VLM) LFM2.5-VL-3B 发布了一个实验性的 DSpark 草稿模型。正如我们最近发布的 LFM2.5-DSpark 草稿模型一样，它增加了一条推测解码路径，以极小的内存占用增加换取了显著的速度提升，且不会改变输出质量。
- Faster inference: decode speedups up to 3.13x on device and 2.66x on an H100, with end-to-end gains up to 2.62x and 2.27x.
- Small memory cost: the drafter adds 280M parameters, 8.9% on top of the 3B target.
- Day-one support: LFM-compatible DSpark integrations for llama.cpp, MLX-VLM, and SGLang.
- 更快的推理： 设备端解码速度提升高达 3.13 倍，H100 上提升高达 2.66 倍，端到端性能提升分别达到 2.62 倍和 2.27 倍。
- 较小的内存成本： 草稿模型增加了 2.8 亿参数，仅为 3B 目标模型参数量的 8.9%。
- 首日支持： 为 llama.cpp、MLX-VLM 和 SGLang 提供兼容 LFM 的 DSpark 集成。
How does speculative decoding work for VLMs
推测解码如何应用于视觉语言模型 (VLM)
The vision drafter uses the same architecture as our text LFM2.5-DSpark drafters: it captures the target model’s hidden states at a fixed set of tapped layers and conditions on them to draft a block of k candidate tokens. Image patches and text tokens are projected into a shared representation before those layers, so the drafter operates on hidden-state vectors of identical dimensionality regardless of input modality. The inference algorithm is therefore unchanged from the text models. 视觉草稿模型采用了与我们的文本 LFM2.5-DSpark 草稿模型相同的架构：它在固定的层级捕获目标模型的隐藏状态，并基于这些状态来起草包含 k 个候选 token 的块。图像补丁和文本 token 在进入这些层之前会被投影到共享的表示空间中，因此无论输入模态如何，草稿模型都在相同维度的隐藏状态向量上运行。因此，其推理算法与文本模型保持一致。
Training and Architecture
训练与架构
We follow the DSpark recipe with a mixture of vision-language SFT data, weighted toward the workloads we expect the model to serve. Based on ablations across 3, 4, and 5 layers, the draft model is a simplified attention-only drafter with 4 layers and a block size of 9. We ran 10 epochs on the final mixture and measured acceptance after each, which improved with additional training tokens before reaching diminishing returns. At inference time, we recommend a block size of 8 or 9 depending on the hardware. The resulting drafter has approximately 280M parameters and increases the deployed model’s parameter count by just 8.9%. 我们遵循 DSpark 的配方，使用混合的视觉语言 SFT 数据进行训练，并根据我们预期模型服务的负载进行加权。基于对 3 层、4 层和 5 层架构的消融实验，该草稿模型被设计为仅包含注意力机制的简化模型，拥有 4 层结构和 9 的块大小 (block size)。我们在最终混合数据集上运行了 10 个 epoch，并在每个 epoch 后测量接受率，结果显示随着训练 token 的增加，接受率有所提升，随后达到边际效应递减点。在推理时，我们建议根据硬件情况选择 8 或 9 的块大小。最终的草稿模型拥有约 2.8 亿参数，仅使部署模型的参数量增加了 8.9%。
| Component | Parameters |
|---|---|
| LFM2.5-VL-3B Decoder stack (4 layers) | 193.0M |
| Hidden-state projection | 21.0M |
| Markov head | 65.5M |
| Norms + confidence head | 6.4k |
| Total | 279.5M |
| 组件 | 参数量 |
|---|---|
| LFM2.5-VL-3B 解码器堆栈 (4 层) | 1.93 亿 |
| 隐藏状态投影 | 2100 万 |
| 马尔可夫头 (Markov head) | 6550 万 |
| 归一化 + 置信度头 | 6400 |
| 总计 | 2.795 亿 |
Inference Speedup on CPU and GPU
CPU 和 GPU 上的推理加速
The DSpark draft model for LFM2.5-VL-3B ships with day-one support for llama.cpp, MLX-VLM, and SGLang. We measure both on-device inference and GPU inference. Both configurations use a DSpark block size of 8 and are evaluated on six diverse vision-based tasks, including general VQA, text VQA, image captioning, chart VQA, complex reasoning, and multi-turn conversation, following the MMSpec benchmark. LFM2.5-VL-3B 的 DSpark 草稿模型在发布首日即支持 llama.cpp、MLX-VLM 和 SGLang。我们对设备端推理和 GPU 推理进行了测量。两种配置均使用 8 的 DSpark 块大小，并在遵循 MMSpec 基准测试的六项不同视觉任务上进行了评估，包括通用 VQA、文本 VQA、图像描述、图表 VQA、复杂推理和多轮对话。
- On-device inference: With MLX on an M5 Max, decoding runs 2.30x to 3.13x faster by task. End-to-end latency improves by 1.56x to 2.62x. With llama.cpp on an M3 Ultra, decoding improves by 1.57x to 2.14x and end-to-end by 1.30x to 1.77x.
- GPU inference: On H100, the same drafter delivers 20.4x to 2.66x faster decoding, with end-to-end improvements of 1.64x to 2.27x.
- 设备端推理： 在 M5 Max 上使用 MLX，解码速度根据任务不同提升 2.30 倍至 3.13 倍。端到端延迟提升 1.56 倍至 2.62 倍。在 M3 Ultra 上使用 llama.cpp，解码速度提升 1.57 倍至 2.14 倍，端到端延迟提升 1.30 倍至 1.77 倍。
- GPU 推理： 在 H100 上，相同的草稿模型可实现 2.04 倍至 2.66 倍的解码加速，端到端性能提升 1.64 倍至 2.27 倍。
Limitations of speculation for vision workloads
视觉负载下推测解码的局限性
In LLMs, prefill is mostly compute-bound, and its cost grows (sub)quadratically with prompt length. VLMs add to this because the image first passes through a vision encoder, then the language backbone processes hundreds of visual tokens along with the text prompt. Edge devices have far less compute than datacenter GPUs, so prefill takes up more of the end-to-end latency. Speculative decoding speeds up only decode, not vision encoding or prefill. When those stages already take up much of the wall time, even a large decode speedup gives only a modest end-to-end gain. This is Amdahl’s law, where the overall speedup is capped by the part of the workload that isn’t accelerated. 在大型语言模型 (LLM) 中，预填充 (prefill) 主要受计算限制，其成本随提示词长度呈（次）二次方增长。视觉语言模型 (VLM) 增加了这一负担，因为图像首先需要通过视觉编码器，然后语言主干网络需要处理数百个视觉 token 以及文本提示词。边缘设备的计算能力远低于数据中心 GPU，因此预填充占用了更多的端到端延迟。推测解码仅能加速解码过程，无法加速视觉编码或预填充。当这些阶段已经占据了大部分运行时间时，即使解码速度大幅提升，端到端的增益也十分有限。这就是阿姆达尔定律 (Amdahl’s law)，即整体加速比受限于未被加速的那部分工作负载。
How to use LFM2.5-VL-DSpark
如何使用 LFM2.5-VL-DSpark
(Technical implementation details omitted for brevity, please refer to the original article for specific command-line arguments.) (为简洁起见，此处省略技术实现细节，具体命令行参数请参考原文。)
Get Started
开始使用
Our vision DSpark draft model is available on Hugging Face in Safetensors and GGUF formats. With LFM2.5, we’re delivering on our vision of AI that runs anywhere. These models are: 我们的视觉 DSpark 草稿模型已在 Hugging Face 上提供，包含 Safetensors 和 GGUF 格式。通过 LFM2.5，我们正在实现让 AI 在任何地方运行的愿景。这些模型具有以下特点：
- Open-weight: Download, fine-tune, and deploy without restrictions.
- Fast from day one: Day-one support for llama.cpp, MLX, and SGLang.
- A complete family: From base models for customization to specialized audio and vision variants, one architecture covers diverse use cases.
- 开放权重： 可无限制下载、微调和部署。
- 首日即快： 首日支持 llama.cpp、MLX 和 SGLang。
- 完整系列： 从用于定制的基础模型到专门的音频和视觉变体，单一架构即可覆盖多种用例。
We can’t wait to see what you build. 我们迫不及待地想看到你们的创作。