Native-speed vLLM transformers modeling backend

查看原文 / View Original 编辑 / Edit

Native-speed vLLM transformers modeling backend

原生速度的 vLLM transformers 模型后端

TL;DR: The transformers vLLM backend is now as fast (or faster) than custom vLLM implementations for many LLM architectures. Model authors can automatically leverage their transformers implementations to get ultra fast vLLM inference, for free. 简而言之：对于许多大语言模型（LLM）架构而言，transformers vLLM 后端现在的运行速度已经达到（甚至超过）了 vLLM 的自定义实现。模型作者可以自动利用其 transformers 实现，免费获得超高速的 vLLM 推理性能。

Upgrade the vllm pip package

升级 vllm pip 包

uv pip install --upgrade vllm --torch-backend auto

The transformers library has become the reference modeling library for Machine Learning. It supports 450+ architectures through consistent APIs, and is designed with the main goal that model implementations are self contained and easy to understand. Going through transformers code makes it easy for contributors to learn how an architecture works, and then port it to other frameworks such as vLLM, SGLang, MLX, llama.cpp, and many others. Transformers 库已成为机器学习领域的参考建模库。它通过一致的 API 支持 450 多种架构，其核心设计目标是确保模型实现是自包含且易于理解的。通过阅读 transformers 的代码，贡献者可以轻松了解架构的工作原理，并将其移植到 vLLM、SGLang、MLX、llama.cpp 等其他框架中。

We have fully embraced this role in the ecosystem and are investing a lot of effort to make it easier. A big step in this direction was the integration last year of transformers as a modeling backend in vLLM. This has been allowing model authors to run transformers models (LLMs and VLMs alike) inside vLLM, without having to port anything. Transformers provides the modeling code, and vLLM provides extremely optimized inference techniques such as continuous batching and custom attention kernels. This integration gets better now 🚀! 我们已完全拥抱这一生态角色，并投入大量精力使其变得更加简单。去年将 transformers 集成为 vLLM 的建模后端是朝着这个方向迈出的重要一步。这使得模型作者无需进行任何移植，即可在 vLLM 中运行 transformers 模型（包括 LLM 和 VLM）。Transformers 提供建模代码，而 vLLM 提供诸如连续批处理（continuous batching）和自定义注意力算子（custom attention kernels）等经过深度优化的推理技术。现在，这一集成变得更加强大了 🚀！

Showcase

展示

We put the transformers modeling backend for vLLM head to head with vLLM’s hand written native implementations across three very different Qwen3 models: 我们选取了三个截然不同的 Qwen3 模型，将 vLLM 的 transformers 建模后端与 vLLM 手写的原生实现进行了对比：

  • 4B dense model on a single GPU
  • 单 GPU 上的 4B 稠密模型
  • 32B dense model on tensor parallelism
  • 使用张量并行（tensor parallelism）的 32B 稠密模型
  • 235B-parameter FP8 Mixture-of-Experts on data + expert parallelism on the same 8×H100 node
  • 在同一 8×H100 节点上使用数据并行 + 专家并行（data + expert parallelism）的 235B 参数 FP8 混合专家模型（MoE）

The result: the transformers modeling backend now meets or beats native throughput on every one of them. 结果显示：transformers 建模后端在上述所有模型上的吞吐量均已达到或超过了原生实现。

Running any* Hugging Face model through the transformers modeling backend is a single flag — --model-impl transformers. It composes with the usual parallelism options, so nothing about your serving setup changes: 通过 transformers 建模后端运行任何* Hugging Face 模型只需一个标志 —— --model-impl transformers。它与常规的并行选项兼容，因此你的服务部署配置无需任何更改：

# Qwen3-4B dense, single GPU
# Qwen3-4B 稠密模型，单 GPU
vllm serve Qwen/Qwen3-4B --model-impl transformers

# Qwen3-32B dense, tensor-parallel across 2 GPUs
# Qwen3-32B 稠密模型，跨 2 个 GPU 的张量并行
vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2

# Qwen3-235B-A22B-FP8 MoE, data-parallel + expert-parallel across 8 GPUs
# Qwen3-235B-A22B-FP8 MoE 模型，跨 8 个 GPU 的数据并行 + 专家并行
vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel
# add --max-model-len 8192 if your node is memory constrained
# 如果你的节点内存受限，请添加 --max-model-len 8192

*Models that use linear attention are not currently supported, but they will be soon! Custom models where the code lives in a Hub repo are unlikely to work as they will not have been written compliantly. *目前尚不支持使用线性注意力的模型，但很快就会支持！代码位于 Hub 仓库中的自定义模型可能无法工作，因为它们可能未按规范编写。

How we measured

测量方法

Each model is compared under three conditions that are identical in every way except the code path: 每个模型都在三种条件下进行比较，除了代码路径外，其他所有条件均相同：

  • native — --model-impl vllm, vLLM’s hand-written model (the bar to match)
  • native — --model-impl vllm，vLLM 手写的模型（作为基准）
  • after — --model-impl transformers with the PR
  • after — --model-impl transformers（包含本次 PR）
  • before — --model-impl transformers without the PR
  • before — --model-impl transformers（不包含本次 PR）

The full, reproducible runner is available as a gist: benchmark.sh 完整的、可复现的运行脚本可在 gist 中获取：benchmark.sh

So, what’s new?

那么，有什么新变化？

The transformers modeling backend for vLLM used to focus on attention as the bottleneck for inference. By plugging vLLM’s attention implementation at runtime, we could make a transformers model run efficiently inside the vLLM engine. But there are many dimensions to deployments that only a custom port can target to extract maximum inference performance. Parallelization across GPUs, compilation, fused kernels, and many more, all contribute to leveraging your hardware to achieve ultra-fast inference. vLLM 的 transformers 建模后端过去主要关注将注意力机制作为推理瓶颈。通过在运行时插入 vLLM 的注意力实现，我们可以使 transformers 模型在 vLLM 引擎中高效运行。但部署过程中还有许多维度，只有通过自定义移植才能挖掘出最大的推理性能。跨 GPU 并行化、编译、融合算子（fused kernels）等，都有助于充分利用硬件以实现超高速推理。

A new model used to be integrated once for transformers, and once for vLLM with custom optimizations. When model authors wanted the absolute best performance, they were still writing custom vLLM implementations. 过去，一个新模型需要分别针对 transformers 和 vLLM 进行集成，并进行自定义优化。当模型作者追求极致性能时，他们仍然需要编写自定义的 vLLM 实现。

A new model once integrated to transformers, can now be immediately used in vLLM with native vLLM implementation speed. 现在，一个新模型一旦集成到 transformers 中，就可以立即在 vLLM 中使用，并获得与 vLLM 原生实现相同的速度。

The latest iteration of the transformers modeling backend for vLLM dynamically applies inference specific layer fusions at runtime to match the speed of custom code implementations, for compatible architectures. vLLM 的 transformers 建模后端最新版本在运行时动态应用推理特定的层融合（layer fusions），从而使兼容架构能够达到与自定义代码实现相同的速度。

How does it work?

它是如何工作的？

The transformers modeling backend for vLLM now uses torch.fx to perform static analysis on the model’s graph. This process searches for known patterns that can be optimised. After any patterns have been identified, it uses ast (abstract syntax tree) to manipulate the source code and rewrite some of the operations in place. vLLM 的 transformers 建模后端现在使用 torch.fx 对模型图进行静态分析。此过程会搜索可优化的已知模式。识别出模式后，它会使用 ast（抽象语法树）来操作源代码，并就地重写部分操作。

What can we achieve with this?

我们能实现什么？

Fused operations that are many-to-one mapped to (ultra) optimized vLLM kernels, such as the ones used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models. The main other fused operations are vLLM’s MergedColumnParallelLinear and QKVParallelLinear. These blocks allow us to infer parallel plans for TP (tensor-parallel). PP (pipeline-parallel) plans can also be inferred if the decoder block list is easily identifiable. 可以将多个操作融合映射为（超）优化的 vLLM 算子，例如混合专家模型（MoE）中用于专家并行（EP）的算子。其他主要的融合操作包括 vLLM 的 MergedColumnParallelLinear 和 QKVParallelLinear。这些模块使我们能够推断出 TP（张量并行）的并行计划。如果解码器块列表易于识别，也可以推断出 PP（流水线并行）计划。

The manipulated models are still fully (torch) compilable, being passed through torch.compile and CUDA Graphs, just the same as a dedicated vLLM model implementation. Unlike vLLM model implementations, Transformers model implementations can be used in training. So you can use the same model code for training/evals/RL rollouts. 经过处理的模型仍然完全支持（torch）编译，可以通过 torch.compile 和 CUDA Graphs 运行，与专门的 vLLM 模型实现完全相同。与 vLLM 模型实现不同，Transformers 模型实现可用于训练。因此，你可以使用相同的模型代码进行训练、评估和强化学习（RL）部署。

As shown above, this results in native vLLM inference speed for compatible models, without having to write a single line of code to optimize the model for inference. We are in the process of writing a detailed blog post to dive deep inside these optimized inference methods and explain in detail how we manipulate the model to adapt to them. 如上所述，这使得兼容模型能够获得原生的 vLLM 推理速度，而无需编写任何一行代码来优化模型推理。我们正在撰写一篇详细的博客文章，深入探讨这些优化推理方法，并详细解释我们如何操作模型以适应这些方法。

← 返回新闻列表 / Back to news list