Serving Gemma4 with Rust on vLLM 🦀
Serving Gemma4 with Rust on vLLM 🦀
This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM’s Rust frontend (vllm-rs) built, running, and verified. This paper is a follow-on to the original G5g Gemma 4 build. Everything below was run on the box.
本教程将引导您在 AWS EC2 G5g 实例（搭载 NVIDIA T4G GPU 的 Graviton2 (aarch64)）上安装并配置 vLLM 的 Rust 工具链，并完成 vLLM Rust 前端 (vllm-rs) 的构建、运行与验证。本文是之前 G5g Gemma 4 构建指南的后续内容。以下所有操作均在该实例上完成。
🦀 Wait, vLLM has Rust in it? You betcha. Since PR #40848 (merged 2026-05-21), vLLM vendors a 14-crate Rust workspace: bench, chat, cmd, engine-core-client, llm, managed-engine, metrics, mock-engine, parser, parser/python, server, text, tokenizer, tracing. Edition 2024, resolver 3.
🦀 等等，vLLM 里有 Rust？没错。自 PR #40848（于 2026 年 5 月 21 日合并）以来，vLLM 引入了一个包含 14 个 crate 的 Rust 工作区：bench、chat、cmd、engine-core-client、llm、managed-engine、metrics、mock-engine、parser、parser/python、server、text、tokenizer 和 tracing。采用 Edition 2024 和 resolver 3。
Straight from the vendored rust/Cargo.toml: 直接查看 vendored 的 rust/Cargo.toml：
| Crate | Version | Job |
|---|---|---|
| axum | 0.8.8 | the HTTP server |
| tokio | 1.47.1 | async runtime |
| zeromq | 0.6.0 | talks to the Python engine |
| rmp-serde / rmpv | 1.3.1 | msgpack on the wire |
| minijinja | 2.22 | chat templates |
| tonic / prost | 0.14.6 / 0.14.3 | gRPC — remember this one |
| Crate | 版本 | 作用 |
|---|---|---|
| axum | 0.8.8 | HTTP 服务器 |
| tokio | 1.47.1 | 异步运行时 |
| zeromq | 0.6.0 | 与 Python 引擎通信 |
| rmp-serde / rmpv | 1.3.1 | 网络传输中的 msgpack |
| minijinja | 2.22 | 聊天模板 |
| tonic / prost | 0.14.6 / 0.14.3 | gRPC — 请记住这个 |
It’s a drop-in replacement for the Python FastAPI server. Two artifacts get built: 它是 Python FastAPI 服务器的直接替代品。构建过程会生成两个产物：
-
🦀 vllm-rs — the axum frontend binary
-
🐍 vllm._rust_tool_parser — a PyO3 extension module
-
🦀 vllm-rs — axum 前端二进制文件
-
🐍 vllm._rust_tool_parser — PyO3 扩展模块
Rust is a build requirement now
Rust 现在是构建的必要条件
That’s the headline, and it’s reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, unguarded: from setuptools_rust.build import build_rust. No try, no feature flag, no opt-out. Metadata generation doesn’t happen without it.
这是核心要点，仅此一点就足够了：在 v0.27.2rc0 版本中，如果没有 Rust，你将无法从源码构建 vLLM。setup.py 在第 21 行的模块作用域内直接导入了它，且没有任何保护措施：from setuptools_rust.build import build_rust。没有 try-except，没有功能开关，也没有退出选项。没有它，元数据生成就无法完成。
And this isn’t a quirk of one release. vLLM’s Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes — so it’s worth knowing how to drive it properly rather than working around it.
这并非某个版本的特例。vLLM 的 Rust 覆盖面包括 14 个 crate，涵盖了 HTTP 前端、工具解析器、分词器和基准测试客户端，并且自引入以来一直在增长。如果你从源码构建推理基础设施，Rust 工具链正成为“入场券”——因此，学会如何正确使用它，而不是绕过它，是非常值得的。
Three things do get conflated, though, and they have different scopes: 不过，有三件事容易混淆，它们的作用域各不相同：
| Component | Needed to build vLLM? | Needed to serve? |
|---|---|---|
| setuptools_rust (Python pkg) | yes, always | no |
| cargo / rustc toolchain | for working Rust artifacts | no |
| protoc | for vllm-rs specifically | no |
| 组件 | 构建 vLLM 需要吗？ | 服务运行需要吗？ |
|---|---|---|
| setuptools_rust (Python 包) | 是，必须 | 否 |
| cargo / rustc 工具链 | 用于处理 Rust 产物 | 否 |
| protoc | 专门用于 vllm-rs | 否 |
Then why doesn’t pip install vllm need this? Because normally pip installs it for you. pyproject.toml declares it:
那么为什么 pip install vllm 不需要这些呢？因为通常 pip 会为你安装。pyproject.toml 中声明了它：
[build-system]
requires = [
"cmake>=3.26.1",
"ninja",
"packaging>=24.2",
"setuptools>=77.0.3,<81.0.0",
"setuptools-scm>=8.0",
"setuptools-rust>=1.9.0", # <- pip grabs this automatically
"torch == 2.13.0", # <- ...and this. Which is the problem.
"wheel",
"jinja2",
]Under normal build isolation, pip creates a clean env, installs that list, and builds. You never see setuptools_rust because you never had to think about it. But look at the torch pin. Building in isolation means pip installs torch 2.13.0 from PyPI — and the PyPI aarch64 wheels are built for sm_80 and up. No sm_75. Which destroys the entire reason for building from source on a T4G.
在正常的构建隔离环境下，pip 会创建一个干净的环境，安装上述列表并进行构建。你从未见过 setuptools_rust，因为你根本不需要考虑它。但请看 torch 的版本锁定。在隔离环境下构建意味着 pip 会从 PyPI 安装 torch 2.13.0，而 PyPI 的 aarch64 wheel 是为 sm_80 及以上架构构建的，不支持 sm_75。这彻底破坏了在 T4G 上从源码构建的初衷。
So on this box you must build against the DLAMI’s own torch, and that means: 因此，在这台机器上，你必须针对 DLAMI 自带的 torch 进行构建，这意味着：
python use_existing_torch.py
pip install -e . --no-build-isolation--no-build-isolation turns off the automatic install of everything in that requires list. From that moment on, every build dependency is yours to supply by hand — including setuptools_rust, which is why it turns up as a bare ModuleNotFoundError minutes into a build that has nothing visibly to do with Rust. So the toolchain was always required; isolation was just hiding it. Building this way means you own the dependency list, which is the rest of this walk-through.
--no-build-isolation 会关闭对 requires 列表中所有内容的自动安装。从那一刻起，每一个构建依赖项都需要你手动提供——包括 setuptools_rust。这就是为什么在构建开始几分钟后，即使看起来与 Rust 无关，也会出现 ModuleNotFoundError 的原因。所以工具链一直都是必需的，只是隔离环境把它隐藏了起来。以这种方式构建意味着你需要自行管理依赖列表，这也是本指南后续部分的内容。
⚡ What the DLAMI gives you, and what it doesn’t
⚡ DLAMI 提供了什么，没提供什么
The AWS Deep Learning ARM64 AMI ships a runtime, not a build environment. On a fresh box: AWS Deep Learning ARM64 AMI 提供的是运行时环境，而非构建环境。在全新的实例上：
| Thing | Present? |
|---|---|
| PyTorch 2.12 with sm_75 | ✅ |
| NVIDIA driver | ✅ |
| nvcc / CUDA toolkit | ❌ |
| Rust toolchain | ❌ |
| setuptools_rust | ❌ |
| protoc | ❌ |
| 项目 | 是否存在？ |
|---|---|
| PyTorch 2.12 (含 sm_75) | ✅ |
| NVIDIA 驱动 | ✅ |
| nvcc / CUDA 工具包 | ❌ |
| Rust 工具链 | ❌ |
| setuptools_rust | ❌ |
| protoc | ❌ |
Four of those six are on you. Let’s install them. 这六项中有四项需要你自己处理。让我们开始安装。
Step 1 — Rust itself
第一步 — 安装 Rust
Standard rustup, nothing aarch64-specific about it:
使用标准的 rustup，没有任何针对 aarch64 的特殊要求：
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
stable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 (8bab26f4f 2026-07-14)Rust is installed now. Great! Note the triple: stable-aarch64-unknown-linux-gnu. Rust’s aarch64 support is a complete non-event, which is a lovely change of pace on this hardware. ⚡
Rust 现在安装好了。太棒了！注意目标三元组：stable-aarch64-unknown-linux-gnu。Rust 对 aarch64 的支持非常成熟，这在这种硬件上是一种令人愉悦的体验。⚡
Step 2 — setuptools-rust
第二步 — 安装 setuptools-rust
python3 -m pip install setuptools_rustPer the section above: --no-build-isolation means pip won’t do this for you. Do it early — the failure lands during metadata generation, minutes into a build, as a bare ModuleNotFoundError: No module named 'setuptools_rust' nowhere near anything that looks like Rust. ⚠️ Install it into the same interpreter you’ll build with. On the DLAMI that’s /opt/pytorch/bin/python3, not the system python3 — they’re different, and the one that matters is whichever owns the torch you’re building against.
根据前文所述：--no-build-isolation 意味着 pip 不会为你安装它。请尽早安装——否则构建几分钟后在元数据生成阶段会报错 ModuleNotFoundError: No module named 'setuptools_rust'，而此时构建过程看起来还完全没触及 Rust。⚠️ 请将其安装到你构建时使用的同一个解释器中。在 DLAMI 上，这是 /opt/pytorch/bin/python3，而不是系统自带的 python3——它们是不同的，关键在于你要使用与你所构建的 torch 关联的那个解释器。
Step 3 — protoc
第三步 — 安装 protoc
🔎 This is the one nobody documents: 🔎 这是没人记录的一点：
apt-get install -y protobuf-compiler
protoc --version
# libprotoc 3.21.12Why: vllm-rs depends on the vllm-server crate, vllm-server builds gRPC stubs with tonic/prost, and prost-build shells out to protoc. Skip it and the frontend binary does not get built — see the summary at the end for how loudly that doesn’t fail. The tool parser has no protobuf dependency, which is why it builds either way.
原因：vllm-rs 依赖于 vllm-server crate，vllm-server 使用 tonic/prost 构建 gRPC 存根，而 prost-build 需要调用 protoc。如果跳过这一步，前端二进制文件将无法构建——请参阅文末总结，了解这种失败是多么“静默”。工具解析器没有 protobuf 依赖，这就是为什么无论如何它都能构建成功。
Step 4 — the CUDA toolkit, while you’re here
第四步 — 安装 CUDA 工具包
Not Rust, but the same class of problem, and you need it for vLLM’s kernels: 虽然不是 Rust，但属于同一类问题，而且 vLLM 的内核需要它：
# NVIDIA's **sbsa** repo — not the x86 one, easy reflex to get wrong on Arm
apt-get install -y cuda-toolkit-13-2Step 5 — build the Rust artifacts
第五步 — 构建 Rust 产物
cd /opt/vllm-src
python tools/build_rust.py --release⚠️ Do not omit --release. setuptools-rust builds inplace targets in debug by default, and pip install -e . is an inplace build. The difference is not subtle:
⚠️ 不要省略 --release。setuptools-rust 默认在 debug 模式下构建原地目标，而 pip install -e . 就是一种原地构建。两者的区别非常显著：
| Artifact | Debug | Release |
|---|---|---|
_rust_tool_parser.abi3.so | 100,913,216 B | 1,009,080 B |
| 产物 | Debug | Release |
|---|---|---|
_rust_tool_parser.abi3.so | 100,913,216 字节 | 1,009,080 字节 |
100x. The debug artifact is four times the size of every CUDA kernel in vLLM combined. Timing on a g5g.xlarge (4 vCPU), cold:
100 倍的差距。Debug 产物的大小是 vLLM 中所有 CUDA 内核总和的四倍。在 g5g.xlarge (4 vCPU) 上冷启动构建耗时：
real 9m1.746s
user 25m9.199s
sys 1m35.023s501 crates. Zero warnings. Exit 0. 🟢 Rust’s aarch64 support does not put up a fight here — which is a pleasant contra
501 个 crate。零警告。退出代码 0。🟢 Rust 对 aarch64 的支持在这里没有造成任何阻碍——这是一种令人愉悦的对比。