Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

查看原文 / View Original 编辑 / Edit

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

在没有 GPU 的 13 年前至强（Xeon）服务器上以每秒 5 个 token 的速度运行 Gemma 4 26B

There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, roughly thirteen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google’s Gemma 4, a 26-billion-parameter open-weights mixture-of-experts model, at about five tokens per second. Reading speed.

我地下室里有一台服务器，按理说它根本不该运行现代语言模型。这是一台重新利用的 HP StoreVirtual 存储设备，大约有十三年历史，配备两颗 Ivy Bridge 架构的至强处理器，没有 GPU。它是为存储磁盘而设计的，而不是为了进行数学运算。但从本周开始，它能够以每秒约 5 个 token 的速度运行谷歌的 Gemma 4——一个拥有 260 亿参数的开放权重混合专家（MoE）模型。这已经是阅读速度了。

Hardware

硬件配置

  • Repurposed HP StoreVirtual: dual Xeon E5-2690 v2 (Ivy Bridge, 2013), DDR3, no GPU

  • Instruction sets: AVX1 only — no AVX2, no FMA3

  • Model: Gemma 4 26B-A4B (MoE), Q8_0

  • Decode: ~5.2 tokens/sec

  • Prompt eval: ~16 tokens/sec

  • Cost of the box: under $300

  • 重新利用的 HP StoreVirtual： 双路 Xeon E5-2690 v2（Ivy Bridge 架构，2013 年），DDR3 内存，无 GPU

  • 指令集： 仅支持 AVX1 — 不支持 AVX2，不支持 FMA3

  • 模型： Gemma 4 26B-A4B (MoE), Q8_0 量化

  • 解码速度： ~5.2 token/秒

  • 提示词评估： ~16 token/秒

  • 设备成本： 低于 300 美元

Anybody can rent a GPU. It’s harder to take a modern MoE model and a dead enterprise box and make them meet in the middle, and that gap is the whole reason I’m writing this up. “Good with AI” has quietly come to mean “pays for a subscription.” I think the real skill is different: knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is actually correct. So rather than claim we’re good at this, here’s a worked example, on hardware that had no business cooperating.

任何人都可以租用 GPU。但要将现代 MoE 模型与一台过时的企业级设备结合起来，并让它们协同工作则要困难得多，而这种差距正是我写这篇文章的原因。“擅长 AI”在不知不觉中已经变成了“付得起订阅费”。我认为真正的技能在于：足够了解一个模型，从而将其指向一个无人为你打包好的问题，并能判断它给出的答案是否真正正确。因此，与其自夸，不如看看这个在“本不该合作”的硬件上实现的案例。

The post that started it

启发我的文章

A couple of weeks ago a piece called “A 10 year old Xeon is all you need” made the rounds on Hacker News. The author runs Gemma 4 on a single 2016 Xeon with no GPU and 128 GB of slow DDR3, using ik_llama.cpp and about 25 carefully chosen flags. It’s a great read, and it leans on every trick in the modern inference playbook: speculative decoding, CPU-aware mixture-of-experts routing, flash attention ported to the CPU, run-time weight repacking. Real engineering.

几周前，一篇名为《你只需要一台 10 年前的至强服务器》的文章在 Hacker News 上流传。作者使用 ik_llama.cpp 和大约 25 个精心挑选的参数，在一台 2016 年的至强处理器上运行 Gemma 4，没有 GPU，只有 128GB 的慢速 DDR3 内存。这是一篇很棒的文章，它利用了现代推理手册中的每一个技巧：推测解码、CPU 感知的混合专家路由、移植到 CPU 的 Flash Attention 以及运行时权重重打包。这才是真正的工程技术。

“I have a Xeon too,” I thought. Several, in fact. So I tried it. It didn’t run.

“我也有至强处理器，”我想。事实上，我有好几台。于是我尝试了一下。结果没跑起来。

What an AI agent is actually good for

AI 智能体真正的用处

The build died on startup. I handed the failure to Claude and asked what was wrong. The answer came back fast and specific. The author’s 2016 chip is a Broadwell part. Mine are Ivy Bridge, the generation Intel calls “v2.” The fast kernels in that fork assume AVX2 and FMA3, instruction sets that didn’t ship until Haswell, the “v3” generation, in 2014. My CPUs are older than the instructions the code was written against. The optimized paths weren’t there to execute.

编译在启动时就失败了。我把错误信息交给 Claude，询问出了什么问题。答案很快且非常具体：作者 2016 年的芯片是 Broadwell 架构，而我的是 Ivy Bridge，即英特尔所谓的“v2”代。该分支中的快速内核假设支持 AVX2 和 FMA3，这些指令集直到 2014 年 Haswell（“v3”代）才发布。我的 CPU 比代码所依赖的指令集还要老，因此根本没有优化的路径可供执行。

So I asked the obvious follow-up: can we make it run anyway? I’d already taken a first swing with a free model that got close but couldn’t land it. Claude picked up that half-finished approach, agreed it was the right one, and finished it off, reworking the hot paths so they fall back cleanly on a pre-AVX2 chip instead of reaching for instructions that aren’t there.

于是我问了一个显而易见的问题：我们能让它运行起来吗？我之前已经尝试过一次，用一个免费模型接近了目标，但没能成功。Claude 接手了这个未完成的方法，确认它是正确的方向，并将其完成。它重构了热点路径，使其在没有 AVX2 的芯片上能平滑回退，而不是去调用不存在的指令。

This is the part I care about. This didn’t come from typing “fix it” once and getting a working patch back. Somebody had to read another person’s performance-critical C++, work out why a kernel wasn’t valid on this particular microarchitecture, and route around it without throwing away the optimizations that made the fork worth using. Claude did that work. My job was narrower: run the right experiments and recognize when the output was finally correct. I came away impressed.

这正是我所关心的部分。这并不是通过输入一次“修复它”就能得到补丁那么简单。必须有人阅读他人的高性能 C++ 代码，弄清楚为什么内核在特定的微架构上无效，并在不丢弃使该分支具有价值的优化前提下绕过它。Claude 完成了这项工作。我的工作范围更窄：运行正确的实验，并识别输出何时终于正确了。我对此印象深刻。

The result

结果

Gemma 4’s 26B mixture-of-experts model now generates text at reading speed on hardware that was retired before the model’s architecture existed. The original write-up never published a tokens-per-second figure, just “reading speed,” so here’s the concrete one: about five tokens a second on thirteen-year-old silicon, for borderline free.

Gemma 4 的 26B 混合专家模型现在可以在一台在该模型架构诞生前就已经退役的硬件上以阅读速度生成文本。最初的文章没有发布每秒 token 数，只提到了“阅读速度”，所以这里有一个具体数字：在 13 年前的硅片上，以近乎免费的成本，达到了每秒约 5 个 token。

Proof it runs: Gemma 4 26B answering on the basement box, CPU-only. The patch is up as ikawrakow/ik_llama.cpp#2138 if you want the exact diff — still open and awaiting maintainer review as I write this, so run it from the branch for now. The hope is that anyone else sitting on ancient enterprise iron can keep a local model around: a fallback for when the paid APIs are down, or a cheap way to grind through slow batch jobs when paying per token doesn’t make sense.

运行证明：Gemma 4 26B 在地下室服务器上回答问题，仅使用 CPU。如果你想要确切的补丁，它已提交至 ikawrakow/ik_llama.cpp#2138 —— 在我撰写本文时，它仍处于开放状态并等待维护者审核，所以目前请从该分支运行。希望其他拥有古老企业级硬件的人也能在本地运行模型：作为付费 API 宕机时的备选方案，或者在按 token 付费不划算时，作为处理缓慢批处理任务的廉价方式。

← 返回新闻列表 / Back to news list