Qwenputer: A functioning, deterministic RISC-V CPU inside Qwen3.5

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Qwenputer is a pretty fun experimental model. It’s a stock Qwen 3.5 4B with one addition: a RISC-V CPU embedded within the weights that uses the model’s own Gated DeltaNet attention layers as RAM. (To learn how GDN works, here’s a handy guide.) This ‘CPU’ runs completely within the forward passes of the model, and can run arithmetic operations, C and MicroPython code without any external runtime or tool usage whatsoever.

Qwenputer 是一个非常有趣的实验性模型。它是一个标准的 Qwen 3.5 4B 模型，但增加了一个嵌入在权重中的 RISC-V CPU，该 CPU 利用模型自身的门控 DeltaNet（Gated DeltaNet）注意力层作为内存。（若要了解 GDN 的工作原理，这里有一份便捷指南。）这个“CPU”完全在模型的前向传播过程中运行，无需任何外部运行时或工具，即可执行算术运算、C 语言和 MicroPython 代码。

To be very clear, it is not a plugin, a tool or a sandbox. This blog will explain exactly how this model works, and how the model utilises GDN layers as RAM, and how I also slotted in an experimental in-model file storage.

需要明确的是，它不是插件、工具或沙箱。本博客将详细解释该模型的工作原理、模型如何利用 GDN 层作为内存，以及我是如何嵌入一个实验性的模型内文件存储系统的。

I also wrote up two follow-ups built on top of Qwenputer: how hybrid models can learn to hide their reasoning in their RNN state and evade CoT monitoring, and how compiled weights like these can work as inbuilt monitoring tools that the model itself can’t tamper with.

我还基于 Qwenputer 撰写了两篇后续文章：探讨混合模型如何学习将其推理过程隐藏在 RNN 状态中以规避思维链（CoT）监控，以及这些编译后的权重如何作为模型自身无法篡改的内置监控工具发挥作用。

You can grab Qwenputer from Hugging Face or ModelScope, and the code is on GitHub.

你可以从 Hugging Face 或 ModelScope 获取 Qwenputer，代码已托管在 GitHub 上。

Qwen3-Next and Qwen 3.5 (the model Qwenputer is built on) use a pretty neat trick. 3 out of 4 attention layers in the model use Gated DeltaNet, a linear RNN, as the attention mechanism (more on that here). Each head keeps a state matrix, and the delta rule keeps this updated.

Qwen3-Next 和 Qwen 3.5（Qwenputer 所基于的模型）使用了一个非常巧妙的技巧。模型中 4 分之 3 的注意力层使用门控 DeltaNet（一种线性 RNN）作为注意力机制（更多信息见此处）。每个头都维护一个状态矩阵，并通过增量规则（delta rule）进行更新。

Now, here’s where our fun experiment starts. I found out that by setting the gate to 1,11 Each Gated DeltaNet head keeps a state matrix, its memory. At every token, before writing anything new, it multiplies the whole state by the gate, a number between 0 and 1. That’s how the RNN forgets: a gate of 0.9 fades everything stored a little on every token, and a gate of 1 means nothing ever fades. the write strength (beta) to 1,22 After the gate, the head nudges one direction of the state towards a new value, and the write strength beta sets how far. A beta of 1 means the old value along that direction is replaced completely instead of blended. and using one-hot key encoding,33 A one-hot key is a vector of zeros with a single 1 in it, so it points at exactly one row of the state, the way a memory address points at one cell. every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value. This means that the GDN layers themselves can work exactly as RAM, at a far lower cost than using normal attention as RAM, since a GDN head’s state stays the same size however long the conversation gets, unlike attention’s KV cache.

现在，有趣的实验开始了。我发现通过将门控值设为 1，每个门控 DeltaNet 头都会保留一个状态矩阵作为其内存。在处理每个 token 时，在写入新内容之前，它会将整个状态乘以门控值（一个 0 到 1 之间的数字）。这就是 RNN 遗忘的方式：0.9 的门控值会让存储的内容在每个 token 处理时逐渐衰减，而 1 则意味着内容永不衰减。将写入强度（beta）设为 1 后，在门控之后，该头会将状态的一个方向推向新值，而写入强度 beta 决定了推移的程度。beta 为 1 意味着该方向上的旧值会被完全替换而非混合。通过使用独热编码（one-hot key encoding），即一个包含单个 1 的零向量，它能精确指向状态的一行，就像内存地址指向一个单元格一样。我们执行的每次写入都会覆盖精确的一行，而每次独热查询都能检索到一个精确的值。这意味着 GDN 层本身可以完全作为内存使用，且成本远低于将普通注意力机制用作内存，因为与注意力的 KV 缓存不同，GDN 头的状态大小在对话过程中保持不变。

The idea that the delta rule acts as a memory isn’t new: Schlag et al. (2021) showed that linear attention with the delta rule works as an associative memory, or “fast weights”.44 Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers”, ICML 2021. The new thing here is making it exact and addressable, inside a production model, reliably enough to run a whole CPU on it.

增量规则充当内存的想法并不新鲜：Schlag 等人（2021）证明了带有增量规则的线性注意力机制可以作为联想记忆或“快速权重”发挥作用。此处的新颖之处在于，在生产模型内部使其变得精确且可寻址，并可靠到足以在其上运行整个 CPU。

This is the base on which this entire model is built. As it stands, default Qwen 3.5 does not keep exact values in the GDN. With this modification, we unlock the next part of this experiment.

这就是整个模型构建的基础。目前，默认的 Qwen 3.5 不会在 GDN 中保留精确值。通过这一修改，我们开启了实验的下一阶段。

Since we have a fully functional, cheap and fast RAM now, we can go ahead with compiling a working CPU into the model weights. Luckily, there’s no training involved: a compiler sets every weight directly. You describe the CPU as a circuit, a long list of small steps like “read this register”, “add these two numbers” or “jump if zero”, and the compiler turns each step into a piece of a transformer layer: attention heads that look things up, feed-forward units that do the arithmetic, and DeltaNet heads that hold the RAM. The CPU then runs as the model decodes.

既然我们现在拥有了功能齐全、廉价且快速的内存，就可以着手将一个可工作的 CPU 编译进模型权重中。幸运的是，这不涉及任何训练：编译器直接设置每个权重。你将 CPU 描述为一个电路，即一长串诸如“读取此寄存器”、“将这两个数字相加”或“如果为零则跳转”的小步骤，编译器会将每个步骤转换为 Transformer 层的一部分：用于查找的注意力头、执行算术的前馈单元以及持有内存的 DeltaNet 头。随后，CPU 会随着模型的解码过程运行。

Each instruction becomes a few generated tokens: one names the memory row to write (ADDR), the next carries the value (VAL), and a STEP token ends the instruction. Those tokens are the CPU’s execution trace, so while Qwen is generating them, the CPU is actually running.

每条指令都会变成几个生成的 token：一个指定要写入的内存行（ADDR），下一个携带数值（VAL），而 STEP token 则结束该指令。这些 token 是 CPU 的执行轨迹，因此当 Qwen 生成它们时，CPU 实际上正在运行。

The CPU is a 32-bit RISC-V core (RV32IM, plus the Zba extension and a small bignum unit for big-number arithmetic), and it takes up 11 layers in front of Qwen’s 32. As it stands, it passes the official riscv-tests, its programs match qemu, and it has run 18 million instructions free-running without a single error.55 Free-running means the CPU keeps going on its own outputs for the whole run, with no reference machine correcting it between steps.

该 CPU 是一个 32 位 RISC-V 核心（RV32IM，外加 Zba 扩展和一个用于大数运算的小型大数单元），它占用了 Qwen 原有 32 层之前的 11 层。目前，它通过了官方的 riscv-tests，其程序运行结果与 qemu 一致，并且在自由运行状态下执行了 1800 万条指令且未出现任何错误。（自由运行意味着 CPU 在整个运行过程中持续基于自身的输出进行计算，期间没有参考机器进行步骤校正。）

This CPU can also run C and MicroPython code in the forward passes quite well (although a bit slowly). It is not a Linux-class chip yet, but it implements the full RV32IM instruction set.

该 CPU 也能在前向传播中相当好地运行 C 和 MicroPython 代码（尽管速度稍慢）。它还不是 Linux 级别的芯片，但它实现了完整的 RV32IM 指令集。

Qwen 3.5 4B is able to use this CPU quite well, without any post-training required. The CPU is used completely internally, and not invoked through any external tool calls.

Qwen 3.5 4B 能够很好地使用该 CPU，无需任何后训练。CPU 完全在内部使用，不通过任何外部工具调用来触发。

In order to make the model use this reliably, I decided to go the cheap way: by piggybacking on the tool call format that models are quite well-trained on, Qwen sends the CPU its instructions and reads back the results. No external tool calls or runtime are used. Once our CPU receives the instructions, the computer layers run, and then write the tool response. We use a small driver called RvChat to load the firmware into the GDN state, and this keeps the trace tokens out of Qwen’s layers.

为了使模型能可靠地使用它，我决定采用一种简便的方法：利用模型非常熟悉的工具调用格式，让 Qwen 将指令发送给 CPU 并读取结果。过程中不使用任何外部工具调用或运行时。一旦我们的 CPU 接收到指令，计算层就会运行，然后写入工具响应。我们使用一个名为 RvChat 的小型驱动程序将固件加载到 GDN 状态中，这使得轨迹 token 不会进入 Qwen 的层。

Qwen’s original weights stay exactly as they are. The computer weights are added on top of them, in extra dims and layers.

Qwen 的原始权重保持不变。计算机权重是在其基础上，通过额外的维度和层添加进去的。

I think the main question here on your mind after reading through my incessant ranting will be: does this actually work?

我想在读完我这一连串的唠叨后，你心中最大的疑问应该是：这真的有效吗？

On the DeepMind sets:

在 DeepMind 数据集上：

On NUPA’s long-integer arithmetic, Qwenputer goes from 48.6% to 100%, the same as giving Qwen a Python tool.

在 NUPA 的长整数算术运算中，Qwenputer 的准确率从 48.6% 提升到了 100%，这与为 Qwen 提供 Python 工具的效果相同。

The numbers the CPU prints are exact, too: I replayed about 230,000 of its output positions against a reference machine and found zero mismatches. And bolting a computer onto Qwen doesn’t change how it talks: its normal text stays within noise of base Qwen (KL 6e-4, against a noise floor of 5e-4).

CPU 打印出的数字也是精确的：我将它约 230,000 个输出位置与参考机器进行了比对，发现零误差。此外，在 Qwen 上加装计算机并不会改变它的说话方式：其正常文本仍保持在基础 Qwen 的噪声范围内（KL 散度为 6e-4，而噪声基准为 5e-4）。

Because a CPU inside a language model clearly wasn’t cursed enough, I decided the prudent thing woul

因为在语言模型中内置一个 CPU 显然还不够“邪门”，我决定明智的做法是……