Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM

Mini-AGI is a continual learning byte-level language model that assembles its own architecture, trains from scratch on a single 8 GB VRAM GPU, and keeps learning from everything it reads. It stores its weights as ordinary files on disk and pages them onto the card as it needs them, so the parameter count is bounded by free disk space rather than by VRAM. It grows new capacity while training when it runs short, prunes what nothing asks for, and reads through exactly the same code path it serves on.

Mini-AGI 是一个持续学习的字节级语言模型，它能够组装自己的架构，在单张 8GB 显存的 GPU 上从零开始训练，并不断从其阅读的所有内容中学习。它将权重作为普通文件存储在磁盘上，并根据需要将其分页加载到显卡中，因此参数数量受限于可用磁盘空间而非显存大小。它在训练过程中如果容量不足会自动扩展，修剪掉无用的部分，并且在推理时使用与训练时完全相同的代码路径。

Targeted at a PC or laptop with at least an 8 GB VRAM GPU on the board. NOTE: as of now this is a small toy-level model. Do not expect a frontier level capabilities. This is rather a small experiment to show, that continual learning from the single stream of data without catastrophic forgetting is possible. Furthermore it is possible on a modest hardware. Which means that almost everyone could train their own version of the model (or simply continue training this one) exactly as they see it fit. And the capabilities would be bounded by the actual hardware, scale and quality of the data available and the amount of time one willing to spend on training the model.

该模型针对配备至少 8GB 显存 GPU 的台式机或笔记本电脑。注意：目前这只是一个小型玩具级模型，请勿期待其具备前沿水平的能力。这更像是一个小型实验，旨在证明在不发生灾难性遗忘的情况下，从单一数据流中进行持续学习是可能的。此外，这在普通硬件上也是可行的。这意味着几乎每个人都可以按照自己的意愿训练自己的模型版本（或者直接继续训练当前版本）。其能力将受限于实际硬件、数据的规模与质量，以及投入训练的时间。

Here is how min-run dashboard looks like. The model is pointed to the corpus to constantly read and learn from. History - here is the samples from the whole training run history so far. You can inspect them yourself to see how the model improved over the course of training/reading the corpus. The weights are not published yet. The run is still reading its first pass over the corpus, the weights go up once it has been through all of it, which is a couple of weeks away at the current rate.

这是 min-run 仪表盘的样子。模型指向语料库以持续阅读和学习。历史记录——这里是迄今为止整个训练过程的历史样本。你可以亲自查看模型在训练/阅读语料库过程中的改进情况。权重尚未发布。该运行目前仍在进行语料库的第一轮阅读，一旦完成全部阅读（按当前速度还需要几周时间），权重将会上传。

Motivation

Every language model you can actually own today is a model somebody else trained and then froze. You can fine-tune around the edges of it, but you cannot train one from scratch on your own hardware, and you cannot keep training it on what you do day to day - the moment you try, it forgets what it knew before. The result is that a personal model is always somebody else’s model with a thin layer of you on top, and it stops learning the day it ships.

动机

今天你真正能拥有的每一个语言模型，都是别人训练并冻结后的模型。你可以在其边缘进行微调，但无法在自己的硬件上从零开始训练一个模型，也无法在日常工作中持续训练它——一旦你尝试这样做，它就会忘记之前学到的知识。结果就是，所谓的个人模型永远是别人的模型加上一层薄薄的“你”，而且它在发布的那一天就停止了学习。

mini-AGI model has small enough GPU footprint that it is possible to train end-to-end on one consumer card, and it is built so that training never has to stop. It reads a stream of characters one chunk at a time, takes a gradient step on each, and the same path serves generation. There is no separate fine-tuning regime and no frozen base: reading and being trained are the same event.

mini-AGI 模型的 GPU 占用空间足够小，可以在单张消费级显卡上进行端到端训练，并且其设计使得训练永远不必停止。它一次读取一个字符流片段，在每个片段上执行梯度步长，并且推理生成也使用相同的路径。没有单独的微调机制，也没有冻结的基座：阅读和训练是同一个过程。

Three constraints shape everything else in the design:

It has to fit on 8 GB. Not with quantisation - training needs gradients and optimiser state, which is roughly three times the weights again. So the weights live on disk and only the working set is resident.
It has to not forget. A model that learns continually and overwrites itself is worse than one that does not learn at all.
It has to be able to read anything. The alphabet is the 256 byte values, so there is no tokenizer to fit and no data type that needs a new vocabulary.

设计中的三个核心约束：

必须适配 8GB 显存。 不能仅靠量化——训练需要梯度和优化器状态，这大约是权重大小的三倍。因此，权重存储在磁盘上，只有工作集驻留在显存中。
必须不能遗忘。 一个持续学习并不断覆盖自身知识的模型，比完全不学习的模型更糟糕。
必须能够读取任何内容。 字母表是 256 个字节值，因此不需要适配分词器，也不需要为特定数据类型准备新词汇表。

The model is genuinely yours: trained on your hardware, on your data, that keeps learning from every conversation you have with it, and that nobody else can take it away or switch it off.

这个模型真正属于你：在你的硬件上、使用你的数据进行训练，它会从你与它的每一次对话中持续学习，没有人能把它夺走或将其关闭。

How the architecture works

Characters (bytes) does not pass through a fixed stack of layers as it would be in a traditional LLM. Instead, it passes through two dense prelude blocks and then through one recurrent block applied up to 24 times, each application choosing its own experts from a shared pool. The latent state between applications is never decoded - it is merged with the embedded input by an adapter each time round, so the loop cannot drift away from the text it is reading.

架构原理

字符（字节）不会像传统大语言模型那样通过固定的层堆栈。相反，它通过两个密集的前奏块，然后通过一个最多应用 24 次的循环块，每次应用都会从共享池中选择自己的专家。应用之间的潜在状态从不解码——每次循环都通过适配器与嵌入的输入合并，因此循环不会偏离它正在阅读的文本。

Three distinct blocks, up to 26 block-applications per character. Adaptive depth. A halting head scores every character at every row, and the character stops as soon as another row would not change the answer. Easy characters take one row, hard ones take many. This is the PonderNet recipe: while training, every depth is computed and weighted by its halting probability, so the halting head learns through those weights.

三个不同的块，每个字符最多进行 26 次块应用。自适应深度。一个停止头（halting head）在每一行对每个字符进行评分，一旦增加一行不会改变答案，字符处理就会停止。简单的字符占用一行，复杂的字符占用多行。这就是 PonderNet 的配方：在训练时，每个深度都会被计算并根据其停止概率进行加权，因此停止头可以通过这些权重进行学习。

Routing per block-application, not per character. Each of the 26 applications picks its own top-8 experts, so one character touches far more of the pool than “top-8” suggests, and the same expert can be selected several times at different depths. What varies is which eight at each point. No expert is assigned a subject. There are no labels anywhere. Soft top-k routing distributes capability across the pool by itself, and a character can combine fragments from several experts. The cost is that capabilities share parameters and so can interfere.

路由是基于块应用而非字符进行的。26 次应用中的每一次都会选择自己的前 8 位专家，因此一个字符接触到的专家池远比“前 8 位”所暗示的要多，同一个专家可以在不同深度被多次选中。变化的是在每个点选中的是哪八位。没有任何专家被指定负责特定主题。没有任何地方存在标签。软 Top-k 路由自动将能力分配到整个池中，一个字符可以结合来自多个专家的片段。代价是能力共享参数，因此可能会产生干扰。

(The article continues with descriptions of visual dashboard traces and generation examples…)

（文章后续部分描述了可视化仪表盘的轨迹和生成示例……）