Training a Language Model End-to-End in Rust: An Experience Report

使用 Rust 端到端训练语言模型：一份经验报告

I pretrained a language model end-to-end in Rust - alone, with no team, no PyTorch, and no Python in the training path - for $164 in rented GPU time. I report that as an achievement, not a recommendation: the more useful contribution is a measured failure taxonomy of the two leading Rust ML frameworks, Candle and Burn, as training (not inference) backends in 2026.

我使用 Rust 端到端预训练了一个语言模型——全程独立完成，没有团队，没有 PyTorch，训练路径中也不包含任何 Python 代码——仅花费了 164 美元的 GPU 租赁费用。我将其视为一项成就，而非推荐做法：这份报告更有价值的贡献在于，对 2026 年领先的两个 Rust 机器学习框架（Candle 和 Burn）作为训练（而非推理）后端时，所存在的故障进行了分类评估。

I document five Candle defects, including fused kernels that silently produce no gradient, and three Burn defects, including a backward pass at roughly 3% of theoretical GPU throughput and a kernel-fusion path that segfaults mid-training at multi-billion-parameter scale. Every one passed ordinary loss-curve inspection; none announced itself.

我记录了 Candle 的五个缺陷，包括会导致静默失效（不产生梯度）的融合算子（fused kernels）；以及 Burn 的三个缺陷，包括反向传播速度仅为理论 GPU 吞吐量的 3% 左右，以及在数十亿参数规模下训练中途会发生段错误（segfault）的算子融合路径。每一个缺陷都能通过常规的损失曲线检查，且没有任何预警。

I describe the verification discipline that caught six such silent failures, centered on a gradient-flow arbiter: a test that runs one forward/backward pass and asserts every trainable parameter receives a finite, nonzero gradient, generalizable to any framework.

我描述了一种用于捕捉这六类静默故障的验证准则，其核心是一个“梯度流仲裁器”：通过运行一次前向/反向传播，并断言每个可训练参数都能接收到有限且非零的梯度，这种方法可以推广到任何框架中。

The trained model (roughly 0.4B parameters, Bangla-first) shows strong Bangla language-modeling signal - a per-token negative log-likelihood of 0.93 against 12.60 for a random-initialized twin - while scoring at chance on English commonsense multiple-choice, the expected outcome of a deliberately small, Bangla-weighted budget (about 2 billion tokens, 54.6 hours, one rented H100).

训练出的模型（约 4 亿参数，以孟加拉语为主）表现出了强大的孟加拉语建模能力——每个 token 的负对数似然（NLL）为 0.93，而随机初始化的对照组为 12.60——但在英语常识多项选择题上仅达到随机水平。这是在刻意限制规模且侧重孟加拉语预算（约 20 亿 token，54.6 小时，单张租赁 H100）下的预期结果。

I also report a tokenizer-fertility trap in Bengali script: naive byte-level tokenization collapsed Bangla to roughly 1.4 characters per token against English’s 3.9, silently inverting the corpus’s language balance; fixing it reached roughly 4.1. To my knowledge, this is among the first documented end-to-end LM pretraining runs in pure Rust.

我还报告了孟加拉文字中存在的一个分词器（tokenizer）陷阱：原始的字节级分词导致孟加拉语的每个 token 平均仅包含约 1.4 个字符，而英语为 3.9 个，这静默地颠倒了语料库的语言平衡；修复后该数值达到了约 4.1。据我所知，这是首批记录在案的纯 Rust 端到端语言模型预训练实验之一。

After this run I moved training to PyTorch and kept Rust for on-device serving: in my hands, Rust is not yet a competitive place to train a language model, though it may be a good place to serve one.

在这次实验后，我将训练工作转回了 PyTorch，并保留 Rust 用于端侧推理服务：以我的经验来看，Rust 目前还不是训练语言模型的理想选择，尽管它在模型部署方面表现出色。