Kairos: Toward a Byte-Level Foundation Model for Every Modality

Computer Science > Machine Learning arXiv:2610.12495 (cs) [Submitted on 26 Sep 2026] Title: Kairos: Toward a Byte-Level Foundation Model for Every Modality Authors: Fabien Furfaro.

计算机科学 > 机器学习 arXiv:2610.12495 (cs) [提交于 2026 年 9 月 26 日] 标题：Kairos：迈向适用于每种模态的字节级基础模型 作者：Fabien Furfaro。

Abstract: Small language models trained on a single local machine leave little room for architectural trial and error. Kairos combines bidirectional gated-DeltaNet linear attention and sliding-window attention over shared projections (LiZAttention2), block-wise Attention-Residual aggregation, a sparse Mixture-of-Experts (MoE) feed-forward layer and a multi-scale byte-level codec, trained with a uniform-noise denoising objective over one byte space shared by text, image, audio, video, lidar and control streams; this report asks whether the combination optimizes at all before asking whether it is useful.

摘要：在单台本地机器上训练的小型语言模型几乎没有架构试错的空间。Kairos 结合了双向门控 DeltaNet 线性注意力机制和共享投影上的滑动窗口注意力机制（LiZAttention2）、块级注意力残差聚合、稀疏专家混合（MoE）前馈层以及多尺度字节级编解码器，并通过在文本、图像、音频、视频、激光雷达和控制流共享的单一字节空间上进行均匀噪声去噪目标训练；本报告在探讨其是否有用之前，先探讨了这种组合是否能够进行优化。

We give code-derived algorithms for each component, including a cross-session recurrent-state memory gate, then run convergence sanity checks: overfitting a single 440-byte sentence at 1—15M parameters, text only, with three seeds for codec and attention choices and single-run sweeps for seven further factors. The two attention variants tested at 3,000 steps (LiZAttention2 and window attention alone) reach a minimum loss near $10^{-3}$.

我们为每个组件提供了代码导出的算法，包括跨会话循环状态记忆门，然后进行了收敛性完整性检查：在 100 万至 1500 万参数下对单个 440 字节的句子进行过拟合（仅限文本），并为编解码器和注意力选择设置了三个随机种子，同时对其他七个因素进行了单次扫描。在 3000 步测试中，两种注意力变体（LiZAttention2 和单独的窗口注意力）的最小损失均接近 $10^{-3}$。

After 600 steps, the mean minimum loss ranges from 0.018 (patch codec) to 0.354 (depthwise convolutional codec) with an identical ranking on three seeds. An apparent advantage of shared over separate attention projections closes with more steps or more diverse data and reverses when the projections are frozen: it is a convergence-speed effect, not a capacity effect. These checks suggest that the implementation optimizes; they say nothing about generalization, multimodal training or comparison with existing models, which a planned pretraining run is meant to address.

在 600 步之后，平均最小损失范围从 0.018（补丁编解码器）到 0.354（深度卷积编解码器），且在三个随机种子上的排名相同。共享注意力投影相对于独立注意力投影的明显优势会随着步数的增加或数据的多样化而缩小，并在投影被冻结时发生逆转：这是一种收敛速度效应，而非容量效应。这些检查表明该实现能够优化；但它们并未说明泛化能力、多模态训练或与现有模型的比较，这些问题将在计划中的预训练运行中解决。