Speedrunning Relational Foundation Model Pretraining

查看原文 / View Original 编辑 / Edit

Computer Science > Artificial Intelligence arXiv:2610.28855 (cs) [Submitted on 20 Oct 2026] Title: Speedrunning Relational Foundation Model Pretraining Authors: Gabriel Wedson Mendonça de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa, Francisco Galuppo Azevedo.

计算机科学 > 人工智能 arXiv:2610.28855 (cs) [提交于 2026 年 10 月 20 日] 标题：关系型基础模型预训练的竞速（Speedrunning）作者：Gabriel Wedson Mendonça de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa, Francisco Galuppo Azevedo。

Abstract: Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception. Recipes are developed inside individual labs, reported under private protocols, and do not accumulate. Speedrun leaderboards remove that barrier by decentralizing the search rather than the training run. The problem is fixed, the stack is open, entries are ranked by wall-clock time to a quality target, and one GPU is enough to hold the record.

摘要：基础模型预训练的进展受限于谁能负担得起试验成本，关系型预训练也不例外。相关方案在各个实验室内部开发，以私有协议报告，且无法积累。竞速排行榜通过去中心化搜索而非训练过程来消除这一障碍。问题是固定的，技术栈是开放的，参赛作品按达到质量目标所需的挂钟时间进行排名，且单张 GPU 足以刷新纪录。

We introduce modded-nanoRT, the first such leaderboard for Relational Transformers. Which database to hold out decides whether the benchmark can resolve improvements at all, so we fix it by calibration, ranking six candidates by how sharply a 45-minute learning curve separates improvement from seed noise. We open the leaderboard with typed trilinear attention, a relational operator that conditions attention scores and values on the foreign key traversed, together with Muon and SIGReg.

我们推出了 modded-nanoRT，这是首个针对关系型 Transformer 的此类排行榜。选择哪个数据库作为留出集决定了基准测试能否有效识别改进，因此我们通过校准解决了这个问题，根据 45 分钟学习曲线区分改进与种子噪声的清晰程度，对六个候选数据库进行了排名。我们以类型化三线性注意力（typed trilinear attention）开启了该排行榜，这是一种根据所遍历外键来调节注意力分数和值的关系算子，并结合了 Muon 和 SIGReg。

The recipe reaches the target in 305.63 seconds on one NVIDIA H200, 8.11x faster than the reference implementation, and wins 13 of 15 pairs outside the development database. We will release the benchmark, its frozen artifacts, and the record runs. Our code is available at this https URL.

该方案在单张 NVIDIA H200 上于 305.63 秒内达到目标，比参考实现快 8.11 倍，并在开发数据库之外的 15 个配对中赢得了 13 个。我们将发布该基准测试、其冻结的工件以及创纪录的运行结果。我们的代码可在该 HTTPS 链接获取。