本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Ripple-65B Preview: U.S. Open-Weight SOTA on Agentic Coding

Achieving U.S. Open-Weight SOTA on Agentic Coding

在代理编码领域实现美国开源权重 SOTA

Ricursive Intelligence is a full-stack artificial intelligence lab building self-improving systems by co-optimizing hardware, software, and model.

Ricursive Intelligence 是一家全栈人工智能实验室，致力于通过协同优化硬件、软件和模型来构建自我改进系统。

To power our chip design platform, we trained a language model from scratch that sets a new U.S. open-weight SOTA on agentic coding. We optimized the model architecture for our training and serving platforms, driving substantial efficiency gains.

为了支持我们的芯片设计平台，我们从零开始训练了一个语言模型，该模型在代理编码方面创下了美国开源权重的新 SOTA。我们针对训练和推理平台优化了模型架构，从而实现了显著的效率提升。

Today, we release Ripple-65B Preview, a general-purpose variant of the proprietary model that powers our AI chip design platform. The model is pretrained on 12 trillion tokens with support for up to 1M context length. Ripple-65B Preview delivers strong results on agentic coding, achieving U.S. open-weight state-of-the-art (80.52) on TerminalBench 2.1 and surpassing significantly larger US open-weight models such as Reflection Beam 501B (80.1), Meta Muse Spark Open 816B (76.4), Thinking Machines Inkling 975B (63.8), and NVIDIA Nemotron 3 Ultra 550B (56.4).

今天，我们发布了 Ripple-65B Preview，这是驱动我们 AI 芯片设计平台的专有模型的通用版本。该模型在 12 万亿个 token 上进行了预训练，支持高达 100 万的上下文长度。Ripple-65B Preview 在代理编码方面表现出色，在 TerminalBench 2.1 上达到了美国开源权重的最高水平（80.52），超越了诸如 Reflection Beam 501B (80.1)、Meta Muse Spark Open 816B (76.4)、Thinking Machines Inkling 975B (63.8) 和 NVIDIA Nemotron 3 Ultra 550B (56.4) 等规模大得多的美国开源模型。

Ripple’s dense model architecture provides strong reasoning performance with low-latency token generation. The model’s small parameter envelope also enables it to run efficiently on local devices such as NVIDIA DGX Spark, which feature high compute FLOPs with low memory bandwidth relative to datacenter accelerators.

Ripple 的稠密模型架构在提供低延迟 token 生成的同时，具备强大的推理性能。该模型较小的参数规模使其能够在 NVIDIA DGX Spark 等本地设备上高效运行，这些设备相对于数据中心加速器而言，具有高计算 FLOPs 但内存带宽较低的特点。

Ripple-65B Preview is available on Hugging Face, Together AI, GMI Cloud, OpenRouter, and Fireworks.

Ripple-65B Preview 现已在 Hugging Face、Together AI、GMI Cloud、OpenRouter 和 Fireworks 上发布。

LLM pretraining is legendarily expensive. In 2023, OpenAI CEO Sam Altman said that GPT-4 training costs exceeded $100 million. In the following year, Dario Amodei said that frontier AI models can cost up to $1 billion to train, projecting that costs would rise to $10-100B by 2026 to 2027 for pretraining alone.

大语言模型预训练的成本高得惊人。2023 年，OpenAI 首席执行官 Sam Altman 表示 GPT-4 的训练成本超过了 1 亿美元。次年，Dario Amodei 表示前沿 AI 模型的训练成本可能高达 10 亿美元，并预计到 2026 年至 2027 年，仅预训练成本就将升至 100 亿至 1000 亿美元。

In contrast, we developed and trained Ripple-65B Preview for only $32M, which is one-tenth the cost of the previous open-weight SOTA on agentic coding, as estimated to the best of our knowledge.¹ Note that $32M is our full R&D cost starting from scratch, including pretraining ladder development, data mixing, evals, debugging, synthetic data generation, posttraining recipe development, and the final hero pretraining and posttraining run.

相比之下，我们开发并训练 Ripple-65B Preview 的成本仅为 3200 万美元，据我们所知，这仅为之前代理编码领域开源权重 SOTA 成本的十分之一。¹ 请注意，3200 万美元是我们从零开始的全部研发成本，包括预训练阶梯开发、数据混合、评估、调试、合成数据生成、后训练配方开发，以及最终的预训练和后训练运行。

Rather than investing in a brute-force RL climb, we carefully iterated on our SFT and RL recipes with a focus on compute efficiency. We opted not to buy any posttraining data, instead devising our own synthetic data pipelines, quality filtering, and coverage metrics.

我们没有投入资源进行蛮力强化学习（RL）攀升，而是专注于计算效率，仔细迭代了我们的监督微调（SFT）和强化学习配方。我们选择不购买任何后训练数据，而是设计了自己的合成数据流水线、质量过滤和覆盖率指标。

Our mixed RL recipe uses truncated importance sampling, score centering, and linear length penalty with high-quality long-horizon synthetic environments. The RL tasks are tuned for our model capabilities, and we continuously added harder tasks as the model improved. The end result was a steep RL climb from a strong initial SFT checkpoint. We reached a TerminalBench 2.1 score of 80.5 with only 1.4M rollouts, roughly 70x fewer rollouts than Reflection Beam, the prior SOTA.

我们的混合强化学习配方使用了截断重要性采样、分数中心化和线性长度惩罚，并结合了高质量的长程合成环境。强化学习任务针对我们的模型能力进行了调整，随着模型的改进，我们不断增加任务难度。最终结果是从一个强大的初始 SFT 检查点出发，实现了陡峭的强化学习性能提升。我们仅用 140 万次 rollout 就达到了 80.5 的 TerminalBench 2.1 分数，这比之前的 SOTA 模型 Reflection Beam 所需的 rollout 次数少了约 70 倍。

Ripple-65B Preview’s architecture was designed to be trained and served efficiently on NVIDIA hardware. A key objective was to make design decisions which reduce pretraining and posttraining infrastructure complexity while still enabling strong reasoning and agentic performance. Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE) like other labs. This decision quickly paid off, and we were able to train a strong model from scratch while maintaining a solid MFU (40%) throughout pretraining and posttraining, even with simple sharding, no custom kernels, and no quantization.

Ripple-65B Preview 的架构旨在在 NVIDIA 硬件上高效训练和推理。一个关键目标是做出能够降低预训练和后训练基础设施复杂性的设计决策，同时仍能实现强大的推理和代理性能。因此，我们选择训练一个现代化的稠密 Transformer 模型，而不是像其他实验室那样使用混合专家模型（MoE）。这一决定很快见效，我们能够从零开始训练出一个强大的模型，并在整个预训练和后训练过程中保持了稳健的 MFU（40%），即使在仅使用简单分片、无自定义内核且无量化的情况下也是如此。

Ripple-65B Preview uses a thin-and-deep model aspect ratio, with a narrow d_model (5120) and 80 total layers. To improve reasoning performance, we overprovision the number of attention heads while increasing the SWA-to-global layer ratio and reducing SWA window size to compensate for the increased cost. In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.

Ripple-65B Preview 使用了“瘦长”的模型纵横比，具有较窄的 d_model (5120) 和总共 80 层。为了提高推理性能，我们超额配置了注意力头数量，同时增加了 SWA（滑动窗口注意力）与全局层的比例，并减小了 SWA 窗口大小以补偿增加的成本。此外，我们还添加了 FFN 激活裁剪、门控注意力、全局 NoPE 和值残差。

We optimized Ripple’s pretraining recipe through a series of careful model laddering experiments.

我们通过一系列严谨的模型阶梯实验优化了 Ripple 的预训练配方。

Our starting point was Olmo 3’s model ladder, from which we tested interventions at small scale with a focus on improving coding capabilities. Our v1.0 ladder implemented StepFun batch size and learning rate scaling laws, and v1.1 implemented our new ladder which smoothly scales architectural dimensions and model depth. Ladder v1.2 switched from the post-norm baseline to pre+post norm, which was neutral at small scale but showed improvements starting at 4B. Ladder v1.3 implemented sigmoid gated attention. Ladder v1.4 (2-stream mHC) showed good results at smaller scale but did not improve at 4B scale, and was therefore rejected. Ladder v1.5 switched to global NoPE and added a final LR cooldown to zero, and ladder v1.6 added value residuals. In the final version, we switched to our custom pretraining dataset with substantially upsampled code.

我们的起点是 Olmo 3 的模型阶梯，在此基础上，我们在小规模下测试了干预措施，重点在于提高编码能力。我们的 v1.0 阶梯实现了 StepFun 批处理大小和学习率缩放定律，v1.1 实现了我们平滑缩放架构维度和模型深度的新阶梯。阶梯 v1.2 从后归一化基准切换为预+后归一化，这在小规模下表现中性，但在 4B 规模下开始显示出改进。阶梯 v1.3 实现了 Sigmoid 门控注意力。阶梯 v1.4（双流 mHC）在较小规模下表现良好，但在 4B 规模下没有改进，因此被弃用。阶梯 v1.5 切换到全局 NoPE 并添加了最终的学习率冷却至零，阶梯 v1.6 添加了值残差。在最终版本中，我们切换到了大幅上采样代码的自定义预训练数据集。

Together, the compounded changes improved compute efficiency on code as measured by dolma_stack CE by up to 16x at 4B model scale. In other words, our final Ripple model ladder can be used to achieve similar code performance to the Olmo 3 ladder with 16x less compute. Note that the slope of the Olmo 3 ladder baseline is shallower than ladder v1.0+, which shows that the compute efficiency improvement will increase with additional scale.

总的来说，这些复合变化使代码计算效率（以 dolma_stack CE 衡量）在 4B 模型规模下提高了多达 16 倍。换句话说，我们最终的 Ripple 模型阶梯可以用少 16 倍的计算量实现与 Olmo 3 阶梯相似的代码性能。请注意，Olmo 3 阶梯基准的斜率比阶梯 v1.0+ 更平缓，这表明随着规模的扩大，计算效率的提升将会增加。

We built Ripple-65B Preview to begin closing the loop between models and the hardware that fuels them. An internal variant of this model powers our AI chip design platform, and we look forward to seeing the capabilities that emerge as we scale toward the frontier.

我们构建 Ripple-65B Preview 是为了开始闭合模型与驱动它们的硬件之间的循环。该模型的一个内部版本为我们的 AI 芯片设计平台提供支持，我们期待在向技术前沿扩展的过程中，看到更多涌现出的能力。

We were fortunate to build on top of existing open-source contributions, including those from the Olmo team at the Allen Institute for AI (AI2), Prime Intellect, RadixArk, and NVIDIA. A strong open-source ecosystem is also self-improving, where contributors strengthen the ecosystem, which in turn draws more contributors.

我们很幸运能够建立在现有的开源贡献之上，包括来自艾伦人工智能研究所 (AI2) 的 Olmo 团队、Prime Intellect、RadixArk 和 NVIDIA 的贡献。一个强大的开源生态系统也是自我改进的，贡献者加强了生态系统，而这反过来又吸引了更多的贡献者。