Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

Over the past several months, we’ve seen an explosion in the capabilities of open source models. With DeepSeek V4-Pro and GLM5.2 reaching near-Opus levels of intelligence, open source has emerged as a real, cost-efficient alternative to the closed source models we’ve been married to. But we have yet to see one like Kimi K3. Promising Fable/Sol levels of intelligence, Kimi K3 marks the start of a new era for open source. 在过去的几个月里，我们见证了开源模型能力的爆发式增长。随着 DeepSeek V4-Pro 和 GLM5.2 的智能水平接近 Opus，开源模型已成为我们长期依赖的闭源模型的一种切实且经济高效的替代方案。但我们从未见过像 Kimi K3 这样的模型。Kimi K3 承诺达到 Fable/Sol 级别的智能，标志着开源新时代的开启。

But a smarter model means a bigger model — and these models are expanding in size just as fast as they are in capabilities. GLM5.2 has 753B parameters, DeepSeek V4-Pro 1.6T, and Kimi K3 weighs in at 2.8T (!!) parameters. That’s over 1.5TB of VRAM before allocating a KV cache for 1M tokens of context. Not even a B200 node (8 GPUs) can fit Kimi K3. That leaves you with limited options: serve on a node of B300s, which have 288GB of VRAM per GPU, or commit two B200 nodes (TP16) to serving Kimi. 但更聪明的模型意味着更大的体积——这些模型的规模增长速度与它们的能力提升速度一样快。GLM5.2 拥有 7530 亿参数，DeepSeek V4-Pro 为 1.6 万亿，而 Kimi K3 则高达 2.8 万亿（！！）参数。这意味着在为 100 万 token 上下文分配 KV 缓存之前，就需要超过 1.5TB 的显存。即使是 B200 节点（8 个 GPU）也无法容纳 Kimi K3。这让你只有有限的选择：要么在 B300 节点上运行（每个 GPU 拥有 288GB 显存），要么投入两个 B200 节点（TP16）来部署 Kimi。

But guess which other non-NVIDIA GPU has 288GB of VRAM? AMD’s MI355X. Can you tell we like these chips yet? At around ~2.4× cheaper per GPU on average versus a B300 and ~1.7× cheaper than a B200, the MI355X is a cost-efficient alternative to Blackwells with comparable hardware specs. The only problem with AMD is software support — slower kernels and less day-0 support on inference frameworks make serving frontier models on AMD a real engineering effort. 但你猜还有哪款非 NVIDIA GPU 拥有 288GB 显存？AMD 的 MI355X。看得出我们有多喜欢这些芯片了吗？与 B300 相比，MI355X 平均每个 GPU 的成本便宜约 2.4 倍，比 B200 便宜约 1.7 倍，它是 Blackwell 系列的一种经济高效的替代方案，且硬件规格相当。AMD 唯一的问题在于软件支持——较慢的内核和推理框架上较少的“首日支持”（day-0 support），使得在 AMD 上部署前沿模型成为一项真正的工程挑战。

Our claim at Wafer is that agents are improving at kernel and model optimization, closing this gap as we speak. But with AMD shipping day-0 support for Kimi K3, most of the work was already done for us. The results are great: on a 1,024-token input / 400-token output benchmark, the MI355X reaches 952 tok/s/node and 118 tok/s single stream — over 3.8× the aggregate throughput per node and over 1.3× the single-stream decode of our TP16 B200 deployment (whose 498 tok/s is a 16-GPU, 2-node total — ~249/node). 我们在 Wafer 的观点是，智能体在内核和模型优化方面正在不断进步，我们正在见证这一差距的缩小。由于 AMD 为 Kimi K3 提供了首日支持，大部分工作已经为我们完成。结果非常出色：在 1024 token 输入 / 400 token 输出的基准测试中，MI355X 达到了每节点 952 tok/s 和单流 118 tok/s 的性能——这是我们 TP16 B200 部署（其 498 tok/s 是 16 个 GPU、2 个节点的总和，即每节点约 249 tok/s）每节点总吞吐量的 3.8 倍以上，单流解码性能的 1.3 倍以上。

B300 nodes still win ~1.65× on aggregate throughput over the MI355X, but at 2.4× the price, the MI355X crushes the B300 on performance per dollar. B300 节点在总吞吐量上仍比 MI355X 高出约 1.65 倍，但考虑到 2.4 倍的价格差异，MI355X 在性价比上完胜 B300。

Performance Comparison Table

性能对比表

	8× MI355X (TP8)	2×8 B200 (TP16)	B300 (TP8+DCP8)
Decode tok/s per stream	118 tok/s	90 tok/s	172 tok/s
Peak aggregate	952 tok/s	498 tok/s	1,568 tok/s
Peak aggregate per GPU	119 tok/s	31 tok/s	196 tok/s
Peak aggregate per $/GPU-hr	48 tok/s/$	7 tok/s/$	33 tok/s/$

Perf/dollar at $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200. 性价比计算基于：MI355X 为 2.50 美元/GPU-小时，B300 为 6.00 美元，B200 为 4.25 美元。

To the B200’s defence, its numbers are somewhat deflated by the fact that it pays a cross-node all-reduce on the decode critical path (RoCE v2 at ~195 Gb/s) — it’s the only config here that spans two nodes, because Kimi K3 won’t fit weights plus a 1M-token KV pool on a single 8×192GB node. But that’s exactly the point: Kimi K3 at its size is one of the first models we’ve seen where the MI355X’s focus on HBM capacity gives it a practical, measurable edge over the B200. 为 B200 辩护一下，它的数据在一定程度上受到了解码关键路径上跨节点 all-reduce（RoCE v2，约 195 Gb/s）开销的影响——这是此处唯一跨两个节点的配置，因为 Kimi K3 的权重加上 100 万 token 的 KV 池无法放入单个 8×192GB 节点中。但这正是重点所在：Kimi K3 是我们见过的首批模型之一，MI355X 对 HBM 容量的侧重使其在实际应用中对 B200 具有可衡量的优势。

How we did it

我们是如何做到的

While Kimi K3 served out of the box, there was still work to be done to get it to its current throughput number. The main lever was speculative decode. K3 ships zero draft tensors — no MTP, no EAGLE — so the only speculative path is an external block-diffusion draft: RadixArk’s Kimi-K3-DSpark. On CUDA it just runs. On ROCm our first real request breaks the scheduler with this error: NameError: name 'top_k_renorm_prob' is not defined. Did you mean: 'top_p_renorm_prob'? 虽然 Kimi K3 可以开箱即用，但要达到目前的吞吐量数字仍需一番努力。主要的手段是投机解码（speculative decode）。K3 没有提供草稿张量（draft tensors）——没有 MTP，没有 EAGLE——所以唯一的投机路径是外部块扩散草稿：RadixArk 的 Kimi-K3-DSpark。在 CUDA 上它可以直接运行。但在 ROCm 上，我们的第一个真实请求就导致调度程序崩溃，并报错：NameError: name 'top_k_renorm_prob' is not defined. Did you mean: 'top_p_renorm_prob'?

sglang’s accept-sampling verifier has two ways to build the target distribution: a dense path that calls top_k_renorm_prob, and a sparse fast path that routes through torch.topk directly. The CUDA build imports top_k_renorm_prob from sgl_kernel; the ROCm build aliases only a Triton top-p kernel and leaves top_k_renorm_prob undefined — there’s no top-k renorm kernel for gfx950 to alias. So the moment a request lands on the dense path, the verifier hits that NameError and takes the scheduler down with it. sglang 的接受采样验证器有两种构建目标分布的方法：一种是调用 top_k_renorm_prob 的密集路径，另一种是直接通过 torch.topk 路由的稀疏快速路径。CUDA 构建版本从 sgl_kernel 导入 top_k_renorm_prob；而 ROCm 构建版本仅别名了一个 Triton top-p 内核，导致 top_k_renorm_prob 未定义——因为 gfx950 没有可供别名的 top-k renorm 内核。因此，一旦请求进入密集路径，验证器就会触发该 NameError 并导致调度程序崩溃。

The fix is a single PyTorch function. Top-k renorm is a small operation: take the model’s probability vector, keep the k highest entries, zero the rest, and rescale what’s left to sum to 1. A sort, a masked_fill, a divide — dropped straight into sglang’s ROCm sampling branch, the same computation the CUDA build gets from sgl_kernel. No custom kernel: the reflex on ROCm is to assume you need one, but here it was a missing definition, not a missing kernel. 修复方法是一个简单的 PyTorch 函数。Top-k renorm 是一个小操作：获取模型的概率向量，保留最高的 k 个条目，将其余部分归零，并重新缩放剩余部分使其总和为 1。通过排序、masked_fill 和除法，直接放入 sglang 的 ROCm 采样分支中，这与 CUDA 构建版本从 sgl_kernel 获得计算结果是一样的。不需要自定义内核：在 ROCm 上人们通常会下意识地认为需要自定义内核，但在这里，问题仅仅是定义缺失，而不是内核缺失。

With spec dec fixed and hardened, we gained ~2.2× performance single-stream, ~1.7× per-stream at moderate load, and +18% peak aggregate. More importantly, our peak aggregate throughput landed on much higher concurrency (c64 vs c24 no-spec). 随着投机解码的修复和加固，我们获得了约 2.2 倍的单流性能提升，在中等负载下每流性能提升约 1.7 倍，峰值总吞吐量提升了 18%。更重要的是，我们的峰值总吞吐量是在更高的并发度下实现的（c64 对比无投机时的 c24）。

Prefill optimizations

预填充（Prefill）优化

Discussion around model performance tends to highlight decode tokens per second. But in many cases decode tok/s is fool’s gold — decode is over-glorified, while time-to-first-token, the number users feel the most, gets overlooked. The MI355X struggles here: an identical 172k-token cold prefill took ~51s on MI355X versus ~23s on a B300. On a 1M-context model, a lot of workloads have huge prefills (sometimes cold), and having GPUs spin on prefill for minutes can render entire fleets of nodes useless. 关于模型性能的讨论往往强调每秒解码 token 数。但在许多情况下，解码 tok/s 只是虚假的繁荣——解码被过度美化，而用户感受最深的首字延迟（time-to-first-token, TTFT）却被忽视了。MI355X 在这方面表现吃力：同样的 17.2 万 token 冷预填充在 MI355X 上耗时约 51 秒，而在 B300 上仅需约 23 秒。在 100 万上下文的模型中，许多工作负载都有巨大的预填充（有时是冷启动），让 GPU 在预填充上空转几分钟可能会导致整个节点集群瘫痪。

The gap was almost entirely one kernel. K3 on ROCm was falling back to slow generic Triton attention because the fast AITER MLA prefill kernel wouldn’t load. The problem was a shape mismatch, not a missing kernel — K3 at TP8 gives 12 attention heads per rank, and AITER’s MLA path is built for 4, 8, or multiples of 16. The fix was trivially simple: zero-pad the head count 12→16, run the fast kernel, and extract the real 12 heads from the output. 这个差距几乎完全是由一个内核造成的。K3 在 ROCm 上回退到了缓慢的通用 Triton 注意力机制，因为快速的 AITER MLA 预填充内核无法加载。问题在于形状不匹配，而不是内核缺失——K3 在 TP8 下每个 rank 提供 12 个注意力头，而 AITER 的 MLA 路径是为 4、8 或 16 的倍数构建的。修复方法非常简单：将头数从 12 零填充（zero-pad）到 16，运行快速内核，然后从输出中提取真实的 12 个头。

The result: on the same 172k cold prefill, the AITER MLA prefill ASM runs at ~13k tok/s steady-state vs the Triton fallback’s ~4–7k, speeding up prefill by ~2–3×. It’s a TTFT lever, not an aggregate-throughput one — decode is unchanged, so it doesn’t move the numbers above; it moves the number a user waits on before the first token appears. 结果是：在同样的 17.2 万 token 冷预填充中，AITER MLA 预填充 ASM 以约 1.3 万 tok/s 的稳态速度运行，而 Triton 回退方案仅为约 4-7k，预填充速度提升了约 2-3 倍。这是一个针对 TTFT 的优化，而不是针对总吞吐量的优化——解码速度保持不变，因此它不会改变上面的性能数据；它改变的是用户在第一个 token 出现前等待的时间。

Takeaways

总结

Achieving the best performance-per-dollar ratio on the MI355X was relatively out of the box. There were some expected f… 在 MI355X 上实现最佳性价比相对来说是开箱即用的。虽然遇到了一些预料之中的……