Model Routing Is Simple. Until It Isn’t.

模型路由很简单。直到它变得复杂。

Building a router into your agent sounds like an easy win. Send simple requests to cheaper models, reserve expensive ones for harder tasks, or route by specialty — Claude for code, Gemini for multimodal, and so on. A classifier or heuristic makes the call, costs go down, performance stays up. Done. Except it’s not. Most routing systems assume that model selection is a classification problem. In our experience building routing into agentic systems, what looks like a model-selection problem quickly becomes a systems optimization problem. Three dimensions made this surprisingly hard for us.

在你的智能体（Agent）中构建一个路由系统听起来是个轻松的胜利。将简单的请求发送给更便宜的模型，将昂贵的模型留给更难的任务，或者按专业领域进行路由——比如用 Claude 处理代码，用 Gemini 处理多模态任务，等等。通过一个分类器或启发式算法做出决策，成本降低，性能保持不变。大功告成。但事实并非如此。大多数路由系统都假设模型选择是一个分类问题。根据我们在构建智能体系统路由方面的经验，看起来像是模型选择的问题，很快就会变成一个系统优化问题。有三个维度让我们感到这出奇地困难。

1. Cost Is More Than Model Pricing

1. 成本不仅仅是模型定价

We expected GPT-4.1 to be cheaper than Claude Sonnet 4.6. It wasn’t. Across 417 tasks on the AppWorld Test Challenge using the same CodeAct agent, Sonnet cost $79 total ($0.19/task) while GPT-4.1 cost $155 ($0.37/task) — nearly double. On paper, this makes no sense. GPT-4.1’s token pricing is lower on both input and output, and Sonnet takes roughly three times as many reasoning steps to finish the same tasks. By sticker price alone, GPT-4.1 should win easily. The explanation? Caching — something most routing discussions ignore entirely. Agent workloads tend to reuse large chunks of context across steps. When cache hit rates are high, effective input costs drop dramatically. Sonnet’s lower cache-read pricing meant it benefited disproportionately from this pattern, enough to overcome both its higher base pricing and its longer trajectories. The takeaway: actual cost depends on the interaction between the model, the workload, and the serving infrastructure. A router that only looks at pricing sheets is optimizing against the wrong numbers.

我们曾预期 GPT-4.1 会比 Claude Sonnet 4.6 更便宜。但事实并非如此。在使用相同的 CodeAct 智能体对 AppWorld 测试挑战中的 417 个任务进行测试时，Sonnet 的总成本为 79 美元（每个任务 0.19 美元），而 GPT-4.1 的成本为 155 美元（每个任务 0.37 美元）——几乎是前者的两倍。从纸面上看，这毫无道理。GPT-4.1 的输入和输出 Token 定价都更低，而且 Sonnet 完成相同任务所需的推理步骤大约是前者的三倍。仅从标价来看，GPT-4.1 应该轻松胜出。原因是什么？缓存——这是大多数关于路由的讨论完全忽略的一点。智能体的工作负载往往会在不同步骤间重复使用大量的上下文。当缓存命中率很高时，实际的输入成本会大幅下降。Sonnet 更低的缓存读取定价意味着它能从这种模式中获得不成比例的收益，足以抵消其更高的基础定价和更长的推理路径。结论是：实际成本取决于模型、工作负载和服务基础设施之间的相互作用。一个只看价格表的路由器是在用错误的数字进行优化。

2. Complexity Is More Than Task Difficulty

2. 复杂性不仅仅是任务难度

A common routing strategy is to estimate how hard a task is and send harder tasks to stronger models. Intuitive, but it breaks down in two ways. First, difficulty is often invisible at routing time. A request like “summarize this contract” looks simple, but might trigger retrieval, compliance checks, tool use, and multiple rounds of refinement before it’s done. Meanwhile, a highly technical prompt might be handled efficiently by a smaller specialized model. You often don’t know how hard a task actually is until execution is underway. Second, even if you could perfectly estimate difficulty, it’s only one signal among many. In production, routers need to balance cost, latency, model specialization, and reliability simultaneously. Enterprise deployments pile on more: compliance requirements, data residency rules, privacy constraints, approved model lists. A task that would ideally go to one model might need to go elsewhere because of governance — and the router has to handle that gracefully. Routers aren’t solving one problem. They’re constantly juggling cost, quality, latency, compliance, and reliability all at once.

一种常见的路由策略是评估任务的难度，并将较难的任务发送给更强大的模型。这很直观，但在两个方面行不通。首先，难度在路由时往往是不可见的。像“总结这份合同”这样的请求看起来很简单，但在完成之前可能会触发检索、合规性检查、工具使用以及多轮优化。与此同时，一个高度技术性的提示词可能由一个较小的专用模型高效处理。你往往在执行开始之前都不知道任务到底有多难。其次，即使你能完美地估计难度，它也只是众多信号中的一个。在生产环境中，路由器需要同时平衡成本、延迟、模型专业化和可靠性。企业级部署还增加了更多要求：合规性要求、数据驻留规则、隐私约束、已批准的模型列表。一个理想情况下应该交给某个模型的任务，可能因为治理原因必须交给其他模型——而路由器必须优雅地处理这种情况。路由器解决的不是单一问题。它们必须时刻同时兼顾成本、质量、延迟、合规性和可靠性。

3. Latency Is More Than Model Speed

3. 延迟不仅仅是模型速度

It’s tempting to think about latency purely in terms of model size — bigger models are slower, smaller ones are faster. But what the user actually experiences depends on much more than that. Routing itself adds overhead. Infrastructure factors — which hardware a model is running on, whether the cache is warm, how busy the endpoint is — often dominate end-to-end response times. A theoretically faster model can still produce a slower experience if the serving conditions aren’t right. Then there’s routing granularity. Routing once per task adds minimal overhead. But routing at every step — which gives you more flexibility to adapt mid-execution — means every additional decision point introduces latency and operational complexity. A router that ignores the serving system is optimizing against the wrong reality.

人们很容易单纯从模型大小来考虑延迟——更大的模型更慢，更小的模型更快。但用户实际体验到的远不止于此。路由本身会增加开销。基础设施因素——模型运行在什么硬件上、缓存是否预热、端点有多繁忙——往往主导了端到端的响应时间。如果服务条件不理想，理论上更快的模型也可能产生更慢的体验。此外还有路由粒度的问题。每个任务路由一次带来的开销很小。但在每一步都进行路由——这虽然让你在执行过程中有更大的适应灵活性——意味着每一个额外的决策点都会引入延迟和操作复杂性。一个忽略服务系统的路由器是在错误的现实基础上进行优化。

So How Did We Handle This?

那么我们是如何处理的呢？

These lessons shaped how we built our router. The key shift: we stopped treating routing as a classification problem and started treating it as an optimization problem. Rather than asking “which model is best for this task?”, our algorithm optimizes across cost, quality, and latency simultaneously — while staying lightweight enough to avoid becoming a bottleneck itself. The figure below shows the result on the AppWorld Test Challenge with a CodeAct agent. Each blue square is a different configuration of our router, tracing out a cost-accuracy frontier. The important thing isn’t any single point — it’s that the router gives you a range of operating points to choose from depending on whether you want to prioritize cost, latency, or accuracy. Configuration 1 (latency-optimized) lands at 84% accuracy for $93 and 83s — a 21% cost reduction and 9% latency reduction compared to running Opus alone, with only a 4% accuracy drop. Configuration 2 pushes cost even lower. Notice that a standard difficulty-based router (the teal diamond) lands in a similar accuracy range but at higher cost — it doesn’t explore the full tradeoff space the way an optimization-based approach can. And because the optimization itself is lightweight (roughly 6 ms and 2 kB of memory per task), the router doesn’t become the bottleneck we warned about earlier.

这些经验塑造了我们构建路由器的方式。关键的转变是：我们不再将路由视为分类问题，而是将其视为优化问题。我们的算法不是问“哪个模型最适合这个任务？”，而是同时在成本、质量和延迟之间进行优化——同时保持轻量级，以避免自身成为瓶颈。下图显示了在 AppWorld 测试挑战中使用 CodeAct 智能体的结果。每个蓝色方块都是我们路由器的不同配置，勾勒出了一条成本-准确率边界。重要的不是某一个点，而是路由器为你提供了一系列操作点，你可以根据自己是想优先考虑成本、延迟还是准确率来进行选择。配置 1（延迟优化型）在 84% 的准确率下成本为 93 美元，耗时 83 秒——与单独运行 Opus 相比，成本降低了 21%，延迟降低了 9%，而准确率仅下降了 4%。配置 2 将成本进一步降低。请注意，标准的基于难度的路由器（青色菱形）虽然处于相似的准确率范围，但成本更高——它无法像基于优化的方法那样探索完整的权衡空间。而且由于优化本身很轻量（每个任务大约 6 毫秒和 2 kB 内存），路由器不会成为我们之前警告过的瓶颈。

The Bigger Picture

更宏观的视角

The lesson we took away from this work is that routing isn’t really about choosing models. It’s about optimizing systems. Models are one variable — an important one, but just one among caching behavior, infrastructure state, compliance constraints, and workload patterns. When routing works well, it’s rarely because it found the “best” model for a given task. It’s because it found the best operating point for the entire system. That’s a harder problem than classification, but it’s the one worth solving. We’ll be sharing more about the technical details behind our approach in a follow-up post.

我们从这项工作中得到的教训是，路由实际上并不是关于选择模型。它是关于优化系统。模型只是一个变量——一个重要的变量，但它仅仅是缓存行为、基础设施状态、合规性约束和工作负载模式中的一个。当路由工作良好时，很少是因为它为特定任务找到了“最好”的模型。而是因为它为整个系统找到了最佳的操作点。这是一个比分类更难的问题，但却是值得解决的问题。我们将在后续文章中分享更多关于我们方法背后的技术细节。