Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
并非所有 Token 都生而平等：面向智能体 LLM 系统的通胀感知路由策略
Abstract: When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time. This retry overhead creates a gap between what a model’s per-token price implies and what a full workflow actually costs. We call this gap \emph{token inflation} and define it as the ratio of true workflow cost to single-call cost.
摘要： 当语言模型无法在第一次尝试时回答查询时，智能体系统会进行重试，每次重试都会消耗额外的 Token。这种重试开销导致了模型单 Token 价格所暗示的成本与完整工作流实际成本之间的差距。我们将这种差距称为“Token 通胀”（token inflation），并将其定义为真实工作流成本与单次调用成本之比。
Systems like FrugalGPT route based on the latter, which can underestimate real cost by more than $2\times$ on difficult tasks. We address this with InflationAgent, a four-stage router that (1) measures token inflation systematically across model tiers and task types, finding inflation as high as $4.25\times$ for a 7B 参数 model on multi-hop question answering; (2) introduces CoT Branching Entropy (CBE), a pre-execution difficulty signal computed entirely from local inference, which predicts high inflation with AUROC 0.887; and (3) selects models by maximizing a Semantic Exchange Rate (SER) that divides expected accuracy by predicted true cost, with a fresh-escalation policy that discards failed chains before routing to a stronger model.
像 FrugalGPT 这样的系统基于后者进行路由，这在处理困难任务时可能会低估超过 2 倍的实际成本。我们通过 InflationAgent 解决了这个问题，这是一个四阶段路由系统，它：(1) 系统地测量了不同模型层级和任务类型下的 Token 通胀情况，发现在多跳问答任务中，7B 参数模型的通胀率高达 4.25 倍；(2) 引入了思维链分支熵（CoT Branching Entropy, CBE），这是一种完全通过本地推理计算出的预执行难度信号，能够以 0.887 的 AUROC 预测高通胀；(3) 通过最大化“语义汇率”（Semantic Exchange Rate, SER）来选择模型，该指标将预期准确率除以预测的真实成本，并采用一种“全新升级策略”（fresh-escalation policy），在将任务路由到更强大的模型之前丢弃失败的推理链。
On GSM8K under a fixed budget, InflationAgent achieves 94.7% accuracy versus 91.0% for FrugalGPT while using 31% fewer tokens, and we show that forwarding a failed reasoning chain to GPT-4o reduces its accuracy by up to 34.8 percentage points, validating the fresh-escalation design.
在固定预算下的 GSM8K 测试中，InflationAgent 达到了 94.7% 的准确率，而 FrugalGPT 为 91.0%，同时 InflationAgent 的 Token 消耗减少了 31%。此外，我们证明了将失败的推理链直接转发给 GPT-4o 会使其准确率下降高达 34.8 个百分点，从而验证了“全新升级策略”设计的有效性。