Optimal Model Activation Policies for Inference Networks of Large Language Models

大型语言模型推理网络的最佳模型激活策略

Recent advances in large language models (LLMs) have rendered them necessary for Natural Language Processing (NLP) tasks, and their high inference cost motivates the study of cost-performance trade-offs. In practice, several expert LLMs are used in synergy for inference, either in an ensemble mode or in series, yet without a principled approach on how to best use the available models. An adaptive approach can route simple queries to cheaper LLMs and complex ones to more capable, costly models. However, a clear understanding on how to best leverage available expert models is missing.

大型语言模型（LLM）的最新进展使其成为自然语言处理（NLP）任务中不可或缺的工具，而其高昂的推理成本促使人们开始研究成本与性能之间的权衡。在实践中，多个专家级 LLM 常被协同用于推理，无论是通过集成模式还是串行模式，但目前尚缺乏一种原则性的方法来最优地利用这些可用模型。一种自适应的方法可以将简单的查询路由至成本较低的 LLM，而将复杂的查询路由至能力更强、成本更高的模型。然而，目前对于如何最好地利用现有专家模型，仍缺乏清晰的理解。

We introduce inference networks, a graph-based framework, where nodes denote different LLMs, and links denote conditional model activations. The inference network design problem is to determine the best topology, namely the best way to use the models that best addresses the cost-performance trade-off. We start from the basic topology of a series of LLM experts, each of which has a different cost and a different level of expertise, which is captured via model confidence.

我们引入了“推理网络”（inference networks），这是一个基于图的框架，其中节点代表不同的 LLM，链接代表条件模型激活。推理网络的设计问题在于确定最佳拓扑结构，即以何种方式使用模型才能最好地解决成本与性能的权衡问题。我们从一系列 LLM 专家组成的基准拓扑结构出发，每个专家模型具有不同的成本和专业水平，后者通过模型置信度来衡量。

We formulate the problem of optimal activation of these models so as to minimize the expected inference cost subject to a target performance constraint. For this special class of inference networks, we prove that the optimal activation policy has a threshold structure: query the lowest-cost LLM first, and invoke the more expensive LLM only if the confidence falls below a defined threshold. For discriminative tasks, the optimal policy consists of a set of thresholds, one threshold for each class, while for generative tasks, it consists of a single threshold.

我们对这些模型的最佳激活问题进行了建模，旨在在满足目标性能约束的前提下，最小化预期推理成本。对于这类特殊的推理网络，我们证明了最佳激活策略具有阈值结构：首先查询成本最低的 LLM，仅当置信度低于设定的阈值时，才调用更昂贵的 LLM。对于判别式任务，最佳策略由一组阈值组成（每个类别一个阈值）；而对于生成式任务，它则由单一阈值组成。

We provide a structured method to compute the thresholds, and practical confidence estimation mechanisms for both task types. Experiments with open-source LLMs show substantial cost reductions while meeting the specified performance budget.

我们提供了一种计算这些阈值的结构化方法，并针对上述两种任务类型提出了实用的置信度评估机制。使用开源 LLM 进行的实验表明，该方法在满足指定性能预算的同时，能够显著降低推理成本。