本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Designing an AI serving harness built to last

SponsoredProvided byEXL The next competitive advantage in enterprise AI has less to do with which model you choose than with what you build around it. That’s an important shift. For the past several years, much of the AI conversation has centered on model capability: bigger models, better reasoning, larger context windows, higher benchmark scores. Those advances have been extraordinary. But as AI moves from experimentation to production, the question facing enterprise leaders has changed—it’s no longer, “what can the model do?”—it’s now, “Can we deliver that intelligence reliably, economically, and at scale?” Increasingly, the answer depends on the serving harness—the engineering system surrounding the model, including the architecture, memory management, scheduling, and optimization decisions that determine how efficiently model intelligence becomes a production capability. For technical executives, that changes where AI investment should go—and what they should demand from the architecture supporting it.

企业人工智能的下一个竞争优势与其说取决于你选择了哪种模型，不如说取决于你围绕它构建了什么。这是一个重要的转变。在过去几年里，关于人工智能的大部分讨论都集中在模型能力上：更大的模型、更好的推理能力、更大的上下文窗口以及更高的基准测试分数。这些进步是非凡的。但随着人工智能从实验阶段走向生产阶段，企业领导者面临的问题已经改变——不再是“模型能做什么？”，而是“我们能否可靠、经济且大规模地交付这种智能？”答案越来越依赖于服务框架（serving harness）——即围绕模型构建的工程系统，包括架构、内存管理、调度和优化决策，这些决定了模型智能转化为生产能力的效率。对于技术高管而言，这改变了人工智能投资的方向，以及他们对支撑架构的要求。

Start with the workload, not the model There is no universally “best” model or AI architecture. There is only the right solution for the business problem. A customer-facing assistant optimized for fast responses has fundamentally different requirements from a document-processing pipeline that can run overnight. An internal knowledge application and a high-volume customer service platform may use the same foundation model but require very different serving architectures. The workload defines the constraints. The constraints should determine the architecture. That sounds simple, but it runs counter to the way many organizations have approached early AI deployments. During a proof of concept, model accuracy naturally dominates the discussion. Production introduces constraints that are much easier to ignore when only a handful of users are involved: latency, concurrency, memory pressure, GPU utilization, infrastructure cost, and unpredictable traffic. A model that produces marginally better answers is not necessarily the better production choice if it doubles inference costs or cannot meet the application’s latency requirements. Not every workload needs a Ferrari. Sometimes the smarter engineering decision is the model that delivers exactly the performance the business needs without paying for capability it doesn’t. The objective should be to deploy the right intelligence at the right cost for the workload. That makes serving architecture more than an engineering concern. It becomes an investment decision: where to spend on compute, where to optimize, and where additional model capability simply isn’t worth the cost.

从工作负载入手，而非模型。不存在普遍意义上“最好”的模型或人工智能架构，只有针对特定业务问题的正确解决方案。一个为快速响应而优化的面向客户的助手，其需求与可以隔夜运行的文档处理流水线有着根本的不同。内部知识应用和高容量客户服务平台可能使用相同的基准模型，但需要截然不同的服务架构。工作负载定义了约束条件，而约束条件应决定架构。这听起来很简单，但与许多组织处理早期人工智能部署的方式背道而驰。在概念验证期间，模型准确性自然主导了讨论。而生产环境引入了在用户较少时容易被忽视的约束：延迟、并发性、内存压力、GPU利用率、基础设施成本和不可预测的流量。如果一个模型虽然能产生稍微好一点的答案，但却使推理成本翻倍或无法满足应用的延迟要求，那么它未必是更好的生产选择。并非每个工作负载都需要“法拉利”。有时，更明智的工程决策是选择一个能精准提供业务所需性能的模型，而不是为不需要的能力买单。目标应该是以合适的成本为工作负载部署合适的智能。这使得服务架构不仅仅是一个工程问题，更成为了一项投资决策：在何处投入计算资源，在何处进行优化，以及在何处额外的模型能力不值得投入成本。

Engineer the system, not the model Once AI reaches production, seemingly isolated engineering decisions begin affecting one another. Serving architecture influences memory utilization. Memory management affects concurrency. Concurrency shapes batching and scheduling, and scheduling affects GPU utilization and latency. Only then can techniques such as quantization, speculative decoding, or optimized attention mechanisms deliver their full value. This interdependence matters because organizations can spend considerably more on infrastructure without solving the constraint limiting performance. GPU memory is a good example. During inference, the model’s weights remain fixed, but the key-value (KV) cache grows with active requests and context length. As concurrency increases, a system can run short of usable GPU memory even when compute capacity is still available. Adding more compute without understanding that bottleneck may simply make an inefficient system more expensive. The lesson for technology leaders is broader than memory management: Measure before you optimize. Identify the dominant constraint. Address it. Measure the result. Then determine if another bottleneck emerged. Repeat. Don’t think of optimization techniques as Pokémon cards: The goal isn’t to collect them all. The goal is to apply the technique that removes the highest-impact constraint while preserving the application’s business requirements. For technology leaders, that demands a different governance question. Instead of asking which AI capabilities the organization can deploy, ask which constraints stand between a proven capability and sustainable business value.

构建系统，而非模型。一旦人工智能进入生产阶段，看似孤立的工程决策就会开始相互影响。服务架构影响内存利用率，内存管理影响并发性，并发性塑造批处理和调度，而调度又影响GPU利用率和延迟。只有这样，量化、推测解码或优化注意力机制等技术才能发挥其全部价值。这种相互依赖性至关重要，因为组织可能会在基础设施上花费更多，却无法解决限制性能的瓶颈。GPU内存就是一个很好的例子。在推理过程中，模型权重保持不变，但键值（KV）缓存会随着活跃请求和上下文长度的增加而增长。随着并发性增加，即使计算能力尚有剩余，系统也可能出现可用GPU内存不足的情况。如果不了解这一瓶颈而盲目增加计算资源，只会让低效的系统变得更加昂贵。对于技术领导者来说，经验教训不仅仅在于内存管理：先测量，再优化。识别主要约束，解决它，测量结果，然后确定是否出现了新的瓶颈，如此循环。不要把优化技术看作宝可梦卡片：目标不是收集所有卡片，而是应用能消除最高影响约束的技术，同时保持应用的业务需求。对于技术领导者而言，这需要一个不同的治理问题。不要问组织可以部署哪些人工智能能力，而要问哪些约束阻碍了已验证的能力转化为可持续的业务价值。

Build for change, not today’s benchmark Perhaps the most important reason to invest in the serving harness is that today’s leading model will probably not be tomorrow’s. Models will improve. Hardware will evolve. New optimization techniques will emerge. The architecture surrounding them should make those changes easier to absorb, not force the enterprise to rebuild its AI platform every time technology advances. That makes adaptability an increasingly important outcome of AI strategy. Organizations that build serving systems around workload requirements, clear service-level objectives, and measurable production economics gain the ability to evaluate new models and infrastructure against what the business actually needs. A model change can become an evaluation and configuration decision rather than an architectural reset. EXL researchers have explored these production challenges in greater depth in the EXL Engineering the Harness series. It examines how serving architecture, memory management, scheduling, and inference optimization work together to make enterprise AI more efficient, scalable, and economically sustainable. Increasingly, enterprises have access to many of the same models. Sustainable differentiation comes from how effectively an organization turns that intelligence into business outcomes. The model provides the intelligence. The harness determines whether the enterprise can deliver outcomes reliably, economically, and at scale. This content was produced by EXL. It was not written by MIT Technology Review’s editorial staff.

为变革而构建，而非为今天的基准而构建。投资服务框架最重要的原因或许是：今天的领先模型很可能不会是明天的领先模型。模型会改进，硬件会演进，新的优化技术会涌现。围绕它们构建的架构应该使这些变化更容易被吸收，而不是迫使企业在每次技术进步时都重建其人工智能平台。这使得适应性成为人工智能战略中日益重要的成果。那些围绕工作负载需求、明确的服务水平目标和可衡量的生产经济效益来构建服务系统的组织，能够根据业务的实际需求来评估新的模型和基础设施。模型变更可以成为一种评估和配置决策，而不是架构上的重置。EXL的研究人员在《EXL Engineering the Harness》系列中更深入地探讨了这些生产挑战。该系列研究了服务架构、内存管理、调度和推理优化如何协同工作，使企业人工智能更高效、可扩展且在经济上可持续。越来越多的企业能够获得许多相同的模型。可持续的差异化优势来自于组织将这些智能转化为业务成果的有效性。模型提供智能，而框架决定了企业能否可靠、经济且大规模地交付成果。本文内容由EXL制作，非《麻省理工科技评论》编辑部撰写。