A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
大模型服务的一年：工作负载演变、缓存与负载均衡
Abstract: Large Language Model (LLM) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving systems. However, existing LLM serving workload studies remain limited in scale and scope. They often observe short time periods and provide limited visibility into how users interact with models in production. As a result, they do not fully capture how LLM serving workloads evolve over time or how user-model interactions shape production traffic.
摘要： 大语言模型（LLM）服务已成为关键的云工作负载，而真实的工作负载追踪数据对于推动和基准测试服务系统至关重要。然而，现有的 LLM 服务工作负载研究在规模和范围上仍然有限。它们通常仅观察较短的时间周期，且对于用户如何在生产环境中与模型交互的可见性不足。因此，这些研究未能全面捕捉 LLM 服务工作负载随时间的演变过程，也未能揭示用户与模型的交互如何塑造生产流量。
In this work, we further the understanding of real-world LLM serving workloads through both a global characterization and a longitudinal study of a one-year production trace from Chutes. Unlike prior studies, our trace captures full production behavior across many models and users, including both popular and long-tail models. We analyze the workload from aggregate, temporal, model-level, and user-level perspectives, revealing workload evolution and user-model structure that are typically hidden behind aggregate views.
在这项工作中，我们通过对 Chutes 平台一年期生产追踪数据的全局表征和纵向研究，进一步加深了对真实世界 LLM 服务工作负载的理解。与以往的研究不同，我们的追踪数据捕捉了跨多个模型和用户的完整生产行为，涵盖了热门模型和长尾模型。我们从聚合、时间、模型级和用户级等多个维度分析了工作负载，揭示了通常隐藏在聚合视图背后的工作负载演变规律和用户-模型结构。
To support future research, we will release the full one-year trace with the paper, enabling downstream studies of production behavior without relying on sampled or synthetically generated workloads.
为了支持未来的研究，我们将随论文发布完整的年度追踪数据，从而使后续研究能够基于真实生产行为进行，而无需依赖采样或合成生成的工作负载。
Paper Details:
- Authors: William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, Juncheng Yang
- arXiv ID: 2608.13573
- Subject: Artificial Intelligence (cs.AI)
- Submission Date: 3 Jul 2026
论文详情：
- 作者： William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, Juncheng Yang
- arXiv ID: 2608.13573
- 学科： 人工智能 (cs.AI)
- 提交日期： 2026年7月3日