FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

FineServe：全球大语言模型服务负载的细粒度数据集与特征分析

Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge. Achieving low latency and high throughput under volatile demand requires deep understanding of real-world serving workloads, yet existing studies often rely on proxy traces or coarse-grained characterizations that fail to capture the heterogeneity of modern multi-model LLM platforms.

大语言模型（LLM）正越来越多地作为全天候在线服务进行部署，这使得高效的 LLM 服务成为一项关键的系统挑战。要在波动的需求下实现低延迟和高吞吐量，需要对现实世界的服务负载有深刻的理解；然而，现有的研究往往依赖于代理追踪或粗粒度的特征描述，无法捕捉现代多模型 LLM 平台的异构性。

We present FineServe, an in-the-wild, multi-model LLM serving workload dataset collected from a global commercial marketplace, enabling fine-grained characterization of real-world serving dynamics across heterogeneous models and tasks. Leveraging FineServe, we conduct a comprehensive analysis of arrival dynamics and token behavior, revealing fundamentally different fluctuation regimes across model architectures, scales and task intents.

我们提出了 FineServe，这是一个从全球商业市场收集的、处于真实环境下的多模型 LLM 服务负载数据集，能够对跨异构模型和任务的真实服务动态进行细粒度特征分析。利用 FineServe，我们对请求到达动态和 Token 行为进行了全面分析，揭示了不同模型架构、规模和任务意图之间存在根本不同的波动机制。

Building on these insights, we develop the FineServe workload generator, which composes fine-grained model-aware workloads into configurable mixtures tailored for benchmarking multi-model serving platforms. By exposing these fine-grained workload dynamics, FineServe provides a realistic foundation for evaluating routing, scheduling, and capacity-planning strategies in LLM serving systems. FineServe is available at this https URL.

基于这些见解，我们开发了 FineServe 负载生成器，它将细粒度的、具备模型感知能力的负载组合成可配置的混合负载，专为基准测试多模型服务平台而设计。通过揭示这些细粒度的负载动态，FineServe 为评估 LLM 服务系统中的路由、调度和容量规划策略提供了现实的基础。FineServe 可通过此链接获取。