Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata
Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata
超越能力基准：从生产事故元数据中学习大模型云服务的运行指纹
Abstract: Managed LLM services are now part of real production systems, but model selection and service planning still rely heavily on capability benchmarks that reveal little about operational behavior after deployment. 摘要： 托管式大语言模型（LLM）服务现已成为实际生产系统的一部分，但模型选择和服务规划仍然严重依赖能力基准测试，而这些测试对于部署后的运行行为几乎无法提供参考。
We present Operational Embedding (OpEmbed), a framework for learning compact operational fingerprints of LLM cloud services from structured, privacy-preserving support-case metadata, without using case text. 我们提出了“运行嵌入”（Operational Embedding, OpEmbed）框架，该框架能够从结构化且保护隐私的支持案例元数据中学习大模型云服务的紧凑型运行指纹，且无需使用案例文本内容。
OpEmbed aggregates model—time windows into an eight-channel operational signature and learns a low-dimensional representation via temporal contrastive learning, cross-view reconstruction, and generational-ordinality regularization. OpEmbed 将模型与时间窗口聚合为八通道运行特征，并通过时间对比学习、跨视图重构以及代际序数正则化来学习低维表示。
Evaluated on more than 33,000 production support cases spanning seven LLM families over 26 months at Google Cloud, OpEmbed recovers interpretable family- and version-level structure, improves leave-one-model-out operational forecasting over non-learned baselines, remains useful under limited early-window data, and supports cross-model fault-type transfer. 通过对 Google Cloud 26 个月内涵盖七个大模型家族的 33,000 多起生产支持案例进行评估，OpEmbed 成功恢复了具有可解释性的家族级和版本级结构，在“留一模型法”（leave-one-model-out）运行预测任务中优于非学习基准模型，在早期窗口数据有限的情况下依然有效，并支持跨模型的故障类型迁移。
We report the practical lessons learned from building and evaluating this tool for model onboarding, support readiness assessment, and operational monitoring. 我们报告了在构建和评估该工具用于模型入驻、支持就绪度评估及运行监控过程中所获得的实践经验。