InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

InferenceBench：AI 智能体开放式大模型推理优化基准测试

AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces. Even nominally open-ended tasks can often be solved by retrieving a well-known recipe and tuning a few hyperparameters, making it unclear whether strong results reflect genuine optimization or memorized solutions.

AI 智能体正越来越多地被用于自动化研发任务，然而现有的基准测试通常仅在预设的工作流或狭窄的动作空间内对其进行评估。即使是名义上的开放式任务，往往也可以通过检索已知的方案并调整几个超参数来解决，这使得人们难以判断其优异表现究竟是源于真正的优化能力，还是仅仅是对已知方案的机械记忆。

We introduce InferenceBench, where an agent must deploy an OpenAI-compatible inference server and optimize the speed of LLM inference. Each agent receives a target LLM, one H100 GPU, an optimization scenario, and a wall-clock time budget of two hours. Three optimization scenarios isolate distinct bottlenecks of inference (prefill latency, decode latency, and concurrent request throughput) and a fourth balances all three at the same time.

我们推出了 InferenceBench，要求智能体部署一个兼容 OpenAI 的推理服务器，并优化大模型的推理速度。每个智能体都会获得一个目标大模型、一块 H100 GPU、一个优化场景以及两小时的实际运行时间预算。其中三个优化场景分别针对推理的不同瓶颈（预填充延迟、解码延迟和并发请求吞吐量），第四个场景则要求同时平衡这三者。

Across 15 frontier agent configurations, agents reliably improve over a naive PyTorch baseline (up to $8.08\times$) and often match or exceed serving engines with default settings ($4.05\times$ for vLLM), but still fall below a simple hyperparameter search under the same time budget (up to $11.53\times$).

在 15 种前沿智能体配置的测试中，智能体能够稳定地超越原始 PyTorch 基准（最高提升 8.08 倍），并经常达到或超过使用默认设置的推理引擎（如 vLLM 的 4.05 倍），但其表现仍不及在相同时间预算下进行的简单超参数搜索（最高可达 11.53 倍）。

Qualitative analysis of agent trajectories shows that although agents enumerate many relevant optimization techniques, they overwhelmingly converge on a single inference framework. They test only a few distinct configurations and spend the remaining budget re-measuring, repairing, or optimizing hyperparameters rather than exploring substantially different strategies. This suggests the bottleneck is not domain knowledge, but the ability to propose diverse configurations, evaluate them systematically, and submit the best identified solution.

对智能体运行轨迹的定性分析表明，尽管智能体能够列举出许多相关的优化技术，但它们绝大多数都收敛于单一的推理框架。它们仅测试了少数几种不同的配置，并将剩余的时间预算花费在重复测量、修复或优化超参数上，而不是探索本质上不同的策略。这表明瓶颈不在于领域知识，而在于提出多样化配置、进行系统性评估并提交最优解的能力。

Overall, InferenceBench reflects the ability of agents to operate in an open-ended AI engineering setting, where memorized solutions lead to limited improvements.

总的来说，InferenceBench 反映了智能体在开放式 AI 工程环境中的运作能力，在这种环境下，单纯依赖记忆的方案只能带来有限的性能提升。