BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems

BBOWP-Bench：评估大语言模型在黑盒优化应用题中的表现

Abstract: Formulating an optimization problem strongly affects the quality of the final solution, yet good formulations usually require substantial expertise. Recent studies have therefore examined how to automatically derive optimization problems from natural-language descriptions, but existing benchmarks focus on settings where objectives and constraints can be written explicitly as mathematical expressions.

摘要： 优化问题的建模方式对最终解的质量有着深远影响，但高质量的建模通常需要丰富的专业知识。因此，近期的研究开始探索如何从自然语言描述中自动推导优化问题；然而，现有的基准测试主要集中在目标函数和约束条件可以明确写成数学表达式的场景。

Many practically important problems are naturally treated as black-box optimization (BBO) problems, in which only objective values are observable, and the functional form is unavailable. In BBO, the search space design, a part of the problem formulation, and the selection of the optimization algorithm are crucial for problem-solving. Automating these processes with large language models (LLMs) is a significant challenge.

许多具有实际意义的重要问题本质上属于黑盒优化（BBO）问题，即只能观测到目标函数值，而无法获得其函数形式。在黑盒优化中，作为问题建模一部分的搜索空间设计以及优化算法的选择，对于解决问题至关重要。利用大语言模型（LLMs）实现这些过程的自动化是一项重大挑战。

This paper introduces Black-Box Optimization Word Problems (BBOWP), a novel problem setting in which a system must infer both a search space and an optimization algorithm from a natural-language description of a black-box optimization task. To support research on this setting, we establish the BBOWP Benchmark Suite (BBOWP-Bench), a dataset and evaluation framework for BBOWP. Each instance combines a natural-language problem description, an executable evaluation environment, and a human-designed baseline formulation, allowing evaluation of both search-space design and algorithm selection.

本文引入了“黑盒优化应用题”（BBOWP），这是一种全新的问题设定，要求系统根据黑盒优化任务的自然语言描述，同时推断出搜索空间和优化算法。为了支持该领域的研究，我们建立了 BBOWP 基准测试套件（BBOWP-Bench），这是一个针对 BBOWP 的数据集和评估框架。每个实例都结合了自然语言问题描述、可执行的评估环境以及人工设计的基准建模，从而能够对搜索空间设计和算法选择进行综合评估。

Using this benchmark, we provide the first evaluation of LLMs and show that current LLMs are capable of selecting suitable algorithms based on the given evaluation budget. However, they sometimes struggle with search space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific. Our code and dataset are available at this https URL.

利用该基准测试，我们对大语言模型进行了首次评估，结果表明当前的大语言模型能够根据给定的评估预算选择合适的算法。然而，它们在搜索空间设计方面有时会遇到困难，特别是在问题描述信息不足或搜索空间具有高度特定性时，模型在识别重要变量及平衡变量范围方面表现欠佳。我们的代码和数据集已在指定链接发布。