Automatically Evolving Prompt Guidelines for Task-Specific Optimization

自动演化针对特定任务优化的提示词指南

Abstract: For Large Language Models to reliably answer user queries, users must clearly specify requirements, context, and constraints. In practice, however, user queries are often underspecified, forcing models to infer unstated assumptions that may misalign with the actual user intent. Existing prompt engineering guidelines aim to mitigate this issue, they are typically generic and task-agnostic, limiting their practical utility. Additionally, existing guidelines are formed manually and in a non-systematic way.

摘要： 为了让大语言模型能够可靠地回答用户查询，用户必须清晰地说明需求、背景和约束条件。然而在实践中，用户的查询往往描述不足，迫使模型去推断那些可能与用户真实意图不符的隐含假设。现有的提示词工程指南旨在缓解这一问题，但它们通常是通用的且与任务无关，这限制了其实际效用。此外，现有的指南大多是通过人工且非系统化的方式形成的。

To this end, we study prompt guideline optimization: the problem of automatically generating task-specific guidelines that help write better-specified prompts for a given task and model. Our key observation is that existing (completed) task examples (aka reference answers) often implicitly encode the missing information required to complete underspecified queries, including behavioral constraints, contextual assumptions, and evaluation criteria.

为此，我们研究了提示词指南优化问题：即如何自动生成针对特定任务的指南，以帮助用户为给定的任务和模型编写描述更准确的提示词。我们的核心观察是，现有的（已完成的）任务示例（即参考答案）往往隐含地编码了完成描述不足的查询所需的信息，包括行为约束、背景假设和评估标准。

We therefore propose AGOPS, an automatic approach that evolves task-specific guidelines via an optimization scheme that involves a prompt LLM writer, a solver LLM and prompt evolution, which maximize downstream effectiveness on a set of examples (user queries with reference answers). At inference time, our guidelines help users write well-specified prompts, boosting the effectiveness of LLMs.

因此，我们提出了 AGOPS，这是一种自动化的方法，通过包含提示词编写 LLM、求解器 LLM 和提示词演化机制的优化方案来演化特定任务的指南，从而在一组示例（包含参考答案的用户查询）上最大化下游任务的效果。在推理阶段，我们的指南能够帮助用户编写描述准确的提示词，从而提升大语言模型的效果。

We show across mathematical reasoning, medical question answering, and coding tasks, that prompt underspecification leads to major drops (up to 95.3%) in downstream task performance (compared to well-specified prompts) and, perhaps more importantly, that this drop can hardly be recovered by existing prompt optimization techniques. Users following AGOPS guidelines can regain this loss (increasing performance between 15.5 to 81.7% on average) consistently across all benchmarks.

我们在数学推理、医学问答和编程任务中证明，提示词描述不足会导致下游任务性能大幅下降（与描述准确的提示词相比，降幅高达 95.3%），更重要的是，这种性能损失很难通过现有的提示词优化技术来恢复。遵循 AGOPS 指南的用户可以在所有基准测试中持续挽回这一损失（平均性能提升 15.5% 到 81.7%）。