MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization

MAGE：理解多组件提示词优化中的稳定性与性能权衡

Abstract: How do different components of iterative prompt optimization interact, and what happens when they are combined? We investigate this through MAGE (Memory-Augmented Goal-directed Prompt Evolution), a controlled analysis framework for studying component interaction in prompt optimization. MAGE is not proposed as a superior optimizer in absolute terms; it integrates episodic memory, multi-objective Pareto selection, and adaptive evaluation as a platform for controlled ablation.

摘要： 迭代提示词优化中的不同组件是如何相互作用的？当它们结合在一起时会发生什么？我们通过 MAGE（记忆增强目标导向提示词演化）对此进行了研究，这是一个用于研究提示词优化中组件交互的受控分析框架。MAGE 并非旨在成为绝对意义上的最优优化器；它集成了情景记忆、多目标帕累托选择和自适应评估，作为一个受控消融研究的平台。

Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE): when multiple stochastic optimization signals operate within a closed reflective loop, they interact in ways that simultaneously improve performance and amplify variance, behavior that cannot be predicted by analyzing components in isolation.

我们的实验揭示了一种此前未被报道的现象——提示词优化耦合效应（POCE）：当多个随机优化信号在闭环反射回路中运行时，它们会以某种方式相互作用，在提升性能的同时放大方差，这种行为无法通过孤立地分析各个组件来预测。

Three main findings emerge. First, failure-grounded reflection is essential: methods relying only on scores (OPRO) or abstract critique (Self-Refine) fail to improve prompts. Second, MAGE achieves 46.4% versus GEPA’s 34.0% on GSM8K-Hard (+12.4%, P(MAGE>GEPA)=0.998, 5 seeds on gpt-4o-mini), with comparable variance (7.3% vs. 7.0%). Third, increasing candidate diversity reveals the clearest POCE signal: expanding the candidate pool from n=3 to n=5 improves mean accuracy by +21.6% while increasing variance by 3.7x.

研究得出了三个主要发现。首先，基于失败的反射至关重要：仅依赖分数（如 OPRO）或抽象批评（如 Self-Refine）的方法无法有效改进提示词。其次，在 GSM8K-Hard 测试集上，MAGE 达到了 46.4% 的准确率，而 GEPA 为 34.0%（提升了 12.4%，P(MAGE>GEPA)=0.998，在 gpt-4o-mini 上运行 5 次随机种子），且方差相当（7.3% 对比 7.0%）。第三，增加候选多样性揭示了最明显的 POCE 信号：将候选池从 n=3 扩大到 n=5，平均准确率提高了 21.6%，但方差增加了 3.7 倍。

We further validate on Llama 3.1 8B and show POCE is headroom-dependent: when the base model already achieves high accuracy, variance amplification disappears. Finally, in low-data regimes (Ntrain=30), well-designed fixed prompts outperform all reflective optimizers, indicating that scaffold choice dominates optimizer choice. Our results suggest prompt optimization systems behave as coupled stochastic processes and should be evaluated in terms of both performance and stability, not just peak accuracy.

我们进一步在 Llama 3.1 8B 上进行了验证，并表明 POCE 具有“上限依赖性”：当基础模型已经达到高准确率时，方差放大现象就会消失。最后，在低数据量环境下（Ntrain=30），精心设计的固定提示词优于所有反射式优化器，这表明框架（scaffold）的选择比优化器的选择更重要。我们的研究结果表明，提示词优化系统表现为耦合的随机过程，因此在评估时应同时考虑性能和稳定性，而不仅仅是峰值准确率。