Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

超越提示工程：提示词词汇敏感度及其对质量影响的系统性分析

Large Language Models (LLMs) exhibit extreme sensitivity to surface-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations. 大型语言模型（LLMs）对表层提示词的变化表现出极端的敏感性，微小的词汇变动往往会引发不成比例的性能波动。

Moving beyond black-box optimization and coarse-grained templates, we present the first large-scale, n-gram token-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants. 为了超越黑盒优化和粗粒度模板，我们利用包含 132,000 个提示词变体的数据集，首次对提示词稳定性进行了大规模、基于 n-gram 词元（token）层面的机制分析。

Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation. 我们的研究揭示了提示词性能稳定性的基本缩放定律（Scaling Law）：更高的平均任务性能与更低的方差以及在提示词扰动下更强的鲁棒性密切相关。

We identify two core linguistic drivers underlying this robustness: (1) Domain-Specific Terminology, which tightly anchors semantic boundaries, and (2) Explicit Action Directives, which formalize reasoning trajectories. 我们确定了支撑这种鲁棒性的两个核心语言驱动因素：（1）领域特定术语，它能紧密锚定语义边界；（2）明确的行动指令，它能规范化推理轨迹。

Together, these elements constrain the model’s interpretative space, effectively “locking in” more deterministic generation behavior. 这些要素共同约束了模型的解释空间，有效地“锁定”了更具确定性的生成行为。

Building on these insights, we introduce an automated Prompt-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints. 基于这些见解，我们引入了一个自动化的提示词优化智能体（Prompt-Refining Agent），通过注入领域锚定和操作约束，系统地重构输入查询。

Empirical evaluation shows that our approach reduces performance variance by 40.7% in code generation task, while preserving or improving mean performance. 实证评估表明，我们的方法在代码生成任务中将性能方差降低了 40.7%，同时保持或提升了平均性能。

These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering. 这些发现为实现稳健的提示工程提供了一个具有统计学基础且机制上可解释的框架。