Counterexamples as Feedback for Agent Self-Correction

反例作为智能体自我修正的反馈机制

Abstract: Single-turn code-generation metrics understate a central property of deployed agents: whether they can repair a wrong artifact after receiving concrete feedback. This paper presents A-CEGIS, a lightweight framework that uses counterexamples as feedback for evaluating multi-turn refinement in natural-language-to-regex synthesis.

摘要： 单轮代码生成指标低估了已部署智能体的一个核心属性：即它们在收到具体反馈后是否能够修复错误的产物。本文提出了 A-CEGIS，这是一个轻量级框架，利用反例作为反馈，用于评估自然语言到正则表达式（NL-to-regex）合成中的多轮优化过程。

An agent proposes a regex, a deterministic oracle checks it under full-match semantics, and compact false-positive or false-negative witnesses guide the next turn. On 30 NL-RX-Turk tasks, diagnostic counterexample feedback solves 90% of tasks within a four-turn ablation budget, compared with 17% for zero-shot generation, 27% for generic self-correction, and 23% for error-only feedback.

智能体提出一个正则表达式，确定性预言机（deterministic oracle）在全匹配语义下对其进行检查，随后紧凑的假阳性或假阴性见证（witnesses）将引导下一轮的修正。在 30 个 NL-RX-Turk 任务中，诊断性反例反馈在四轮消融预算内解决了 90% 的任务，相比之下，零样本生成（zero-shot）的解决率为 17%，通用自我修正为 27%，仅错误反馈为 23%。

In a full diagnostic run with hardening, all tasks are solved on the hidden set by the final turn, with mean time-to-success of 2.7 turns and robust success of 77% after targeted probing. These results show that A-CEGIS measures how efficiently an agent improves across turns while adding a practical robustness check beyond the original held-out cases.

在包含加固（hardening）的完整诊断运行中，所有任务在最后一轮时均在隐藏测试集上得到解决，平均成功轮数为 2.7 轮，在定向探测后的稳健成功率为 77%。这些结果表明，A-CEGIS 不仅衡量了智能体在多轮交互中的改进效率，还提供了一种超越原始留出测试集的实用稳健性检查方法。

---

Paper Details:

• Authors: Sidhesh Badrinarayan, Adithya Parthasarathy
• arXiv ID: 2609.02892
• Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

论文详情：

• 作者： Sidhesh Badrinarayan, Adithya Parthasarathy
• arXiv ID: 2609.02892
• 学科分类： 计算与语言 (cs.CL)；人工智能 (cs.AI)