Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
干预式基础审计：通过谓词替换对大语言模型思维链进行黑盒前提依赖性测试
Large language models produce chain-of-thought (CoT) reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step’s normalized conclusion (canonical predicate form) changes.
大语言模型生成的思维链（CoT）推理看起来逻辑严密，但可能并非真正依赖于其陈述的前提。我们引入了“干预式基础审计”（Interventional Grounding Audits），这是一种针对前提依赖性的黑盒、步骤级测试方法：我们通过将单个前提中的目标谓词替换为一个新符号来进行干预，重新运行模型，并检查每个推理步骤的归一化结论（规范谓词形式）是否发生变化。
We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies (F1 = 0.885 on predicate-determining dependencies; Recall = 100%), significantly outperforming a self-consistency baseline (F1 = 0.343; 95% bootstrap CIs non-overlapping).
我们在 ProntoQA 上进行了评估，这是一个包含黄金证明树的合成多跳演绎推理基准，其中步骤级的前提依赖关系是已知的。将该方法应用于 GPT-4o 处理的 50 个 ProntoQA 问题时，我们在检测证明树依赖关系方面达到了 F1 = 0.806（在谓词确定依赖关系上 F1 = 0.885；召回率 = 100%），显著优于自洽性基准（F1 = 0.343；95% 自助法置信区间无重叠）。
We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution — all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator — a “right answer, wrong reasoning” signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.
我们进一步发现，66% 的正确解答问题中至少包含一个在一致性替换下对直接证明树依赖不敏感的对齐步骤——这些步骤全部涉及实体引入前提，这是一致性替换评估器中已知的盲点——这是一种被动方法无法察觉的“答案正确，推理错误”信号。所有审计证书、原始输出和复现脚本均可在公开的 GitHub 仓库中获取，我们还讨论了超出形式化、可解析基准之外的适用范围限制。