What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting
What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting
是什么驱动了大型语言模型的自我反思？武装冲突预测中不确定性路由的受控消融研究
Abstract: Self-reflection is widely assumed to improve LLM reasoning, yet which component drives the gain remains poorly understood. We present a controlled six-condition ablation isolating four components of LLM self-reflection: evidence exposure, diagnostic scaffolding, taxonomy vocabulary, and action routing.
摘要： 人们普遍认为自我反思能提升大型语言模型（LLM）的推理能力，但究竟是哪一部分组件驱动了这种提升，目前尚不明确。我们提出了一项包含六种条件的受控消融研究，旨在分离出 LLM 自我反思的四个组件：证据暴露、诊断框架、分类词汇表和行动路由。
Two precise null results converge on a single mechanism. First, structured diagnostic questions add no measurable value over unstructured reflection ($\text{F1} = 0.296$ vs $0.297$, $p = 1.000$, 95% CI $[-0.041, +0.040]$). Second, presenting the full uncertainty taxonomy while collapsing the action space to a single generic action also adds no value ($\Delta\text{F1} = +0.008$, overlapping 95% CIs), ruling out taxonomy vocabulary as the mechanism.
两项精确的零结果指向了同一个机制。首先，结构化的诊断问题相比非结构化的反思并没有带来可衡量的价值（$\text{F1} = 0.296$ 对比 $0.297$，$p = 1.000$，95% 置信区间 $[-0.041, +0.040]$）。其次，在展示完整的不确定性分类体系的同时，将行动空间压缩为单一的通用行动，也同样没有产生价值（$\Delta\text{F1} = +0.008$，95% 置信区间重叠），这排除了分类词汇表作为驱动机制的可能性。
Typed action routing provides consistent directional gains ($\text{F1} = 0.379$ vs $0.296$); the conservative estimate controlling for taxonomy vocabulary is $\Delta\text{F1} = +0.075$, and the overall gain over the single-shot baseline is significant by bootstrap CI ($\Delta\text{F1} = +0.101$, 95% CI $[+0.020, +0.185]$). The vocabulary-routing decomposition replicates on GPT-4o: taxonomy vocabulary adds no significant value over generic reflection ($p = 0.773$), while action routing provides significant gains ($p = 0.025$), confirming the mechanism holds across backbones.
类型化行动路由（Typed action routing）提供了持续的方向性增益（$\text{F1} = 0.379$ 对比 $0.296$）；在控制了分类词汇表变量后的保守估计为 $\Delta\text{F1} = +0.075$，且相对于单次提示（single-shot）基准线的整体增益通过自助法置信区间（bootstrap CI）验证具有显著性（$\Delta\text{F1} = +0.101$，95% 置信区间 $[+0.020, +0.185]$）。这种词汇表与路由的分解在 GPT-4o 上得到了复现：分类词汇表相比通用反思没有显著价值（$p = 0.773$），而行动路由则提供了显著增益（$p = 0.025$），证实了该机制在不同模型底座上均成立。
Gains concentrate on structurally novel conflicts: in Myanmar ($\text{F1}: 0.000 \rightarrow 0.353$) and Ukraine ($0.167 \rightarrow 0.500$), the vocabulary-only condition recovers no more than generic reflection while action routing breaks the degenerate prior. These findings identify typed action routing — not diagnostic scaffolding or taxonomy vocabulary — as a promising design principle for metacognitive LLM forecasting agents, while motivating larger-scale evaluation across conflict typologies.
增益主要集中在结构上较新颖的冲突中：在缅甸（$\text{F1}: 0.000 \rightarrow 0.353$）和乌克兰（$0.167 \rightarrow 0.500$）的案例中，仅有词汇表的条件表现并不优于通用反思，而行动路由则打破了退化的先验。这些发现表明，类型化行动路由——而非诊断框架或分类词汇表——是元认知 LLM 预测智能体的一种有前景的设计原则，同时也推动了在不同冲突类型中进行更大规模评估的必要性。