When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning
When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning
思维链（CoT）何时有益，何时有害：对大语言模型推理中串行深度瓶颈的实证研究
It is widely assumed that chain-of-thought (CoT) prompting universally improves LLM reasoning. We investigate this through the conceptual framework of the H_dp bandwidth bound (Chen et al., 2024): although the formal bound binds only asymptotically (at astronomically large prompt lengths), it identifies a real architectural bottleneck — serial computation exceeding a transformer’s single-pass capacity must be externalised, which is what CoT does. 人们普遍认为，思维链（CoT）提示词可以普遍提升大语言模型（LLM）的推理能力。我们通过 H_dp 带宽界限（Chen 等人，2024 年）的概念框架对此进行了研究：尽管该形式界限仅在渐近意义上（即在极长的提示词长度下）成立，但它揭示了一个真实的架构瓶颈——即超过 Transformer 单次传递（single-pass）容量的串行计算必须被外部化，而这正是 CoT 所实现的功能。
Our central finding is a within-benchmark serial-depth gradient: single-pass (no-CoT) accuracy degrades monotonically with per-item serial depth, while CoT is approximately depth-invariant. We measure CoT effects across three instruction-tuned models (Qwen-2.5-7B/32B, Llama-3.1-8B) and five standard NLP benchmarks at practical context lengths. 我们的核心发现是基准测试内部存在一种“串行深度梯度”：单次传递（无 CoT）的准确率会随着每个任务项的串行深度增加而单调下降，而 CoT 的表现则几乎与深度无关。我们在实际上下文长度下，针对三种指令微调模型（Qwen-2.5-7B/32B、Llama-3.1-8B）和五个标准 NLP 基准测试评估了 CoT 的效果。
On high-depth P-complete tasks (GSM8K, MATH), CoT gives a +54 to +68 pp recovery gap across all models. On shallow TC^0 tasks (MMLU, ARC), CoT is structurally redundant (Delta in [0.0, +4.6] pp, no significant negative effect) — though high no-CoT baselines (up to 95% on ARC) may reflect contamination, so this null is not a clean architectural test. 在深度较高的 P-完全任务（如 GSM8K、MATH）中，CoT 在所有模型中带来了 +54 到 +68 个百分点的性能恢复差距。在浅层的 TC^0 任务（如 MMLU、ARC）中，CoT 在结构上是冗余的（增幅在 0.0 到 +4.6 个百分点之间，无显著负面影响）——尽管无 CoT 模式下的高基准线（ARC 上高达 95%）可能反映了数据污染，因此这一零结果并非纯粹的架构测试。
The intermediate class L (HumanEval) shows a model-size-dependent transition: +23.2 pp (32B), +9.1 pp (8B), -28.7 pp (7B). The cross-benchmark depth-recovery correlation is Spearman rho = 0.661 (p = 0.007, n = 15); 9 of 15 benchmark-level McNemar tests are significant after Bonferroni correction. 中间类别 L（HumanEval）显示出一种依赖于模型规模的转变：32B 模型提升了 23.2 个百分点，8B 模型提升了 9.1 个百分点，而 7B 模型则下降了 28.7 个百分点。跨基准测试的深度恢复相关性为 Spearman rho = 0.661 (p = 0.007, n = 15)；在 Bonferroni 校正后，15 项基准测试级别的 McNemar 检验中有 9 项具有显著性。
Pre-registered on OSF, our results indicate that CoT is not a universal reasoning enhancer but acts as a bandwidth bypass: it helps serial computation that strains single-pass capacity and is redundant for tasks that already fit. 我们的研究结果已在 OSF 上预注册，表明 CoT 并非通用的推理增强器，而是充当了一种“带宽旁路”：它有助于处理那些超出单次传递容量的串行计算，而对于已经能够处理的任务，它则是冗余的。