Evaluating Strategic Reasoning in Forecasting Agents

评估预测智能体的战略推理能力

Abstract: Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to the Future 2 (BTF-2), 1,417 pastcasting questions with a frozen 15M-document research corpus in which agents reproducibly research and forecast offline, producing full reasoning traces.

摘要： 现有的预测基准测试虽然能生成准确率排行榜，但却难以深入揭示为何某些预测者比其他人更准确。我们推出了“Bench to the Future 2 (BTF-2)”，包含 1,417 个回溯预测问题，并配备了一个包含 1,500 万份文档的静态研究语料库。在该框架下，智能体可以离线进行可复现的研究与预测，并生成完整的推理轨迹。

BTF-2 detects accuracy differences of 0.004 Brier score, and can distinguish differential agent strengths in research vs. judgment. We build a forecaster 0.011 Brier more accurate than any single frontier agent, and use it to evaluate agent strategic reasoning without hindsight bias.

BTF-2 能够检测到 0.004 Brier 分数的准确率差异，并能区分智能体在“研究”与“判断”方面的不同优势。我们构建了一个预测器，其 Brier 分数比任何单一的前沿智能体高出 0.011，并利用它在无后见之明偏差（hindsight bias）的情况下评估智能体的战略推理能力。

We find the better forecaster differs primarily in its pre-mortem analysis of its blind spots and consideration of black swans. Expert human forecasters found the dominant strategic reasoning failures of frontier agents are in assessing political and business leaders’ incentives, judging their likelihood to follow through on stated plans, and modeling institutional processes.

我们发现，更优秀的预测器在“针对盲点的预先死亡分析（pre-mortem analysis）”以及“对黑天鹅事件的考量”方面表现出显著差异。人类专家预测者指出，前沿智能体在战略推理上的主要缺陷在于：评估政治和商业领袖的动机、判断他们执行既定计划的可能性，以及对制度流程的建模能力。