When Should Forecasting Agents Reason? Behavioral Stress Tests for Reliability Routing

预测智能体何时应该进行推理？可靠性路由的行为压力测试

Abstract: Forecasting agents increasingly combine language-model reasoning, retrieval, ensembling, and calibration, but it remains unclear when each behavior should be trusted. We study this question on ForecastBench-style binary forecasting tasks, treating the choice to retrieve, reason, defer to a market prior, or use a historical analog as an observable agent behavior rather than a hidden implementation detail.

摘要： 预测智能体正越来越多地结合语言模型推理、检索、集成和校准技术，但目前尚不清楚何时应该信任这些行为。我们在 ForecastBench 风格的二元预测任务上研究了这一问题，将“选择检索、推理、参考市场先验，还是使用历史类比”视为一种可观察的智能体行为，而非隐藏的实现细节。

Our central finding is that mechanism choice is source-dependent: structured analogs dominate for some data-generating processes, while market/crowd-style and conservative baselines are better for others. We introduce ReliabilityRoute, a structural intervention that steers forecasting-agent behavior using reliability features such as historical coverage, market-prior availability, source-prior sharpness, evidence strength, evidence disagreement, and horizon.

我们的核心发现是，机制的选择取决于数据源：对于某些数据生成过程，结构化类比占主导地位；而对于其他过程，市场/群体风格和保守的基准模型表现更好。我们引入了 ReliabilityRoute，这是一种结构性干预措施，通过历史覆盖率、市场先验可用性、源先验尖锐度、证据强度、证据分歧和预测时限等可靠性特征来引导预测智能体的行为。

A fixed 2024-fitted rule closely matches a hand taxonomy without hard-coded source-name decisions, while a walk-forward self-adjusting rule refits thresholds from previously resolved vintages and obtains the best mean Brier score among our deterministic systems across 16 later LLM vintages. The gain is modest and historical/search baselines remain highly competitive.

一个基于 2024 年数据拟合的固定规则，在无需硬编码源名称决策的情况下，与人工分类法高度吻合；而一种向前滚动（walk-forward）的自适应规则，通过从先前已解决的预测版本中重新拟合阈值，在 16 个后续的大语言模型版本中，获得了我们确定性系统中的最佳平均 Brier 分数。尽管提升幅度适中，但历史/搜索基准模型依然具有极强的竞争力。

The main contribution is therefore a behavioral stress test showing that more reasoning is not always better; forecasting agents should first estimate which evidence source deserves control, routing policies should themselves adapt under auditable constraints, and reproducibility artifacts are available at this https URL.

因此，本文的主要贡献在于进行了一项行为压力测试，表明“更多的推理并不总是更好”；预测智能体应首先评估哪个证据源值得采信，路由策略本身应在可审计的约束下进行调整。相关的可复现性资源可在提供的链接中获取。