Counting the Bandages: On measuring the time that AI saves

Title: Counting the Bandages: On measuring the time that AI saves

标题：计算绷带：关于衡量人工智能节省的时间

Abstract: I consider what it means for AI assistance to save time. I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter. The argument has four parts. I define a model-free pair of quantities, time saved and excess escaped defects, and one decision number built from them: the break-even defect cost.

摘要：我探讨了人工智能辅助节省时间意味着什么。我对比了“秒表视角”（即节省的时间由时钟记录）与“分类账视角”（即节省的时间是一个反事实概念，必须结合工作遗漏的错误来解读），并主张采用后者。该论证分为四个部分。我定义了一对与模型无关的量——节省的时间和额外遗漏的缺陷，以及由此构建的一个决策数值：盈亏平衡缺陷成本。

I give a baseline model of checking in which, until the model is good enough not to check, a calibrated worker’s escaped-defect rate does not depend on how good the model is; checking time grows with the logarithm of the model’s error rate; and over-trust buys minutes on the clock at a larger cost in total minutes. I show what randomized studies can and cannot identify, and I replicate METR’s developer study from its public data: the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them.

我提供了一个检查基准模型，在该模型中，除非模型好到无需检查，否则校准后的工作人员的缺陷遗漏率并不取决于模型的优劣；检查时间随模型错误率的对数增长；过度信任虽然在时钟上节省了时间，但却以总时间上的更大代价为代价。我展示了随机研究能识别和不能识别的内容，并根据公开数据复制了 METR 的开发者研究：同样的 246 个问题支持 19% 或 24% 的减速（取决于估计量，若不调整偶然失衡则为 34%），而人工智能在其中 3% 到 82% 的问题上提供了帮助。

Finally, I restate the published record, with stated bounds in place of imported point assumptions. In GDPval’s own review-every-output workflow, review time alone caps GPT-5’s speedup at 3.6x; the consultants outside the frontier were faster, and more often wrong. Last, I test two of the model’s predictions in a small pre-specified experiment on AI reviewers checking AI drafts of the same kind of brief, built from SEC filings made after the models’ training data were collected.

最后，我重述了已发表的记录，用明确的界限取代了引入的点假设。在 GDPval 自身的“审查每一项输出”工作流程中，仅审查时间就将 GPT-5 的加速上限限制在 3.6 倍；处于前沿之外的顾问速度更快，但出错频率更高。最后，我在一个小型预设实验中测试了该模型的两个预测，即由人工智能审查员检查人工智能起草的同类简报，这些简报基于模型训练数据收集后提交的 SEC 文件。

Told the drafter’s true error rate, the reviewer checked more as it rose, but all-or-nothing, switching near where its prompt’s costs break even. Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none: faster and worse, entirely through reduced effort.

当被告知起草者的真实错误率时，审查员会随着错误率的上升而增加检查力度，但这种行为是全有或全无的，在提示词成本达到盈亏平衡点附近进行切换。当被告知起草者的错误率比实际低十倍时，审查员在 9 个数字中少检查了 7 个，并在 22.5% 的简报中放行了错误数字（原本为零）：通过减少工作量，实现了更快的速度，但质量更差。