90% of US working hours are spent on tasks not covered well by AI benchmarks

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

AI benchmarks have multiplied in recent years, but coverage of economically valuable work remains limited. To measure this, we compiled relevant benchmarks to measure their coverage of American jobs. This Benchmark Map is our first step toward building a comprehensive, continuously updated resource for understanding AI capabilities across the economy.

近年来，人工智能基准测试的数量成倍增加，但对具有经济价值的工作的覆盖范围仍然有限。为了衡量这一点，我们汇编了相关基准测试，以评估它们对美国就业岗位的覆盖情况。这份“基准测试地图”是我们迈出的第一步，旨在构建一个全面且持续更新的资源，以了解人工智能在整个经济体中的能力。

Each tile represents one of the 923 ONET occupations making up the US labor force, grouped into 22 job families. Occupations are shaded according to the share of their tasks which are represented in our initial list of AI benchmarks. A single benchmark prompt can be mapped to multiple ONET tasks.

每个方块代表构成美国劳动力的 923 个 ONET 职业之一，并被归入 22 个职业类别。职业的阴影深浅取决于其任务在我们的初始人工智能基准测试列表中的覆盖比例。单个基准测试提示词可以映射到多个 ONET 任务。

In our data set: only 208 of 923 occupations have a task with at least 5 benchmark prompts, and 6 of 22 job families have no such occupations.

在我们的数据集中：923 个职业中只有 208 个拥有至少包含 5 个基准测试提示词的任务，而在 22 个职业类别中，有 6 个类别完全没有此类职业。

Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.

我们的研究结果表明，美国 90% 的工作时间都花在现有基准测试几乎没有或完全没有覆盖的任务上。这使得我们很难了解人工智能模型在处理美国人实际工作时的表现究竟如何。

The coverage of existing benchmarks is not random, either. Our map suggests that the AI community is focusing on the largest labor markets first:

现有基准测试的覆盖范围也并非随机。我们的地图显示，人工智能社区正优先关注最大的劳动力市场：

For example, benchmarks in our initial list heavily cover AI-exposed lawyers’ tasks, but barely cover any of the O*NET tasks of similarly AI-exposed legal secretaries or court clerks. The long tail of smaller, middle-income knowledge work jobs appears undercovered.

例如，我们初始列表中的基准测试大量覆盖了受人工智能影响的律师任务，但几乎没有覆盖同样受人工智能影响的法律秘书或书记员的任何 O*NET 任务。那些规模较小、中等收入的知识型工作岗位似乎处于覆盖不足的长尾地带。

As we build out our coverage of model capabilities across the economy, we hope to both surface and help fill more gaps in our understanding of what models can do.

随着我们不断扩大对整个经济体中模型能力的覆盖，我们希望既能发现又能帮助填补我们在理解模型能力方面的更多空白。

To break down the economy into sectors, occupations, and tasks, we used the US Department of Labor’s ONET taxonomy. ONET divides the US workforce into 923 occupations; for each occupation, O*NET attempts to provide a comprehensive list of “tasks” which make up the occupation, totaling 18,797 tasks across all occupations.

为了将经济划分为部门、职业和任务，我们使用了美国劳工部的 ONET 分类法。ONET 将美国劳动力划分为 923 个职业；对于每个职业，O*NET 试图提供一份构成该职业的“任务”综合列表，所有职业的任务总数达到 18,797 项。

To measure benchmark coverage, we start with a list of 29 benchmarks selected for their quality and economic relevance. (The full list and selection criteria are in the Benchmark Index tab). In selecting benchmarks for this list, we prioritized:

为了衡量基准测试的覆盖率，我们首先选取了 29 个因其质量和经济相关性而被选中的基准测试。（完整列表和选择标准位于“基准测试索引”选项卡中）。在为该列表选择基准测试时，我们优先考虑了：

Following work by Wang et al., we then ask an LLM judge to tag each benchmark prompt with as many ONET tasks as it thinks correspond to that prompt. We validate our LLM tagging with human expert tags on Mercor’s APEX benchmarks, a subset of this list of 29. On the Benchmark Map, an ONET task counts as “covered” if our LLM judge has mapped it to at least 5 prompts from our 29 benchmarks.

继 Wang 等人的研究之后，我们要求一个大语言模型（LLM）裁判为每个基准测试提示词标记它认为与之对应的所有 ONET 任务。我们使用 Mercor 的 APEX 基准测试（这 29 个基准测试的一个子集）上的人类专家标签来验证我们的 LLM 标记。在基准测试地图上，如果我们的 LLM 裁判将其映射到我们 29 个基准测试中的至少 5 个提示词，则该 ONET 任务被视为“已覆盖”。

To estimate the total share of working hours Americans spend on tasks covered by benchmarks on this list, we combine our coverage estimates with Hatgis-Kessell et al.’s LLM-derived estimates of the working time share of each O*NET task.

为了估算美国人花在这些基准测试所覆盖任务上的工作时间总份额，我们将我们的覆盖率估算值与 Hatgis-Kessell 等人通过 LLM 得出的每个 O*NET 任务的工作时间份额估算值相结合。

Our estimates of total economic coverage, and of patterns in coverage, all come from the initial list of 29 benchmarks we chose. This is not a complete or perfectly representative sample of the broader universe of unsaturated, public, agentic economic benchmarks. Any incomplete list will necessarily bias down estimates of economic coverage, and it is difficult to bound that bias. We want to stress, therefore, that these results are provisional. We will update them as we expand the initial benchmark list.

我们对总经济覆盖率及覆盖模式的估算，均来自我们选择的 29 个基准测试的初始列表。这并不是更广泛的未饱和、公开、代理型经济基准测试领域的完整或完美代表性样本。任何不完整的列表必然会使经济覆盖率的估算值偏低，且很难界定这种偏差。因此，我们要强调，这些结果是暂时的。随着我们扩展初始基准测试列表，我们将更新这些结果。

We also capped the number of included benchmarks in each job family in order to maximize breadth of coverage. We think this makes our estimate of economy-wide benchmark coverage more accurate, at the cost of biasing down our estimates of coverage in the most-benchmarked job families (e.g. software engineering).

为了最大限度地扩大覆盖范围，我们还限制了每个职业类别中包含的基准测试数量。我们认为这使得我们对全经济范围基准测试覆盖率的估算更加准确，但代价是降低了我们在基准测试最密集的职业类别（如软件工程）中的覆盖率估算值。

By selecting for unsaturated benchmarks, we risk understating benchmark coverage on O*NET tasks which are easier for LLMs to complete. On the other hand, including saturated benchmarks would risk overstating benchmark coverage if saturation is downstream of lower task realism. The latter claim seems likely, given that benchmarks often overstate LLM performance in real-world deployments. We think that the bias from including saturated benchmarks would be greater than the bias from excluding them, but neither option is ideal.

通过选择未饱和的基准测试，我们冒着低估 LLM 更容易完成的 O*NET 任务的基准测试覆盖率的风险。另一方面，如果饱和度是由于任务真实性较低导致的，那么包含饱和的基准测试可能会导致高估基准测试的覆盖率。后一种说法似乎很有可能，因为基准测试往往会夸大 LLM 在实际部署中的表现。我们认为，包含饱和基准测试所带来的偏差要大于排除它们所带来的偏差，但两种选择都不理想。

Finally, while we outline our benchmark selection criteria in the Benchmark Index tab, several criteria (like “Economic Realism”) are difficult to formalize and introduce some subjectivity in our choices.

最后，虽然我们在“基准测试索引”选项卡中概述了我们的基准测试选择标准，但一些标准（如“经济真实性”）很难形式化，并在我们的选择中引入了一些主观性。

The benchmarks in our index have relatively deep and narrow coverage of the overall economy. They disproportionately focus on the economic tasks most exposed to AI automation (as measured by Eloundou et al.), with O*NET tasks rated as “exposed” about 6x more likely to be represented on benchmarks than tasks rated “unexposed”. This makes sense: benchmarks are designed to measure AI progress, and AI progress is most plausible on the most AI-exposed tasks. This explains why benchmarks are full of tasks in Software Engineering and Financial Analysis while sparse on tasks in Truck Driving and Gas Plant Operation.

我们索引中的基准测试对整体经济的覆盖相对深入且狭窄。它们不成比例地集中在受人工智能自动化影响最大的经济任务上（根据 Eloundou 等人的衡量），被评为“受影响”的 O*NET 任务出现在基准测试中的可能性大约是“未受影响”任务的 6 倍。这很有道理：基准测试旨在衡量人工智能的进步，而人工智能在受影响最大的任务上最有可能取得进展。这就解释了为什么基准测试中充满了软件工程和金融分析方面的任务，而在卡车驾驶和天然气厂运营方面的任务却很少。

But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresent

但即使在控制了人工智能影响因素后，来自工资总额较高职业的 O*NET 任务仍然被显著过度代表。