I Put a Bunch of AI Agents in Charge of a Fantasy Soccer Team. They’re Not Better Than Humans—Yet

I have consulted my AI-powered crystal ball, and it’s telling me that frontier models are going to get a lot better at making predictions in the near future. Just ask my Premier League fantasy team.

我咨询了我的 AI 水晶球，它告诉我前沿模型在不久的将来预测能力会大幅提升。问问我的英超梦幻足球队就知道了。

A few weeks ago, I deployed a small horde of AI agents to craft and manage an experimental roster. Given the proliferation of AI agents into pretty much every aspect of our lives, I wanted to know if a swarm of bots could put together a winning team.

几周前，我部署了一小群 AI 智能体来组建和管理一个实验性阵容。鉴于 AI 智能体已经渗透到我们生活的方方面面，我想知道一群机器人是否能组建出一支获胜的队伍。

I first looked at the potential for AI here a few years ago, playing around with a tool from the UK’s Turing Institute that selected teams after modeling previous fantasy league data. Its suggestions were remarkably bonkers—it didn’t incorporate real-world information, like a player’s injury or a team’s change in management.

几年前，我首次研究了 AI 在这方面的潜力，当时我试用了一个来自英国图灵研究所的工具，它通过对以往梦幻联赛数据进行建模来选择球队。它的建议简直荒唐透顶——它没有整合现实世界的信息，比如球员受伤或球队管理层变动。

This time, while on a flight to visit family, I asked Codex for help. First, I fed the model a couple of relevant research papers. Then, I asked it to build a harness—instructions and code that guides different agents through various steps—that would check recent news and run a model to make predictions.

这一次，在探亲的航班上，我向 Codex 寻求帮助。首先，我向该模型输入了几篇相关的研究论文。然后，我要求它构建一个“框架”——即引导不同智能体完成各项步骤的指令和代码——用以查看最新新闻并运行模型进行预测。

It quickly came up with a setup that seemed to collate and analyze way more information than I ever would have. And yet I was surprised by how dumb the initial team suggestions were. Leaving Erling Haaland on the bench? Really, Codex?

它很快就想出了一套方案，看起来能收集和分析比我多得多的信息。然而，最初的球队建议之愚蠢让我感到惊讶。把埃尔林·哈兰德留在替补席上？认真的吗，Codex？

I then asked Codex to create a team of agents that did different things: several that read news and blog posts; others that analyzed recent results; more that ran models; and a bunch that critiqued suggestions. This seemed to significantly improve the suggestions. I tried increasing the number of agents to more than 100, hoping this would make the system smarter still, but it didn’t change much besides running up my API bill.

随后，我要求 Codex 创建一个分工明确的智能体团队：一些负责阅读新闻和博客文章；另一些负责分析近期比赛结果；还有一些负责运行模型；以及一群负责评估建议的智能体。这似乎显著改善了建议的质量。我尝试将智能体数量增加到 100 多个，希望这能让系统更聪明，但除了让我的 API 账单飙升外，并没有太大改变。

Still, the improvement from a few years ago was striking. According to Haifeng Xu, a computer scientist at the University of Chicago who studies the uses of AI in market situations, newer models may be even better at making predictions because they’re able to consume and analyze huge amounts of real-world information, and they’re able to orchestrate more complex systems for their guesses.

尽管如此，与几年前相比，进步是惊人的。芝加哥大学研究 AI 在市场环境中应用的计算机科学家徐海峰（音译）表示，较新的模型在预测方面可能表现更好，因为它们能够摄取和分析海量的现实世界信息，并能协调更复杂的系统来进行推测。

“AI is just getting smarter in understanding the world—I feel like that’s the fundamental reason they’re improving,” Xu tells me.

“AI 在理解世界方面变得越来越聪明——我觉得这是它们不断进步的根本原因，”徐告诉我。

Late last year, Xu was marveling at the growing popularity of prediction markets like Kalshi and Polymarket. He wondered how the best commercial models might fare on those markets, so alongside several colleagues, he began testing their forecasting skills.

去年年底，徐对 Kalshi 和 Polymarket 等预测市场的日益普及感到惊叹。他想知道最优秀的商业模型在这些市场上表现如何，于是他与几位同事一起开始测试它们的预测能力。

They had models guess, for example, who would win the NBA Championship and who President Trump would meet with.

他们让模型进行猜测，例如谁将赢得 NBA 总冠军，以及特朗普总统将会见谁。

“Very quickly we started to realize that models did really well,” Xu tells me. The models were sometimes able to do better than the average person without any special instructions or guidance, he says. In their testing, GPT-5 was the best at making predictions, followed by Grok 4, Claude Sonnet, Gemini, and Llama.

“我们很快意识到模型的表现非常好，”徐告诉我。他说，在没有任何特殊指令或指导的情况下，这些模型有时能比普通人做得更好。在他们的测试中，GPT-5 的预测能力最强，其次是 Grok 4、Claude Sonnet、Gemini 和 Llama。

The team went on to develop Prophet Arena, a benchmark that measures a model’s ability to make predictions. It’s one of a number of benchmarks designed to test AI’s future-gazing capabilities; another, Prediction Arena, from researchers at Harvard University and a company called Arcada Labs, recently gave frontier models $10,000 to wager on prediction markets over 57 days. The models consistently lost money in their experiment, but the researchers saw evidence of them improving consistently over time.

该团队随后开发了 Prophet Arena，这是一个衡量模型预测能力的基准。它是旨在测试 AI 未来预测能力的众多基准之一；另一个来自哈佛大学研究人员和 Arcada Labs 公司的 Prediction Arena，最近给了前沿模型 1 万美元，让它们在 57 天内对预测市场进行投注。在他们的实验中，这些模型持续亏损，但研究人员看到了它们随时间推移不断进步的证据。

Xu tells me that AI can most effectively make predictions using a custom harness, and notes that having agents with unique tasks—reading the news, collecting other data, and analyzing the whole system, for instance—seems to be the most effective approach.

徐告诉我，AI 使用自定义框架可以最有效地进行预测，并指出让智能体承担独特任务（例如阅读新闻、收集其他数据以及分析整个系统）似乎是最有效的方法。

Interestingly, AI still often lags behind very skilled experts known as superforecasters. Xu explains that, although humans cannot consume as much information, their understanding of the real world—that is, of the dynamics behind a news story—is deeper.

有趣的是，AI 往往仍然落后于被称为“超级预测者”的资深专家。徐解释说，虽然人类无法摄取那么多的信息，但他们对现实世界的理解——即新闻报道背后的动态——更为深刻。

That said, there’s growing evidence that the human superforecaster’s edge is eroding. A contest called the Metaculus Cup recently showed that AI can sometimes beat even the best human predictors. And a benchmark called ForecastBench could provide further evidence that AI can match human skills when the results of an ongoing contest conclude soon. In fact, I think we can expect the big labs to turn their attention to future-gazing as a powerful new way to measure progress in machine intelligence.

话虽如此，越来越多的证据表明，人类超级预测者的优势正在减弱。最近一场名为 Metaculus Cup 的比赛表明，AI 有时甚至能击败最优秀的人类预测者。而一个名为 ForecastBench 的基准测试可能会在不久后比赛结果出炉时，进一步证明 AI 能够匹敌人类技能。事实上，我认为我们可以预见，各大实验室会将注意力转向未来预测，将其作为衡量机器智能进步的一种强大的新方式。

All of these predictive experiments seem to suggest that, like math, coding, and law, industries predicated on trading may soon be upended by AI. Perhaps the best human traders will take on a new kind of role: supervising and orchestrating AI agents capable of divining the future far better than they can.

所有这些预测性实验似乎都表明，就像数学、编程和法律一样，以交易为基础的行业可能很快会被 AI 颠覆。也许最优秀的人类交易员将承担一种新的角色：监督和协调那些比他们更能洞察未来的 AI 智能体。

For the time being, my AI’s suggestions still don’t seem all that impressive. Its team hasn’t outscored the average over the past weeks. I certainly won’t be using any of its suggestions in my own selections.

目前，我的 AI 给出的建议看起来仍然不那么令人印象深刻。在过去的几周里，它的球队得分并没有超过平均水平。我当然不会在自己的选择中使用它的任何建议。

But AI will no doubt get better, perhaps making it all too easy to dominate fantasy league pretty soon. I’m kind of hoping it won’t, though, as it would spoil the fun of trying to earn bragging rights by winning all by yourself.

但 AI 无疑会变得更好，也许很快就能轻而易举地统治梦幻联赛。不过，我倒希望它不会，因为这会破坏那种靠自己获胜并赢得吹嘘资本的乐趣。

This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.

这是 Will Knight 的 AI 实验室通讯的一期。点击此处阅读往期通讯。