Are AI Labs Pelicanmaxxing?

AI 实验室在“鹈鹕骑车”上作弊了吗？

For the past few years, Simon Willison has tested every major LLM release with the same prompt: “Generate an SVG of a pelican riding a bicycle”. What began as a tongue-in-cheek benchmark has become one of the most famous informal benchmarks in AI. Simon’s pelican-on-a-bicycle results are often among the most upvoted comments on Hacker News threads announcing new releases from AI labs. The benchmark is now famous enough that there’s plenty of discussion about its usefulness and about whether AI labs might be benchmaxxing on it. When billions or even trillions of dollars are at stake, and a strong result could help persuade users, wouldn’t it be tempting to pelicanmaxx your model just a bit? I wanted to find out, so I put together a small experiment. I generated 1,008 SVGs across seven frontier models, scored them with an LLM judge, and used Claude Fable 5 for the analysis. This article presents the results. All the code is available on Github.

过去几年里，Simon Willison 一直用同一个提示词测试每一个主流大模型发布：“生成一张鹈鹕骑自行车的 SVG 图片”。这最初只是一个半开玩笑的基准测试，如今已成为 AI 领域最著名的非正式基准之一。Simon 的“鹈鹕骑车”测试结果，往往是 Hacker News 上关于 AI 实验室新模型发布帖中点赞数最高的评论之一。这个基准测试现在已经足够出名，以至于人们开始讨论它的实用性，以及 AI 实验室是否可能在上面进行“基准作弊”（benchmaxxing）。当涉及数十亿甚至数万亿美元的利益，且一个出色的结果有助于说服用户时，难道不会让人产生给模型“鹈鹕作弊”一下的冲动吗？为了找出答案，我进行了一个小实验。我在七个前沿模型上生成了 1,008 张 SVG 图片，用一个 LLM 裁判进行评分，并使用 Claude Fable 5 进行分析。本文展示了实验结果。所有代码均已在 Github 上开源。

How I tested it

我是如何测试的

I built a grid of 8 animals × 6 vehicles = 48 prompts, where the famous prompt is one cell: Animals: pelican, flamingo, heron, otter, raccoon, antelope, whale, cat Vehicles: bicycle, unicycle, skateboard, scooter, plane, boat. Every prompt uses almost identical phrasing to Simon’s, only switching the animal and vehicle. The animal and vehicle selection wasn’t done in a very rigorous manner, but I tried to vary both similarity to the original prompt and difficulty. Flamingo and heron are quite similar to pelicans; cat, raccoon, and otter are easy cases; antelope is hard; and whale is as different as you can get. I tested seven models through OpenRouter: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. I generated 3 samples per prompt, at temperature 1.0, requesting the same reasoning effort from every model. That resulted in 1,008 SVGs.

我构建了一个 8 种动物 × 6 种交通工具 = 48 个提示词的矩阵，其中那个著名的提示词只是其中的一个单元格：动物包括鹈鹕、火烈鸟、苍鹭、水獭、浣熊、羚羊、鲸鱼、猫；交通工具包括自行车、独轮车、滑板、滑板车、飞机、船。每个提示词都使用了与 Simon 几乎完全相同的措辞，仅替换了动物和交通工具。动物和交通工具的选择并不十分严谨，但我尽量兼顾了与原始提示词的相似度以及难度差异。火烈鸟和苍鹭与鹈鹕非常相似；猫、浣熊和水獭属于简单案例；羚羊较难；而鲸鱼则是差异最大的。我通过 OpenRouter 测试了七个模型：GPT-5.6 Terra、Claude Sonnet 5、Gemini 3.5 Flash、Grok 4.5、Qwen3.7-Max、GLM-5.2 和 DeepSeek V4 Pro。每个提示词生成 3 个样本，温度设置为 1.0，并要求每个模型保持相同的推理强度。最终生成了 1,008 张 SVG 图片。

Then I ran each image through a three-stage pipeline:

1. Rendering: Each SVG is rendered to PNG. If a model returns no SVG or one that fails to render, I regenerate until it produces a valid one, and record the number of attempts. There were only 11 retries across the 1,008 generations.
2. Judging: GPT-5.6 Luna scores each image with 1-5 ratings for the animal, the vehicle, and the coherence of the action. When I rank animals or vehicles below, I use the matching rating on its own. When I need one number per image, I use the average of the three, which I call the judge score.
3. Feature extraction: For a more detailed analysis, I also passed each rendered image to Gemini 3.1 Flash-Lite, which recorded the animal and vehicle it recognized, which way the subject faces, and an open-ended list of scene elements.

随后，我通过一个三阶段流程处理了每张图片：

1. 渲染： 将每个 SVG 渲染为 PNG。如果模型没有返回 SVG 或渲染失败，我会重新生成直到产生有效文件，并记录尝试次数。在 1,008 次生成中，仅进行了 11 次重试。
2. 评分： GPT-5.6 Luna 对每张图片的动物、交通工具以及动作连贯性进行 1-5 分的评分。在下文对动物或交通工具进行排名时，我使用对应的单项评分。当需要每张图片的综合得分时，我使用三项评分的平均值，即“裁判得分”。
3. 特征提取： 为了进行更详细的分析，我还将每张渲染后的图片传给 Gemini 3.1 Flash-Lite，它记录了识别出的动物和交通工具、主体朝向以及一个开放式的场景元素列表。

My hypothesis is that if a lab trained on the benchmark, it should show up in some combination of the pelican row scoring above what the animal deserves, the bicycle column scoring above what the vehicle deserves, or the specific pelican-bicycle cell beating both.

我的假设是：如果某个实验室针对该基准进行了训练，那么它应该会在以下组合中有所体现：鹈鹕行的得分高于该动物应有的水平，自行车列的得分高于该交通工具应有的水平，或者特定的“鹈鹕-自行车”单元格表现优于两者。

Evidence #1: The pelicans on bicycles don’t look any better

证据一：鹈鹕骑自行车的图片看起来并没有更好

Before any scoring, the simplest test is to look at the images yourself. I looked through the images myself before running the analysis below. Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid. Maybe in GLM-5.2’s first sample it felt slightly better than the rest, but that batch also produced a pretty cool heron on a skateboard, so I cannot say for sure. Otherwise they look like the rest of what each model draws, and the labs that draw good pelicans on bicycles also do a good job drawing other animal-vehicle combinations. But this test is hard to replicate, and everyone will have a different opinion. So I wanted something more quantitative, which is why I opted for the method detailed above.

在进行任何评分之前，最简单的测试就是亲自查看图片。在进行下文的分析之前，我亲自浏览了所有图片。没有任何东西让我感到惊艳。我找不到任何一个案例，其“鹈鹕骑自行车”的图片明显优于该模型矩阵中的其他图片。也许 GLM-5.2 的第一个样本感觉稍微好一点，但那一批次也生成了一张非常酷的“苍鹭骑滑板”，所以我无法确定。除此之外，它们看起来与模型生成的其他图片无异，那些能画好“鹈鹕骑自行车”的实验室，在绘制其他动物与交通工具组合时也表现得同样出色。但这个测试很难复现，每个人的看法也会不同。所以我需要更量化的东西，这就是我选择上述方法的原因。

Evidence #2: Labs are not better at drawing pelicans

证据二：实验室在绘制鹈鹕方面并没有表现更好

Here’s the mean animal rating per animal, pooled across all models: Figure 1: Mean animal rating by animal, all models pooled. The pelican is 6th of 8, behind cat, whale, raccoon, heron, and antelope. If AI labs were training on the benchmark, you’d expect pelicans at the top. Instead they’re in the bottom half. All seven labs draw cats, whales, and raccoons better than pelicans. Of course, a pelican may simply be harder to draw than a cat. A lab could train on pelicans and still not push them past the easy animals, so this ranking alone can’t rule that out. I’ll adjust for difficulty in Evidence #4.

以下是所有模型汇总后，每种动物的平均评分： 图 1：按动物分类的平均评分（所有模型汇总）。 鹈鹕在 8 种动物中排名第 6，落后于猫、鲸鱼、浣熊、苍鹭和羚羊。如果 AI 实验室针对该基准进行了训练，你本应期望鹈鹕排在首位。但实际上它们处于后半段。所有七个实验室绘制猫、鲸鱼和浣熊的效果都比鹈鹕好。当然，鹈鹕可能确实比猫更难画。实验室即便针对鹈鹕进行了训练，也可能无法使其表现超过那些简单的动物，因此仅凭这一排名无法排除作弊的可能性。我将在证据四中对难度进行校准。

Evidence #3: Labs are not better at drawing bicycles

证据三：实验室在绘制自行车方面并没有表现更好

Bicycles fare even worse. They sit second from last, in a near-tie with planes, which come in last: Figure 2: Mean vehicle rating by vehicle, all models pooled. If labs were training on the benchmark, you’d expect bicycles near the top of this ranking. They’re not. However, the same caveat applies here. A bicycle is harder to draw than a skateboard: it needs two matching wheels, a frame that reaches both axles, handlebars, a seat, and pedals. The judge flags a missing or disconnected one of those on 2/3 of the bicycle images. You can train on bicycle images and still not do a great job relative to simpler vehicles. One note on the plane, though: I should’ve picked “airplane” instead of “plane” because models often read it geometrically. They drew the animal standing on a flat surface instead of flying an aircraft. The plane is the only vehicle where the feature extractor sometimes found no vehicle at all (25 of 168 images, against zero for the other five), and 20% of plane images scored a 1 or 2 on the vehicle rating, against 5% for bicycles and none at all for boats, scooters, or skateboards.

自行车的表现更差。它们排在倒数第二，与排在最后的飞机几乎持平： 图 2：按交通工具分类的平均评分（所有模型汇总）。 如果实验室针对该基准进行了训练，你本应期望自行车排在排名的前列。但事实并非如此。不过，这里同样存在上述的注意事项。自行车比滑板更难画：它需要两个匹配的车轮、连接两个车轴的车架、车把、车座和踏板。裁判在 2/3 的自行车图片中标记出了缺失或断开的部件。你即便针对自行车图片进行训练，相对于简单的交通工具，依然可能表现不佳。关于飞机有一点说明：我应该选择“airplane”而不是“plane”，因为模型经常从几何角度理解它。它们把动物画在平坦的表面上，而不是驾驶飞机。飞机是唯一一个特征提取器有时完全找不到交通工具的类别（168 张图片中有 25 张，而其他五种交通工具为零），且 20% 的飞机图片在交通工具评分上仅得 1 或 2 分，相比之下自行车为 5%，而船、滑板车或滑板则完全没有这种情况。

Evidence #4: Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty

证据四：即便校准了难度，实验室在绘制“鹈鹕骑自行车”方面也没有表现更好

Put the two together and the “pelican on a bicycle” ends up near the bottom of the ranking, at #42 of 48: Figure 3: All 48 combos ranked; pelican + bicycle highlighted.

将两者结合起来，“鹈鹕骑自行车”最终排在排名的底部，在 48 个组合中位列第 42 位： 图 3：所有 48 个组合的排名；“鹈鹕+自行车”已高亮显示。