Creepy crawlies

August 29, 2026

You’ve probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact. In a few words, it’s bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model. 你可能以前听我抱怨过“AI 爬虫”，但现在我终于有一些确凿的数据来展示它们的影响了。简而言之，它们造成的系统负载已经形成了一种持续的“背景辐射”，永久性地占用了一部分算力，而这些算力产生的输出仅有一个用途——喂养学习模型。

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. 简而言之：我们为爬虫渲染提交记录所消耗的 CPU 周期，比处理包括 git clone 在内的所有其他合法访问所消耗的还要多。在任何给定时间，分布在 5 个地理节点的服务器上，总共有 14 个 CPU 核心仅仅是在忙于将 git 提交记录渲染为 HTML。

Why is git.kernel.org “interesting” to crawlers

为什么 git.kernel.org 对爬虫来说“很有趣”？

Linux development happens in the open — from git repositories you can clone, to discussion archives you can follow in real time. To a large language model, this is a goldmine of learning data, because all of this is not only immediately available, but is easy to filter in order to guarantee pure unadulterated pre-AI content. Linux 的开发是公开进行的——从你可以克隆的 git 仓库，到你可以实时跟踪的讨论存档。对于大语言模型来说，这是一座学习数据的金矿，因为所有这些内容不仅立即可用，而且易于筛选，从而保证了纯净、未受 AI 污染的原始内容。

Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it’s worth its weight in gold as a source of training data. 用 AI 生成的内容来训练大模型，相当于让它患上“数字朊病毒病”。因此，当一个来源（如内核提交的全部历史记录）被保证不含 AI 生成内容时，它作为训练数据来源就显得价值连城。

The stupidest way of doing it

最愚蠢的做法

We make almost everything clonable, because hey — we may not be around forever, so here — clone the repos. Also, clone the archives. Grab a copy just so we’re not the only ones who have it all. Seriously, it’s just a “git clone” away — and then you’ll have the whole history. For example, did you know you can clone the entirety of LKML and then do whatever you want with it? It’s just git repos all the way down. 我们几乎让所有东西都可以克隆，因为嘿——我们可能不会永远存在，所以拿去吧——克隆这些仓库。还有，克隆这些存档。拿一份副本，这样就不是只有我们拥有全部数据了。说真的，只需要一个“git clone”——你就能拥有完整的历史记录。例如，你知道你可以克隆整个 LKML 并随心所欲地使用它吗？这本质上全是 git 仓库。

So, you’d think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done. But no, let’s in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it. 所以，你可能会认为，某种自称“人工智能”的东西会使用最高效的方式来利用我们的数据进行训练，对吧？克隆仓库，遍历每个提交。搞定。但事实并非如此，它们偏偏选择了最愚蠢的方式——将所有内容逐个提交渲染为 HTML，然后再进行解析。

At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don’t worry, it’s actually extremely efficient on the backend, since it’s mostly the same objects in every fork. Unless, of course, you’re a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing. 在撰写本文时，linux.git 大约有 148 万次提交。哦，我们在 git.kernel.org 上还有大约 922 个分支——但别担心，后端处理效率极高，因为每个分支中大部分对象都是相同的。当然，除非你是爬虫，那样的话，你就有几十亿个有效的 URL 可以抓取，结果只是得到了 922 份相同的 148 万次提交的副本——而这正是爬虫们正在做的事情。

But wait, it’s not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits — cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is AWFUL right about now, because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git. 等等，不仅仅是提交记录。你还可以请求补丁、纯文本渲染、任意提交之间的差异对比——cgit 很乐意满足你。这在互联网属于人类或遵守 robots.txt 的爬虫的时代是完美的，但在现在却糟糕透顶，因为仅针对 linux.git 的一个分支，我们就能生成天文数字般的有效 URL。

Block them

封锁它们

Initially, this was the solution — look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers. 最初，解决方案是这样的——查看日志，找出哪些 IP 是明显的爬虫机器人，然后用 fail2ban 封锁它们。起初这很容易，因为机器人会通过 User-Agent 友好地告诉你它们的身份。后来，它们变聪明了，开始伪装成普通的浏览器。

So, we started banning them by IP — after all, it’s easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen. 于是，我们开始按 IP 封锁它们——毕竟，很容易判断出一个试图抓取一个 8 年前废弃的 Linux 分支中每一个提交的 IP，绝不是某个在 Windows 上使用 Chrome 的普通用户在疯狂点击屏幕上的每一个链接。

The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just pretending to be a Firefox user. Banning the whole ASN was justified, even if this occasionally caught a random legitimate instance trying to automate link checking in commits. 随后，机器人开始分散到整个子网，但这依然没用，因为很明显，来自 Google Compute 的 IP 只是在伪装成 Firefox 用户。封锁整个 ASN（自治系统号）是合理的，即使这偶尔会误伤一些试图自动检查提交链接的合法实例。

Enter… your TV?

轮到……你的电视了？

And… that’s when things turned really, really ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. 然后……事情变得非常非常糟糕。突然间，爬虫开始来自数百万个随机的住宅或移动 IP，它们全都伪装成现代浏览器。这样的 IP 发出 4-5 个请求后，就再也不会出现在日志中了。

There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back. They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. 封锁它们毫无意义，因为当你发现它们是机器人时，它们已经抓取完毕了。你只是在防火墙规则集中添加了永远不会再回来的 IP，徒增负担。它们像蝗虫群一样降临，猛烈而迅速地攻击，直到系统崩溃，然后转移到下一个目标，等你恢复后，它们又回来了。循环往复。

They still do that — welcome to the wonderful world of “proxy SDK monetization.” It’s big business, and your TV is probably doing it. 它们现在还在这样做——欢迎来到“代理 SDK 变现”的美妙世界。这是一门大生意，而你的电视可能就在参与其中。

Make them pay

让它们付出代价

When this first became a problem, oh, about a year ago, we naively thought that there was a way to make it stop. Just make the bots perform a task that would flip the economy of the whole thing upside-down by making them burn some cycles doing throwaway math. Like, calculate what string, when combined with their own IP and a secret we provide, would generate a sha256 sum with 4 leading zeroes. In other words, we put Anubis in front of everything. 当这个问题大约一年前首次出现时，我们天真地以为有办法阻止它。只需让机器人执行一项任务，通过让它们消耗算力进行无意义的数学运算，从而彻底颠覆整个事情的经济模型。比如，计算出什么样的字符串与它们的 IP 以及我们提供的密钥结合后，能生成一个以 4 个零开头的 sha256 哈希值。换句话说，我们在所有东西前面部署了 Anubis（验证系统）。

It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere. 这立即产生了极好的效果——机器人直接放弃了。有几个月的时间，一切都很美好：机器人在边界被拦截并放弃，转向了更容易的目标；用户虽然感到轻微不便但表示理解，而且 Anubis 堆栈很容易部署到各处。

A few months later, the bots were back, solving difficulty 4. No problem, we said, let’s raise difficulty to 5. The legitimate users were more annoyed now. Difficulty 5 takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm as it’s doing the number crunching. However, it was effective and bought us a few more months of peace. Then… the bots started solving difficulty 5. 几个月后，机器人回来了，它们破解了难度 4。没问题，我们说，把难度提高到 5。合法用户现在更烦了。在移动设备上解决难度 5 需要几秒钟，手机在进行数字运算时会变得烫手。然而，这很有效，又为我们争取了几个月的平静。然后……机器人开始破解难度 5 了。

Where we are now

我们现在的处境

Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through. 今天，git.kernel.org 每天收到约 600 万次请求，要求查看随机提交。其中，66% 的请求仍然被 Anubis 挑战拦截，但有 33% 的请求现在能够完成数学计算并成功通过。