An update on the scraper situation

关于爬虫现状的更新

Our article “Fighting the AI scraper bot scourge”, published in early 2025, discussed the problem of widespread scraping of web sites in search of training data for large language models and related projects. This activity overwhelms sites with traffic. Over a year after that article is published, the problem is still growing. The hammering of sites by shadowy actors has reached new heights, and the open web is becoming increasingly difficult to maintain. Where is this traffic coming from, and what can be done about it?

我们在 2025 年初发表的《对抗 AI 爬虫机器人的祸害》一文中，讨论了为大型语言模型及相关项目搜集训练数据而进行的广泛网站爬取问题。这种活动导致网站流量不堪重负。在该文章发表一年多后，这个问题仍在持续恶化。阴暗势力对网站的猛烈攻击已达到新的高度，开放网络正变得越来越难以维护。这些流量从何而来？我们又能做些什么？

Residential proxies

住宅代理

As was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

正如去年所描述的那样，爬虫攻击来自网络上数量庞大的源头。在几个小时内看到来自数百万个独立 IP 地址的协同请求并不罕见，每个 IP 对网站的访问次数最多只有两三次。攻击者控制的数据（如 User-Agent 字段）完全是虚构的；每一次点击都伪装成普通人类用户使用浏览器访问。虽然有办法区分它们——例如，机器人通常不会抓取图片或 CSS——但等到做出判断时，该地址往往就不会再被使用了。此时再封锁该地址纯属浪费时间。

This traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term “residential proxies” is used to describe systems that are used in this way.

这些流量主要来自住宅和移动网络，并由中央指挥控制节点引导。普通系统上被安装了软件，接收来自控制节点的指令，按需抓取网页，并将结果数据转发回控制器。在大多数情况下，这种活动是在设备所有者不知情或未同意的情况下发生的。“住宅代理”一词正是用来描述以这种方式被利用的系统。

There are a few different (on the surface, at least) types of operator running residential-proxy networks to attack web sites. One type is purely criminal, running scrapers on systems that have been compromised with some sort of malware. At the beginning of the year, Google acted to take down a bot network called IPIDEA and provided a lot of information about how these operations work. The shutdown of IPIDEA correlated with a significant reduction in scraper traffic here at LWN; things were relatively peaceful for a few months. That period of peace has since come to an end, though.

运营住宅代理网络以攻击网站的运营商有几种（至少表面上）不同的类型。一种是纯粹的犯罪分子，在被某种恶意软件入侵的系统上运行爬虫。今年年初，谷歌采取行动取缔了一个名为 IPIDEA 的僵尸网络，并提供了大量关于这些操作如何运作的信息。IPIDEA 的关闭与 LWN 爬虫流量的显著减少相关；在那之后的几个月里，情况相对平静。然而，那段平静期现已结束。

More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact. The second sort of operator works more overtly, pretending to a degree of legitimacy and offering “ethically sourced” IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits.

最近，媒体流媒体设备被发现是恶意爬虫软件的主要载体。有时这些设备在出厂时就被植入了恶意软件；有时则是因为安全防护薄弱，在投入使用后轻易被入侵。第二类运营商运作得更为公开，假装具有一定程度的合法性，并提供所谓的“合乎道德来源”的 IP 地址。一家名为 Bright Data 的公司是其中最突出的代表之一；它乐于宣传自己绕过网站访问控制和流量限制的能力。

Bright Data offers a “free” VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user’s device — to become a part of the company’s residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites. There are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users’ network connections.

Bright Data 提供“免费”VPN 服务；用户只需允许 Bright Data 通过其设备路由流量——换句话说，就是成为该公司住宅代理网络的一部分。每一部使用此 VPN 的手机或其他设备，都成为了攻击网站的又一个终端。市面上还有许多此类运营商的例子；它们通常提供一个库，供应用开发者集成到自己的产品中，并通过劫持用户的网络连接来获取报酬。

In general, these companies range from those that aspire toward some appearance of legitimacy, advertising “GDPR compliance” for example, to others that are just overtly sleazy. While these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.

总的来说，这些公司良莠不齐，有的渴望表现出某种合法性（例如宣传“符合 GDPR”），有的则纯粹是卑劣无耻。虽然这些住宅代理网络被用于网站爬取，但值得强调的是，这些运营商有能力运行代码，访问数百万台设备所连接的任何网络资源。如果认为这种访问权限仅用于爬虫，那未免太天真了。

Then, of course, there are the high-profile companies developing models as their core business. These companies do their own scraping; the traffic that can be easily attributed to them is clearly identified in the user-agent field and, as a general rule, observes measures like robots.txt. They, too, will scrape an entire site, repeatedly, seemingly on the theory that articles written in 2003 might somehow have changed in the last day, but they do not generate overwhelming amounts of traffic from millions of systems and are not the biggest problem.

当然，还有那些以开发模型为核心业务的知名公司。这些公司会进行自己的爬取工作；可以轻易归因于它们的流量在 User-Agent 字段中有明确标识，并且通常会遵守 robots.txt 等协议。它们也会反复爬取整个网站，似乎是基于一种假设：即 2003 年写的文章在过去一天里可能会发生变化。但它们不会从数百万个系统中产生海量流量，因此并不是最大的问题。

What isn’t clear is who is using the residential proxies; somebody is paying them to run these attacks on web sites. There is no evidence (that I am aware of) that the frontier-model companies are using those networks. If it were to turn out that they are doing so, though, the increase in global astonishment would barely register. Those companies are feeding their models somehow, they are not forthcoming about how they get their training data, and they have not distinguished themselves with their level of respect toward content creators — or toward anybody who might have concerns about their operations.

目前尚不清楚是谁在使用这些住宅代理；有人在付钱让他们对网站发动这些攻击。据我所知，没有证据表明前沿模型公司正在使用这些网络。不过，如果事实证明他们确实在使用，全球对此的震惊程度恐怕也有限。这些公司以某种方式喂养着他们的模型，他们对获取训练数据的方式讳莫如深，且在尊重内容创作者——或任何对他们的运营有顾虑的人——方面，表现也并不出色。

For every public model, though, there must be a vast number of undercover models. Many companies are surely trying to build their own; after all, we are reliably informed that AI is going to take over the world and the companies that come out on top of that race will be worth untold amounts of money. There must be shadowy government agencies in many countries working on their own models and groping for training data wherever they can find it. Large-scale criminal organization.

然而，每一个公开的模型背后，必然存在着大量不为人知的模型。许多公司肯定都在尝试构建自己的模型；毕竟，我们被反复告知 AI 将接管世界，而在那场竞赛中胜出的公司将拥有无法估量的财富。许多国家的阴暗政府机构也一定在开发自己的模型，并竭尽所能搜寻训练数据。大规模的犯罪组织亦是如此。