Most Googlebots are fake
Most Googlebots are fake
大多数 Googlebot 都是假的
Jul 26 | Written By Bobbie Chen 7月26日 | 作者：Bobbie Chen
I recently saw complaints about Google’s search engine crawler taking down websites by sending them too much traffic: 最近，我看到一些关于谷歌搜索引擎爬虫通过发送过大流量导致网站瘫痪的投诉：
“We had googlebot blast a random customer system and almost cause an outage, this is when I first learnt that google will use it for AI training also. It’s honestly kind of frustrating also because you then search on it and theres (was) nothing on how you are meant to “correctly” tell google to fuck off, and not use it like that.” — User jofzar on Hacker News “我们的一个客户系统被 Googlebot 猛烈抓取，差点导致宕机。我这才知道谷歌也会用它来进行 AI 训练。说实话这很让人沮丧，因为你去搜索相关信息时，根本找不到关于如何‘正确地’拒绝谷歌抓取、不让它那样使用你网站的方法。” —— Hacker News 用户 jofzar
“Google’s web scraping functionality has been acting as a ddos for more than two decades. I’ve seen literally hundreds of reports of them attacking websites and taking them down, where there’s nothing you can do but accept the traffic, or get delisted” — User 20k on Hacker News “谷歌的网页抓取功能在过去二十多年里一直像是在进行 DDoS 攻击。我见过成百上千起关于它们攻击并搞垮网站的报告，你除了接受这些流量，或者被谷歌除名之外，别无他法。” —— Hacker News 用户 20k
But I think these complaints are misguided. This traffic doesn’t come from Google at all. It comes from other people’s bots impersonating Googlebot’s name for misleading and arguably-abusive purposes. Most* Googlebots are fake. 但我认为这些抱怨找错了对象。这些流量根本不是来自谷歌，而是来自其他人的机器人，它们冒充 Googlebot 的名义，进行误导性且可以说是滥用性的行为。大多数* Googlebot 都是假的。
The user agent header is a pinky promise
User-Agent 标头只是一个“拉钩承诺”
When people say that they see traffic from Googlebot, they usually mean that they see Googlebot in the user agent header, usually embedded in a string like this: 当人们说他们看到了来自 Googlebot 的流量时，通常是指他们在 User-Agent 标头中看到了 Googlebot，通常嵌入在这样的字符串中：
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Or, if you’re looking at nginx-style server logs, it might look something like this: 或者，如果你查看 nginx 风格的服务器日志，它看起来可能是这样的：
78.142.228.252 - - [17/Jul/2026:14:57:44 +0000] "GET /blog HTTP/1.1" 200 4821 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
This is all fine and dandy, but this header doesn’t mean it’s the real Googlebot at all. The user agent is a voluntary self-identification. Anyone making an HTTP request can set that user agent value to anything! 这看起来没问题，但这个标头并不代表它就是真正的 Googlebot。User-Agent 是一种自愿的自我标识。任何发起 HTTP 请求的人都可以将该值设置为任何内容！
On the internet, no one knows you’re a dog… and the flip side is that I can claim to be a dog (or a Googlebot) whenever I feel like it. These liars impersonate Googlebot in hopes that it will help them bypass bot controls; indeed, if you look up that IP in my log above, it comes from Virtual Machine Solutions LLC. This is a hosting provider, so someone (not Google) rented a server from them to send this traffic. 在互联网上，没人知道你是一条狗……反过来说，我随时可以声称自己是一条狗（或者 Googlebot）。这些骗子冒充 Googlebot，是希望这能帮助他们绕过机器人防御机制；事实上，如果你查询我上面日志中的那个 IP，它来自 Virtual Machine Solutions LLC。这是一家托管服务提供商，所以有人（不是谷歌）从他们那里租了一台服务器来发送这些流量。
People do this, all the time, and it seems to have hit a new scale recently. Chris Siebenmann theorizes that “it’s a large scale campaign by a single abusive crawler by people who can afford to obtain a lot of servers at a lot of different hosting providers”, rather than a widespread rise in popularity for an old trick. 人们一直在这样做，而且最近似乎达到了新的规模。Chris Siebenmann 推测，“这是一场由单一滥用爬虫发起的大规模行动，背后的人有能力在多家不同的托管服务商处获取大量服务器”，而不是某种旧伎俩的普遍流行。
Lightning Q&A
闪电问答
Q: How do I verify that a request came from the real Googlebot? 问：我该如何验证请求是否来自真正的 Googlebot？
A: Here are Google’s docs for verifying Googlebot; in short, you can spot-check with a couple of host commands or check against Google’s published IP ranges. Most major crawlers publish some kind of IP list, but the details vary; the JAFAR proposal aims to standardize these lists, though it’s still working its way through the standard-making process.
答： 这是谷歌验证 Googlebot 的文档；简而言之，你可以通过几个 host 命令进行抽查，或者对照谷歌公布的 IP 范围进行核对。大多数主流爬虫都会发布某种 IP 列表，但细节各不相同；JAFAR 提案旨在标准化这些列表，尽管它仍处于标准制定过程中。
Q: What if I looked up the IP address and saw it was registered to Google LLC? 问：如果我查询 IP 地址发现它注册在 Google LLC 名下怎么办？
A: I’d still run through the verification process, since “Google LLC” is also the registration for Google Cloud hosting IP addresses. Though personally, I wasn’t able to find any recent examples of this - I wouldn’t be surprised if the Google Cloud anti-abuse team had some kind of controls to crack down on this. 答： 我仍然会进行验证流程，因为“Google LLC”也是谷歌云托管 IP 地址的注册名。不过就我个人而言，我还没找到最近的例子——如果谷歌云的反滥用团队有某种控制措施来打击这种行为，我一点也不会感到惊讶。
Q: I checked and it really is real Googlebot. What should I do? 问：我检查过了，它确实是真正的 Googlebot。我该怎么办？
A: See Google’s docs on how to handle overcrawling of your site. 答： 请参阅谷歌关于如何处理网站被过度抓取的文档。
Q: You mean to tell me people can lie on the internet? 问：你是说人们可以在互联网上撒谎？
A: Yes :( 答： 是的 :(
Q: What if we added another field to stop them from lying? 问：如果我们增加另一个字段来阻止他们撒谎呢？
A: Please refer to RFC 3514, the “evil bit” standard. Unfortunately, bad actors do not comply with this standard. On a more serious note, I’ve written at length about the Web Bot Auth HTTP Signatures proposal for my employer. That proposal is a little more complicated than just another field since it involves a cryptographic signature, but I think it’s a reasonable path to “a user agent you can trust”. 答： 请参考 RFC 3514，“邪恶位”（evil bit）标准。不幸的是，恶意行为者并不遵守这个标准。严肃地说，我曾为我的雇主详细撰写过关于 Web Bot Auth HTTP 签名提案的文章。该提案比仅仅增加一个字段要复杂一些，因为它涉及加密签名，但我认为这是通往“可信 User-Agent”的合理途径。
Q: You’ve convinced me some Googlebots are fake. But is it really “most”? 问：你已经说服我有些 Googlebot 是假的了。但真的是“大多数”吗？
A: Okay, I concede that I don’t have data to prove this. I offer this weaker claim: most stories of disruptive Googlebot traffic are caused by fake Googlebots. In my experience, the real Googlebot is very well-behaved; meanwhile, fake Googlebots are happy to hammer on a site until it explodes. That being said, if you have seen disruptive behavior from Googlebots (real or fake), I’d like to hear about it - reach out at bobbie@digitalseams.com. 答： 好吧，我承认我没有数据来证明这一点。我提出一个较弱的观点：大多数关于 Googlebot 流量干扰的案例都是由假 Googlebot 引起的。根据我的经验，真正的 Googlebot 非常守规矩；而假 Googlebot 则乐于猛烈攻击网站直到其崩溃。话虽如此，如果你见过 Googlebot（无论真假）的干扰行为，我很想听听——请通过 bobbie@digitalseams.com 联系我。