Can LLMs read robots.txt? Easily. Can they write one? Only one of nine got every policy right.

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Kaggle Benchmarking Challenge Submission. This is a submission for the Kaggle Benchmarking Challenge. How this was made: I built this with an AI coding agent (Claude). It wrote most of the code and the first draft of this post. I reviewed the code, reran the checks and edited the post, and I stand behind every number. All numbers come from the Kaggle runs linked below.

Kaggle 基准测试挑战提交。这是针对 Kaggle 基准测试挑战的一份提交。制作过程如下：我使用 AI 编码代理（Claude）构建了此项目。它编写了大部分代码和本文的初稿。我审查了代码，重新运行了检查并编辑了文章，我对每一个数字负责。所有数字均来自下方链接的 Kaggle 运行结果。

What I Benchmarked. Every web crawler, from Googlebot to the AI agents now browsing on our behalf, is supposed to check a site’s robots.txt before fetching a page. Site owners are writing more of these files than ever, often with a chatbot’s help, to keep AI crawlers out. So I asked two questions that have exact answers: Can a model read a robots.txt? Given the file, a crawler name and a URL, is the fetch allowed? Can a model write one? Given a plain-English policy (“keep GPTBot out of /blog/, keep everyone out of /private/ except /private/press/”), does the file it writes actually do that?

我测试的内容。每一个网络爬虫，从 Googlebot 到现在代表我们浏览网页的 AI 代理，在抓取页面之前都应该检查网站的 robots.txt 文件。网站所有者比以往任何时候都更频繁地编写这些文件，通常是在聊天机器人的帮助下，以阻止 AI 爬虫。因此，我提出了两个有确切答案的问题：模型能读取 robots.txt 吗？给定文件、爬虫名称和 URL，抓取是否被允许？模型能编写 robots.txt 吗？给定一个简单的英语策略（“禁止 GPTBot 访问 /blog/，禁止所有人访问 /private/，但允许访问 /private/press/”），它编写的文件真的能实现该策略吗？

Robots.txt has a real standard, RFC 9309 (2022), so neither question needs a judge model. Every answer is checked by a reference parser. The three tasks: Read: rules (a small file testing one rule of the standard), Read: real sites (the real robots.txt of 41 public sites), and Write: policy (a 3–6 line policy).

Robots.txt 有一个真正的标准，即 RFC 9309 (2022)，因此这两个问题都不需要评估模型。每个答案都由参考解析器进行检查。这三项任务分别是：读取规则（测试标准中单条规则的小文件）、读取真实网站（41 个公共网站的真实 robots.txt）以及编写策略（3-6 行的策略）。

A baseline that surprised me: Python’s own urllib.robotparser gets only 70.4% of the rules cases and 73.3% of the real-site cases right. It lets the first matching rule win (the standard says the longest), and it doesn’t understand * or $. If your scraper uses the standard library, it is misreading about one robots.txt question in four.

一个让我惊讶的基准线：Python 自带的 urllib.robotparser 在规则案例中仅有 70.4% 的正确率，在真实网站案例中仅有 73.3% 的正确率。它让第一条匹配的规则生效（标准规定应为最长匹配），并且它不理解 * 或 $。如果你的爬虫使用该标准库，那么它在 robots.txt 相关问题上大约有四分之一是误读的。

Findings: 1. Reading is nearly solved, and the hard cases are the ones the standard library gets wrong. 2. Writing is much harder, and “almost right” is the failure to watch. Across 360 attempts (9 models × 40 policies), only 156 files did everything the policy asked. 3. Parameters and file extensions are where files break. Policies with a query-parameter rule or a file-extension rule were significantly less likely to be fully correct.

研究结果：1. 读取任务已基本解决，困难的案例正是标准库出错的那些。2. 编写任务要困难得多，“几乎正确”是需要警惕的失败。在 360 次尝试（9 个模型 × 40 个策略）中，只有 156 个文件完全实现了策略要求。3. 参数和文件扩展名是文件出错的地方。包含查询参数规则或文件扩展名规则的策略，其完全正确的概率显著降低。