Checking what an LLM writes costs more than the writing

Getting a model to write a page is the easy part. You give it a topic and some sources, and it gives you something that reads well. If that were the whole job, a reference site would take a weekend. This year I built a Spanish political test, and around it a set of reference pages; one for each thinker the test is anchored on, a glossary, and an essay comparing each pair of parties. Agents drafted most of that text. The drafting was fast. Checking it is what took the work, and this post is about that part. The website is testpolitica.

让模型写一页内容是件容易的事。你给它一个主题和一些资料来源，它就能写出读起来很通顺的内容。如果工作仅限于此，做一个参考网站只需一个周末。今年我建立了一个西班牙政治测试，并围绕它制作了一系列参考页面：每个测试所依托的思想家各一个页面、一个术语表，以及一篇比较各党派对的论文。这些内容大部分是由智能体起草的。起草速度很快，但核对工作却耗费了大量精力，这篇文章讲的就是这部分内容。该网站是 testpolitica。

Models quote from memory If you ask a model to read a page and quote it, it reads the page and then writes the quote from memory. The result looks right and is often not literal. Sometimes it comes back translated. I measured it a few times. Asking a small model for literal quotes, between 24% and 35% of them survived a check against the source. In the glossary, written before I knew this, 141 of 285 quotes did not appear in their own source. One entry had translated every quote into Spanish while its sources were in English, French and Russian, so nine of its ten failed. That is, a big part of the LLM generated content on the Internet is completely wrong.

模型是凭记忆引用的。如果你让模型阅读一个页面并进行引用，它会阅读该页面，然后凭记忆写出引文。结果看起来是对的，但往往不是逐字引用的。有时它还会把引文翻译出来。我测量过几次。要求一个小模型进行逐字引用时，只有 24% 到 35% 的引文能通过与原文的核对。在术语表中，由于我当时还不知道这一点，285 条引文中竟有 141 条在原文中找不到。其中一个条目将所有引文都翻译成了西班牙语，而其来源分别是英语、法语和俄语，结果十条引文中有九条是错误的。也就是说，互联网上很大一部分由大语言模型生成的内容完全是错误的。

Download the sources first What fixed it was taking the quoting away from the model: Download the sources with curl before any model sees them. Check that each file is what it says it is. I once had a book of Finnish poetry saved as Kant’s essay on the Enlightenment, and three PDFs that were all the same 25 KB error page. Give the writer the files and tell it not to quote anything that is not in one of them. For research claims the model does not quote at all: it returns the claim, the URL and a few keywords, and a script picks the sentence out of the downloaded HTML. With that, 36 of 39 research quotes came out literal, and the thinker biographies ended with 95% of their quotes verifying against the source. Before trusting a failure, run the verifier on quotes you know are good.

先下载资料来源。解决这个问题的办法是剥夺模型的引用权：在任何模型看到资料之前，先用 curl 下载它们。检查每个文件是否名副其实。我曾经遇到过一本芬兰诗集被保存为康德的《启蒙论》，还有三个 PDF 文件其实都是同一个 25 KB 的错误页面。把文件交给写作模型，并告诉它不要引用任何不在这些文件中的内容。对于研究性声明，模型根本不进行引用：它只返回声明、URL 和几个关键词，然后由脚本从下载的 HTML 中提取出句子。通过这种方式，39 条研究引文中有 36 条是逐字准确的，思想家传记中 95% 的引文都能通过原文核对。在信任一个失败的核对结果之前，先用你确定正确的引文运行一下核对程序。

When the quote does not support the claim A literal quote proves the text is in the source. It does not prove that it supports the sentence next to it. The claim “the PP backed Sánchez’s investiture in 2023” is false, and it still got a literal quote with a perfect score; that is, a sentence about Sánchez being appointed. So there is a second check, a model acting as judge that reads each claim with its quote and says whether it holds. A partial match only survives if the judge rewrites the claim into something the quote does support. Later I ran a review with agents over the 26 riskiest pages. It caught things no script would… a page attributing a court case and a conviction to a living person with no source behind it, a quote about Putin whose source was about Kennedy, and an argument credited to Hayek that is Mises’. Reviewers also over-report, so every finding went to a second agent whose job was to refute it, and 45% of them fell.

当引文不支持声明时。逐字引用只能证明文本确实存在于来源中，并不能证明它支持旁边的句子。例如“人民党（PP）在 2023 年支持了桑切斯的就职”这一声明是错误的，但它仍然获得了一个完美的逐字引用得分，即一句关于桑切斯被任命的句子。因此需要进行第二次检查，由一个模型担任法官，阅读每一条声明及其引文，并判断其是否成立。只有当法官将声明重写为引文确实支持的内容时，部分匹配才能通过。后来，我让智能体对 26 个风险最高的页面进行了审查。它发现了一些脚本无法发现的问题……比如一个页面将一起法庭案件和定罪归咎于一个在世的人，但没有任何来源支持；一条关于普京的引文，其来源却是关于肯尼迪的；还有一个归功于哈耶克的论点其实是米塞斯的。审查员也会过度报告，所以每一项发现都会交给第二个智能体进行反驳，结果 45% 的发现被推翻了。

Wording and context Some fixes had nothing to do with facts. “Fabricated” became “started” in a sentence about a named journalist, and “the best ammunition against” became “the standard reference”. No check flags that kind of thing. Someone made of flesh has to read it. So be prepared before starting a content webstie You need to Budget for checking, as the writing is the cheap part now. Do not let the model produce what you cannot check and give it files instead of URLs, and let code extract the quotes. I also recommende to test the verifier before you trust it and also keeping a threshold. In many cases a shorter hand-written version should be chosen instead of an entry with fewer than four verified quotes. Search engines are getting better at telling thin generated content apart, so this matters for ranking too, as Google might flag your site if it’s basically poor AI-generated content. Next time I start a content site, the verifier will be the first thing I write. Actually, most part of the site was heavily reviewed or just handwritten.

措辞和语境。有些修正与事实无关。在关于一位记者的句子中，“捏造”变成了“开始”，“最好的反击弹药”变成了“标准参考”。没有任何检查程序能标记出这类问题，必须由真人来阅读。因此，在开始建立内容网站之前要做好准备。你需要为核对工作留出预算，因为现在写作反而是廉价的部分。不要让模型生成你无法核对的内容，给它文件而不是 URL，并让代码来提取引文。我还建议在信任核对程序之前先进行测试，并设定一个阈值。在许多情况下，如果一个条目的核对引文少于四条，应该选择更短的手写版本。搜索引擎在区分低质量生成内容方面越来越强，所以这对排名也很重要，因为如果你的网站基本上是劣质的 AI 生成内容，谷歌可能会标记你的网站。下次我再建立内容网站时，核对程序将是我编写的第一件事。事实上，该网站的大部分内容都经过了严格审查或直接由人工编写。