Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

Summary Project Zero compared two automated tools used by Google to find vulnerabilities in Chrome’s JavaScript engine, V8, on a regular basis. Fuzzilli is a generative JavaScript engine fuzzer and BigSleep is an LLM-based bug finding system. In the past six months, BigSleep found more vulnerabilities than Fuzzilli (45 versus 37) with a 25% overlap in the specific bugs reported. Since this comparison methodology has a tendency to underestimate overlap, we also performed dedicated experiments to test theoretical overlap in vulnerabilities reported since 2023. These experiments estimated overlap at roughly 60%. Regardless of which value is accurate, a substantial portion of vulnerabilities were discoverable by only LLMs or only fuzzing. While it is extremely promising that LLMs have become able to find so many vulnerabilities so quickly, fuzzing and LLMs are still complementary approaches. We recommend vendors perform both fuzzing and agentic analysis of their code to maximize the number of vulnerabilities that get reported before they impact users.

Project Zero 比较了谷歌用于定期查找 Chrome JavaScript 引擎 V8 中漏洞的两种自动化工具。Fuzzilli 是一种生成式 JavaScript 引擎模糊测试工具，而 BigSleep 是一种基于大语言模型（LLM）的漏洞查找系统。在过去六个月中，BigSleep 发现的漏洞数量超过了 Fuzzilli（45 个对 37 个），其中报告的具体漏洞重叠率为 25%。由于这种比较方法往往会低估重叠部分，我们还进行了专门的实验，以测试自 2023 年以来报告的漏洞在理论上的重叠情况。这些实验估计重叠率约为 60%。无论哪个数值准确，都有很大一部分漏洞仅能通过大语言模型或仅通过模糊测试发现。虽然大语言模型能够如此迅速地发现大量漏洞的前景非常广阔，但模糊测试和大语言模型仍然是互补的方法。我们建议供应商同时对其代码进行模糊测试和代理分析，以最大限度地增加在漏洞影响用户之前被报告的数量。

Introduction Fuzzing has long been successfully applied to software projects such as Chrome or V8 (Chrome’s JavaScript engine) to detect bugs. Recently, the arrival of powerful LLMs able to read, reason about, and write code is causing a transformative change to software development and software security. With hundreds, sometimes thousands of new bugs and vulnerabilities having been discovered in well-fuzzed targets, a natural question arises: how do new LLMs compare to “traditional” fuzzers? Are they starting to replace them or do the two approaches complement each other? To help answer this, we look at three key questions: How many bugs does each approach find in the real world? What is the overlap between the two approaches? What is the “shape” of bugs discovered by one system but not the other?

引言：长期以来，模糊测试已成功应用于 Chrome 或 V8（Chrome 的 JavaScript 引擎）等软件项目，用于检测漏洞。最近，能够阅读、推理和编写代码的强大大语言模型的出现，正在给软件开发和软件安全带来变革性的变化。随着在经过充分模糊测试的目标中发现了数百甚至数千个新漏洞，一个自然而然的问题出现了：新的大语言模型与“传统”模糊测试工具相比如何？它们是开始取代后者，还是这两种方法相辅相成？为了帮助回答这个问题，我们研究了三个关键问题：每种方法在现实世界中发现了多少漏洞？这两种方法之间的重叠部分是什么？由一个系统发现而另一个系统未发现的漏洞的“形态”是什么？

For this blog post, we will limit ourselves to two concrete systems: Fuzzilli, a JavaScript engine fuzzer, and BigSleep, an agentic bug finding system. Both of these systems run in production and automatically look for bugs in newly added code, which makes them fairly directly comparable. Specifically, Fuzzilli runs continuously across many instances, with the first ones usually picking up new changes within a few hours, while BigSleep currently runs once per day and is prompted to review new commits and write proof-of-concept testcases for any bugs it discovers. Both systems report bugs through Clusterfuzz, which will do basic crash triaging, then file bug reports in Chrome’s issue tracker. We also limit ourselves to V8 as a target, a high-performance JavaScript and WebAssembly engine with a sophisticated and mature testing and fuzzing setup.

在本篇博文中，我们将仅限于讨论两个具体的系统：JavaScript 引擎模糊测试工具 Fuzzilli 和代理漏洞查找系统 BigSleep。这两个系统都在生产环境中运行，并自动查找新添加代码中的漏洞，这使得它们具有相当的可比性。具体来说，Fuzzilli 在多个实例上持续运行，通常在几小时内就能捕捉到新的变更，而 BigSleep 目前每天运行一次，并被提示审查新的提交，并为它发现的任何漏洞编写概念验证测试用例。这两个系统都通过 Clusterfuzz 报告漏洞，Clusterfuzz 会进行基本的崩溃分类，然后在 Chrome 的问题跟踪器中提交漏洞报告。我们还将目标限制为 V8，这是一个高性能的 JavaScript 和 WebAssembly 引擎，具有复杂且成熟的测试和模糊测试设置。

In particular, Fuzzilli is a special-purpose fuzzing tool designed to find bugs in these engines. The usually complicated nature of these bugs makes for an interesting “playground” to compare the two approaches’ ability to find deep and complex bugs in well-tested software. We leave it to future work to perform similar analysis on a broader set of targets, fuzzers, and agentic bug finding systems. We will now look into each of the three main questions in a dedicated section. Each section will first summarize the general approach, then describe the methodology in detail (often as bullet points, so you can skip those blocks if you don’t care too much about the details) before exploring the results. We conclude with an attempt at interpreting the results to get a picture of the current state of fuzzing vs. AI.

特别是，Fuzzilli 是一种专门用于在这些引擎中查找漏洞的模糊测试工具。这些漏洞通常具有复杂的性质，这为比较两种方法在经过充分测试的软件中查找深层和复杂漏洞的能力提供了一个有趣的“试验场”。我们将对更广泛的目标、模糊测试工具和代理漏洞查找系统进行类似分析的工作留给未来。现在，我们将分别在专门的章节中探讨这三个主要问题。每一节将首先总结一般方法，然后详细描述方法论（通常以项目符号形式，如果您不太关心细节，可以跳过这些部分），然后再探讨结果。最后，我们将尝试解读这些结果，以了解模糊测试与人工智能现状的概貌。

Part 1: Production Statistics This is the simplest part to analyze, although with some caveats we’ll see in a bit. The basic idea is to query issues from Chromium’s bugtracker (where V8 bugs are tracked) and extract various statistics from there. This is especially easy since there’s a dedicated hotlist for each of the two systems. We limit our analysis to the six months from April to September 2026 as the automatic BigSleep pipeline only became active in April this year. As such, our raw data comes from the following two bug tracker queries: Fuzzilli: hotlistid:5433121 created:2026-04-01..2026-09-30 BigSleep: hotlistid:8067258 created:2026-04-01..2026-09-30 From there, the raw numbers of bug reports are: Fuzzilli: 316 BigSleep: 441 Small win for BigSleep. However, this includes all types of bugs, and BigSleep initially also found a lot of configuration and integration issues in V8, for example crbug.com/503869331 where the unsafe flag blocking mechanism didn’t work properly yet. These are legitimate issues, but less relevant for our comparison since there’s little real-world security impact.

第一部分：生产统计数据。这是最容易分析的部分，尽管有一些我们稍后会看到的注意事项。基本思路是从 Chromium 的问题跟踪器（V8 漏洞在此处跟踪）查询问题，并从中提取各种统计数据。这尤其容易，因为这两个系统都有专门的热门列表。我们将分析限制在 2026 年 4 月至 9 月的六个月内，因为自动化的 BigSleep 管道直到今年 4 月才开始活跃。因此，我们的原始数据来自以下两个问题跟踪器查询：Fuzzilli: hotlistid:5433121 created:2026-04-01..2026-09-30；BigSleep: hotlistid:8067258 created:2026-04-01..2026-09-30。由此得出的漏洞报告原始数量为：Fuzzilli: 316，BigSleep: 441。BigSleep 小胜。然而，这包括了所有类型的漏洞，而且 BigSleep 最初还在 V8 中发现了很多配置和集成问题，例如 crbug.com/503869331，其中不安全标志阻止机制尚未正常工作。这些是合法的问题，但与我们的比较相关性较低，因为它们对现实世界的安全影响很小。

One easy way to get rid of these is to look specifically at vulnerabilities by limiting the results to type:vulnerability: Fuzzilli: 178 BigSleep: 95 Suddenly a clear win for the fuzzers? Not quite. There are still some problems with this data, notably duplicates and unactionable reports (e.g. bugs closed as WAI). At first, this seems like a simple thing to fix: just include another constraint such as status:(fixed | verified). Unfortunately, things are a bit more complicated. Take for example Fuzzilli bug crbug.com/511935486. The events on this bug are basically: The bug is filed by Clusterfuzz on May 11 and assigned to an engineer The bug is closed by Clusterfuzz as the testcase no longer reproduces later that same day As such, this bug received zero developer interactions. Most likely what happened here is that this bug was a duplicate of another bug, crbug.com/511210507, but Clusterfuzz didn’t realize it as the crash signature (derived from the stack trace) was slightly different. When the other bug received a fix, the testcase stopped reproducing on Clusterfuzz, which closed the issue. Maybe the best thing we can do is to look at “actionable” bugs, defined as those that are not marked as duplicates or WAI and that received at least one fix commit referencing them. Then, looking at only Type-Vuln bugs filters out a

消除这些问题的一个简单方法是通过将结果限制为 type:vulnerability 来专门查看漏洞：Fuzzilli: 178，BigSleep: 95。模糊测试工具突然大获全胜了吗？并非如此。这些数据仍然存在一些问题，特别是重复报告和不可操作的报告（例如被关闭为 WAI 的漏洞）。起初，这似乎是一个很容易解决的问题：只需包含另一个约束，例如 status:(fixed | verified)。不幸的是，事情要复杂一些。以 Fuzzilli 漏洞 crbug.com/511935486 为例。该漏洞的事件基本上是：该漏洞由 Clusterfuzz 于 5 月 11 日提交并分配给一名工程师；该漏洞在当天晚些时候被 Clusterfuzz 关闭，因为测试用例不再重现。因此，该漏洞没有收到任何开发人员的交互。最可能的情况是，该漏洞是另一个漏洞 crbug.com/511210507 的重复，但 Clusterfuzz 没有意识到这一点，因为崩溃签名（源自堆栈跟踪）略有不同。当另一个漏洞得到修复时，测试用例在 Clusterfuzz 上停止重现，从而关闭了该问题。也许我们能做的最好的事情是查看“可操作”的漏洞，定义为那些未被标记为重复或 WAI，并且至少收到一个引用它们的修复提交的漏洞。然后，仅查看 Type-Vuln 漏洞可以过滤掉一个