本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale

By Sreeja Thummala, Sophie Zeng, Mark Harman, Weini Liu, Eric Liu. Over the last year we’ve been sharing our work leveraging large language models to catch bugs on the fly — a novel approach we call Catching Just-in-Time Tests (JiTTests). Now, we’re sharing the results and positive impact we’ve seen with Catching JiTTests deployed into production. Our paper, “Just-in-Time Catching Test Generation at Meta,” fully details our initial deployment and testing with Catching JiTTests.

在过去的一年中，我们一直在分享利用大型语言模型（LLM）实时捕获漏洞的工作，这是一种我们称之为“即时测试”（JiTTests）的新颖方法。现在，我们分享了将“即时测试”部署到生产环境后所见到的成果和积极影响。我们的论文《Meta 的即时捕获测试生成》详细介绍了我们最初的部署和测试情况。

Every code change improves our platform, but it also introduces new complexity. With hundreds of millions of lines of code constantly evolving at Meta, traditional testing can’t keep pace. Catching risks proactively, before code lands, enables us to identify risks earlier, respond more effectively, and deliver reliable experiences at scale.

每一次代码变更都会改进我们的平台，但也会引入新的复杂性。在 Meta，数亿行代码在不断演进，传统的测试方法已无法跟上步伐。在代码落地前主动捕获风险，使我们能够更早地识别风险、更有效地响应，并大规模地提供可靠的体验。

Over the last year, we’ve shared our research into applying large language models (LLMs) to generate software tests. More specifically, we’ve shared our work with Just-in-Time Tests (JiTTests), a novel testing approach where we leverage LLMs to catch bugs on the fly, even ones that traditional testing might not catch. In the five months since we first unveiled this work, we’ve deployed our Catching JiTTests approach to determine its scalability and applicability to the wider industry.

在过去的一年中，我们分享了关于应用大型语言模型（LLM）生成软件测试的研究。更具体地说，我们分享了关于“即时测试”（JiTTests）的工作，这是一种新颖的测试方法，我们利用 LLM 实时捕获漏洞，甚至是传统测试可能无法捕获的漏洞。自我们首次公布这项工作以来的五个月里，我们已经部署了“即时测试”方法，以确定其可扩展性和对更广泛行业的适用性。

Today, Catching JiTTests are no longer a research prototype; they’re part of Meta’s infrastructure, and they’re helping our engineers catch more bugs with less load and a smaller chance of false positives. The Advantage of Catching JiTTests: Most AI-based approaches to preventing bugs in code use LLMs and/or agentic AI to review code and detect faults based on prior experience. They work through the same review process that a human reviewer would, only faster.

如今，“即时测试”不再仅仅是一个研究原型；它们已成为 Meta 基础设施的一部分，并正在帮助我们的工程师以更低的负载和更小的误报率捕获更多漏洞。“即时测试”的优势在于：大多数基于人工智能的代码防错方法使用 LLM 和/或智能体 AI 来审查代码，并根据过往经验检测故障。它们的工作方式与人类审查员的审查过程相同，只是速度更快。

Catching JiTTests is a fundamentally different approach (Table 1): It generates executable tests that demonstrate a concrete behavioral difference introduced by the change. While it requires more computational steps, it catches classes of bugs that LLM- or agent-based code reviews might not.

“即时测试”是一种根本不同的方法（表 1）：它生成可执行的测试，用以证明代码变更所引入的具体行为差异。虽然它需要更多的计算步骤，但它能捕获那些基于 LLM 或智能体的代码审查可能无法发现的漏洞类别。

How We Implemented Catching JiTTests at Meta: JiTTests are automatically generated and designed to fail, surfacing bugs in code changes (diffs) before they land in production. When they’re applied to a codebase as large as Meta’s, one challenge is distinguishing strong catches (i.e., true-positive failures that reveal real bugs) from weak catches (i.e., tests that may be false positives).

我们在 Meta 如何实现“即时测试”：JiTTests 是自动生成的，旨在通过失败来在代码变更（diffs）进入生产环境之前暴露漏洞。当它们应用于像 Meta 这样庞大的代码库时，一个挑战是如何区分强捕获（即揭示真实漏洞的真阳性失败）和弱捕获（即可能是误报的测试）。

It boils down to a specific question: For every proposed diff, how can we automatically generate tests that find the most severe bugs before they cause severe issues, and do so with a high true-positive rate and without impeding the velocity of our engineers? To address this, we developed two tandem workflows: An “intent-aware” workflow that infers the intention and risks of a code change and targets test generation toward likely failure points. A “dodgy diff” workflow that treats every code change as if it were a mutant/bug and generates tests to expose its behavioral differences without reasoning about intent. It implicitly asserts that a diff is buggy (hence the term “dodgy”).

这归结为一个具体问题：对于每一个提议的变更，我们如何自动生成测试，在最严重的漏洞造成严重问题之前将其发现，并确保高真阳性率且不阻碍工程师的开发速度？为了解决这个问题，我们开发了两个串联的工作流：一个“意图感知”工作流，用于推断代码变更的意图和风险，并将测试生成目标指向可能的故障点；一个“可疑变更”（dodgy diff）工作流，将每一次代码变更视为一个突变体/漏洞，并生成测试来暴露其行为差异，而不去推断意图。它隐含地断定变更是有漏洞的（因此得名“dodgy”）。

To filter false positives, we employed a traditional rule-based approach combined with an ensemble approach we call “LLM-as-Judge,” which aggregates judgements from several state-of-the-art (SOTA) LLMs, all tasked with classifying weak catches as likely true or false positives. We also maintained human review, with an eye toward reducing review load. Our goal is to maintain the consistency of human labeling but reduce the likelihood of an engineer being brought in to review a weak catch.

为了过滤误报，我们采用了传统的基于规则的方法，结合了一种我们称为“LLM 作为裁判”（LLM-as-Judge）的集成方法，该方法汇总了来自多个最先进（SOTA）LLM 的判断，所有这些模型都被分配了将弱捕获分类为可能的真阳性或误报的任务。我们还保留了人工审查，旨在减少审查负担。我们的目标是保持人工标注的一致性，同时减少工程师被要求审查弱捕获的可能性。

How It Works: The entire system workflow happens in five stages: Stage 1: Targeting and Triggering. The system runs overnight on spare capacity and targets diffs submitted the previous day that are deemed “risky” using a mechanism similar to Diff Risk Score (DRS) and trained on previous code changes. This helps ensure test generation is focused on diffs that are most likely to contain severe regressions.

工作原理：整个系统工作流分为五个阶段：第一阶段：定位与触发。系统在夜间利用闲置容量运行，并针对前一天提交的被认为“有风险”的变更，使用类似于“变更风险评分”（DRS）且基于过往代码变更训练的机制进行定位。这有助于确保测试生成集中在最可能包含严重回归的变更上。

Stage 2: Two Diff-Aware Test-Generation Workflows. The dodgy diff workflow treats each diff as if it were a mutant and uses mutation-guided LLM-based test generation to produce tests that distinguish the diff’s behavior from its parent. The goal here is to maximize the pool of weak catches and with it the chances of finding a strong catch among them. At the same time, the intent-aware workflow uses an LLM to infer the diff’s intention using the code changes, diff title, and summary. The LLM then describes the “risks”— the ways an attempt to implement the intent could produce errors. It then uses those risk descriptions to construct mutants — each representing a bug that could be introduced while implementing the inferred intent. Tests are generated to kill each mutant (i.e., distinguish mutant behavior from parent). Those tests are then run on the actual diff. Any that fail are harvested as weak catches.

第二阶段：两个感知变更的测试生成工作流。“可疑变更”工作流将每个变更视为一个突变体，并使用基于 LLM 的突变引导测试生成，产生能够区分变更与其父版本行为的测试。这里的目标是最大化弱捕获池，从而增加在其中发现强捕获的机会。同时，“意图感知”工作流使用 LLM，通过代码变更、变更标题和摘要来推断变更意图。LLM 随后描述“风险”——即尝试实现该意图时可能产生错误的方式。然后，它利用这些风险描述构建突变体——每一个都代表在实现推断意图时可能引入的漏洞。生成的测试旨在“杀死”每个突变体（即区分突变体行为与父版本）。这些测试随后在实际变更上运行。任何失败的测试都会被收集为弱捕获。

Stage 3: LLM-as-Judge Ensemble (Automated Assessment). Once weak catches are generated, the system must determine whether each is a false positive or actually a strong catch (true positive). The LLM-as-Judge ensemble consists of a collection of LLMs that review: The weak-catch test-case code. The test-execution outcome (failure code, message, execution trace). The code changes between parent and child (the diff). The inferred diff intent.

第三阶段：LLM 作为裁判集成（自动评估）。一旦生成了弱捕获，系统必须确定每一个是误报还是真正的强捕获（真阳性）。“LLM 作为裁判”集成由一组 LLM 组成，它们审查：弱捕获测试用例代码、测试执行结果（失败代码、消息、执行跟踪）、父版本和子版本之间的代码变更（diff）、以及推断出的变更意图。

Stage 4: Rule-Based Assessor (RubFake). We complement LLM-as-Judge with a more traditional, rule-based approach called Rule-Based False-Positive Killing Environment (RubFake). LLM-as-Judge has rich semantics, but its judgements are non-deterministic and challenging to verify. By contrast, RubFake provides verified, deterministic judgments. Working together, RubFake and LLM-as-Judge produce an overall result.

第四阶段：基于规则的评估器（RubFake）。我们用一种更传统的、基于规则的方法补充了“LLM 作为裁判”，称为“基于规则的误报消除环境”（RubFake）。“LLM 作为裁判”具有丰富的语义，但其判断是非确定性的，且难以验证。相比之下，RubFake 提供经过验证的确定性判断。RubFake 和“LLM 作为裁判”协同工作，产生最终结果。