Coding Agents Keep Shipping Silent Failures — Here Is How to Catch Them

编程代理不断交付“静默故障”——如何捕捉它们

Introduction Vibe coding lets anyone build a web app with nothing but a natural language prompt. You ask an LLM agent for what you want, and you get a polished interface in seconds. The promise is that you’ll never have to look at the code again, yet this is far from reality. In reality, our first prompt results in a great UI, but asking for subsequent modifications starts breaking the app.

引言 “Vibe coding”（直觉编程）让任何人只需通过自然语言提示词就能构建 Web 应用。你向大模型（LLM）代理提出需求，几秒钟内就能获得一个精美的界面。其愿景是你永远不需要再看代码，但这与现实相去甚远。实际上，我们的第一个提示词确实能生成很棒的 UI，但后续的修改请求往往会开始破坏应用。

Silent Behavioral Failures A large problem from this is silent failures, where the UI may appear fine at first glance, but is broken under the hood. For example, clicking an “Add to Cart” button might show a success message and update the cart count on the screen, yet nothing is actually written to the database or storage. To debug this, you must interact with the UI, read through logs, and ultimately end up reading the generated code, which defeats the entire purpose of vibe coding.

静默行为故障 由此产生的一个大问题是“静默故障”，即 UI 乍看之下没问题，但底层逻辑已经损坏。例如，点击“加入购物车”按钮可能会显示成功消息并更新屏幕上的购物车数量，但实际上数据库或存储中并没有写入任何数据。为了调试这个问题，你必须与 UI 交互、阅读日志，最终还得去阅读生成的代码，这完全违背了“直觉编程”的初衷。

We conducted a study where we vibe coded real-world apps through iterative steps, and analyzed the resulting silent failures. We observed that even frontier models frequently introduce silent failures across iterations, and categorised them. These include failures in tracking state updates, cross-handler state disconnects, and disconnected UI feedback (e.g success message is shown, but data was not saved).

我们进行了一项研究，通过迭代步骤对真实应用进行“直觉编程”，并分析了由此产生的静默故障。我们观察到，即使是前沿模型在迭代过程中也经常引入静默故障，我们将这些故障进行了分类。其中包括状态更新跟踪失败、跨处理程序状态断连，以及 UI 反馈脱节（例如显示了成功消息，但数据并未保存）。

Example of Silent failure: The user asks an agent to add a promotional-code feature to a vibe-coded shopping application. The agent creates a promo input and an “Apply” button. When the user enters a valid code, the interface displays “Discount applied!” even though the new total is neither persisted nor shown:

静默故障示例： 用户要求代理为一个“直觉编程”生成的购物应用添加优惠码功能。代理创建了一个优惠码输入框和一个“应用”按钮。当用户输入有效代码时，界面显示“折扣已应用！”，尽管新的总价既没有被持久化，也没有被显示出来：

function applyPromo() { 
  const getById = id => document.getElementById(id); 
  const code = getById('promo-input').value; 
  if (code === 'SAVE20') { 
    // read and update cart state 
    let tot = parseFloat(localStorage.getItem('cartTotal')); 
    tot *= 0.8; 
    // Update success message in UI 
    getById('promo-msg').innerText = 'Discount applied!'; 
    // BUG: did not persist nor display updated total 
    // localStorage.setItem('cartTotal', tot); // missing 
  }
}

Why Current Verification Fails If our app is broken, asking agents to debug often leads to false promises and remaining broken code. And very often, the end-user often does not know what bugs are hidden in the code in the first place. In all these cases, current verification methods are inadequate for vibe coders.

为什么现有的验证方法会失效 如果我们的应用坏了，要求代理进行调试往往只会得到虚假的承诺和依然损坏的代码。而且通常情况下，终端用户根本不知道代码中隐藏了什么 Bug。在所有这些情况下，现有的验证方法对于“直觉编程”用户来说都是不够的。

• LLMs as Judges: Asking LLMs to self-debug, or identify bugs in the code is unreliable, as models can hallucinate or just miss certain bugs. Even frontier models like Claude Opus 4.7, DeepSeek V3, and Gemini Pro frequently miss edge cases, hallucinate fixes, and fail to understand complex data flows.

• Unit Tests: Writing tests requires writing more code to check the generated code. It often does not check UI-to-backend integration, It is limited to the exact scenarios you specify, and is inaccessible to non-programmers. LLMs may be able to write unit tests but it is not guaranteed to be full coverage, and can often be insufficient, and end users cannot verify.

• Static Analysis methods: It is deterministic, accurate, and catches data flow issues perfectly. But the learning curve is high and writing in a static analysis language or queries for the code is highly complex.

• 大模型作为裁判： 要求大模型自查或识别代码中的 Bug 是不可靠的，因为模型可能会产生幻觉或直接漏掉某些 Bug。即使是 Claude Opus 4.7、DeepSeek V3 和 Gemini Pro 等前沿模型，也经常会漏掉边缘情况、虚构修复方案，且无法理解复杂的数据流。

• 单元测试： 编写测试需要编写更多的代码来检查生成的代码。它通常无法检查 UI 到后端的集成，仅限于你指定的具体场景，且对非程序员来说门槛太高。大模型或许能写单元测试，但不能保证完全覆盖，往往是不充分的，且终端用户无法验证。

• 静态分析方法： 它是确定性的、准确的，能完美捕捉数据流问题。但其学习曲线陡峭，且使用静态分析语言或为代码编写查询语句非常复杂。

Introducing FlowCheck Through my research as a PhD student at Columbia’s DAP Lab, I developed FlowCheck. FlowCheck is a constraint language and static analysis pipeline that lets you easily specify how an app should behave directly from the interface, and checks it against the actual code, without ever having to read a line yourself.

介绍 FlowCheck 作为哥伦比亚大学 DAP 实验室的博士生，我在研究中开发了 FlowCheck。FlowCheck 是一个约束语言和静态分析流水线，让你能够直接从界面上轻松指定应用应有的行为，并将其与实际代码进行比对检查，而无需你自己阅读一行代码。

Step 1: Express your constraint Users provide FlowCheck with their web app’s path, which we open in a new tab. We display an overlay template of the form “When I take [action], these update: [component]” and users can select UI components (and a detected list of APIs and storage) by clicking directly on them, the same way they interact with their app.

第一步：表达你的约束 用户向 FlowCheck 提供其 Web 应用的路径，我们会在新标签页中打开它。我们显示一个类似“当我执行 [动作] 时，这些会更新：[组件]”的覆盖模板，用户可以通过直接点击 UI 组件（以及检测到的 API 和存储列表）来选择它们，就像他们平时与应用交互一样。

Step 2: Translation to our language We translate this template into a formal constraint in our language, using a grammar we define further in the paper. The overall format of our constraints looks like: P(event | condition) = [0, 1]. This can be read as, the probability of the event occurring (e.g a write happening to a component), given a condition (e.g. button being clicked), is equal to 1 (always) or 0 (never).

第二步：翻译成我们的语言 我们将此模板翻译成我们语言中的形式化约束，使用我们在论文中进一步定义的语法。我们约束的总体格式如下：P(event | condition) = [0, 1]。这可以解读为：在给定条件（例如点击按钮）下，事件发生（例如组件被写入）的概率等于 1（总是发生）或 0（从不发生）。

For our example above, we expect the total to be written to when the promo_input is applied, which can be written as P(w(total) | A(promo-input)) = 1. Expressing this using our language enables us to parse for the relevant information and compile it directly down into static analysis queries.

对于上面的例子，我们期望当应用 promo_input 时，total 会被写入，这可以写为 P(w(total) | A(promo-input)) = 1。使用我们的语言来表达这一点，使我们能够解析出相关信息，并将其直接编译为静态分析查询。

Step 3: Compilation to CodeQL Now, we have a formal constraint like P(write(e) | action(A)) = 1. FlowCheck parses this constraint and traverses its AST to extract key details (e.g which action was triggered, what type of event occurred (such as a write), and which specific target element must be modified). From this, we can determine what queries to run. We can think about it this way: our constraint simply means that when action A is taken, event E happens on all paths.

第三步：编译为 CodeQL 现在，我们有了一个像 P(write(e) | action(A)) = 1 这样的形式化约束。FlowCheck 解析该约束并遍历其抽象语法树（AST）以提取关键细节（例如触发了哪个动作、发生了什么类型的事件（如写入）、以及必须修改哪个特定的目标元素）。由此，我们可以确定要运行哪些查询。我们可以这样理解：我们的约束简单地意味着当执行动作 A 时，事件 E 在所有路径上都会发生。

Next, we use CodeQL. To provide some background, CodeQL is a static analysis engine which takes our app’s code and converts it into a queryable relational database. This allows us to run queries against the code. To catch silent failures, we focus mostly on data flow queries. CodeQL tracks data flow from a source (like a button click event) to a sink (like local storage or a database update). In FlowCheck, we maps the components mentioned in the constraint directly to sources and sinks, and use this to form queries that correspond to checks.

接下来，我们使用 CodeQL。作为背景介绍，CodeQL 是一个静态分析引擎，它将应用的代码转换为可查询的关系数据库。这允许我们针对代码运行查询。为了捕捉静默故障，我们主要关注数据流查询。CodeQL 跟踪从源（如按钮点击事件）到汇（如本地存储或数据库更新）的数据流。在 FlowCheck 中，我们将约束中提到的组件直接映射到源和汇，并利用这一点形成与检查相对应的查询。