Do Not Hire an AI Test Agent From Its Demo

不要仅凭演示就雇佣 AI 测试代理

AI testing demos are almost unfairly persuasive. You describe a workflow in plain English. The agent opens a browser, finds the correct elements, completes the flow, and produces a passing test. Five minutes later, it feels as if test maintenance has been solved. The problem is that a demo shows the happy path for the agent, not the operating model for your team. AI 测试演示具有一种近乎不公平的说服力。你用简单的英语描述一个工作流，代理便会打开浏览器，找到正确的元素，完成流程，并生成一个通过的测试。五分钟后，你会感觉测试维护问题似乎已经解决了。但问题在于，演示展示的是代理的“理想路径”，而非你团队的实际操作模式。

The real evaluation begins when the application changes, the locator is ambiguous, the environment is partially broken, or the agent makes the wrong decision with high confidence. Evaluate the agent like a junior operator. I would not give a new employee broad production access simply because they completed one task successfully in an interview. I would first want to know: How they explain their decisions, What they do when information is missing, Whether they escalate uncertainty, How easily their work can be reviewed, Whether mistakes can be reversed. An AI agent deserves the same scrutiny. 真正的评估始于应用程序发生变更、定位器变得模糊、环境部分损坏，或者代理在高度自信的情况下做出错误决策之时。请像评估初级员工一样评估 AI 代理。我不会仅仅因为一名新员工在面试中成功完成了一项任务，就授予他广泛的生产环境访问权限。我首先想知道的是：他们如何解释自己的决策？当信息缺失时他们会怎么做？他们是否会反馈不确定性？他们的工作是否易于审查？错误是否可以撤销？AI 代理也应受到同样的审视。

This guide on evaluating AI test agents without losing browser-test debug visibility focuses on an essential requirement: automation should become easier to maintain, not harder to inspect. An agent that produces a passing result without showing how it arrived there creates a trust problem. 这份关于在不丢失浏览器测试调试可见性的前提下评估 AI 测试代理的指南，聚焦于一个核心要求：自动化应该变得更易于维护，而不是更难检查。如果一个代理在产出通过结果的同时，却无法展示其得出结论的过程，这就会产生信任问题。

Do not test it only on your cleanest workflow

不要只在最理想的工作流上进行测试

Vendor demos usually use predictable applications: Stable IDs, Simple forms, Obvious button labels, No iframes, No duplicate elements, No delayed events, No permission prompts, No uncertain outcomes. Your evaluation should do the opposite. Give the agent a workflow where two elements have similar labels. Change a piece of copy after the test has been generated. Remove an element. Slow down an API response. Cause the browser to crash. Return an unexpected validation message. 供应商的演示通常使用可预测的应用程序：稳定的 ID、简单的表单、明显的按钮标签、没有 iframe、没有重复元素、没有延迟事件、没有权限提示、没有不确定的结果。你的评估应该反其道而行之。给代理一个包含两个相似标签元素的工作流。在测试生成后修改一段文案。移除一个元素。减慢 API 响应速度。导致浏览器崩溃。返回一个意外的验证消息。

The article on evaluating browser-flow agents without trusting the demo offers a useful principle: measure recovery behavior, not just task completion. When the agent cannot confidently determine the right action, it should say so. A clarifying question is better than an impressive-looking mistake. 那篇关于在不盲信演示的前提下评估浏览器流程代理的文章提供了一个有用的原则：衡量恢复行为，而不仅仅是任务完成情况。当代理无法自信地确定正确操作时，它应该主动说明。一个澄清性的提问远比一个看起来很厉害的错误要好得多。

Measure more than pass rate

衡量的指标不仅是“通过率”

Pass rate is a poor standalone metric for AI-generated or AI-maintained tests. A test can pass because the product works. It can also pass because the agent selected the wrong element, skipped a failed assertion, modified the workflow, or healed the test into a different scenario. This guide to measuring AI test runs before trusting their pass rate suggests looking at the decisions behind the result. 对于 AI 生成或 AI 维护的测试而言，“通过率”是一个糟糕的单一指标。测试通过可能是因为产品运行正常，但也可能是因为代理选择了错误的元素、跳过了失败的断言、修改了工作流，或者将测试“自愈”成了另一个场景。这份在信任通过率之前衡量 AI 测试运行情况的指南建议：要关注结果背后的决策过程。

Useful metrics include: Percentage of runs requiring agent intervention, Number of locators changed automatically, Frequency of ambiguous element matches, Human acceptance rate for suggested changes, False repair rate, Time saved during investigation, Number of silently changed test steps. I would also measure how frequently the agent requests clarification. A low number is not necessarily good. It may mean the system is guessing. 有用的指标包括：需要人工干预的运行百分比、自动更改的定位器数量、模糊元素匹配的频率、人工对建议更改的接受率、错误修复率、调查过程中节省的时间、静默更改的测试步骤数量。我还会衡量代理请求澄清的频率。数值低并不一定是好事，这可能意味着系统正在盲目猜测。

Procurement questions are product questions

采购问题即产品问题

AI testing purchases often begin as feature comparisons and end as security reviews. That transition should happen earlier. The AI testing procurement scorecard covers questions that become important once the proof of concept is over: What application data is sent to the model? Are screenshots or page sources retained? Can sensitive values be masked? Which employees can enable AI features? Are model actions logged? Can generated changes require approval? Where is the data processed? What happens when an AI feature is disabled? An AI feature is not isolated from the rest of the product. Its governance model becomes part of your testing infrastructure. AI 测试采购通常始于功能对比，终于安全审查。这种转变应该提前发生。AI 测试采购评分表涵盖了在概念验证（PoC）结束后变得至关重要的几个问题：发送给模型的是什么应用数据？是否保留了截图或页面源码？敏感值是否可以脱敏？哪些员工可以启用 AI 功能？模型操作是否被记录？生成的更改是否需要审批？数据在哪里处理？当 AI 功能被禁用时会发生什么？AI 功能并非与产品其他部分隔绝，其治理模型将成为你测试基础设施的一部分。

Test the product on genuinely uncertain UI

在真正不确定的 UI 上测试产品

AI-powered form assistants and guided checkout flows are good evaluation targets because they combine normal browser interaction with probabilistic behavior. The assistant may change its wording while still giving the correct answer. It may suggest different products based on small changes in input. A rigid text assertion can fail even when the experience is acceptable. AI 驱动的表单助手和引导式结账流程是很好的评估目标，因为它们将常规的浏览器交互与概率性行为结合在了一起。助手可能会在给出正确答案的同时改变措辞。它可能会根据输入的微小变化建议不同的产品。即使体验是可接受的，僵化的文本断言也可能会失败。

This review of Endtest for AI-powered form assistants and guided checkout flows illustrates where AI Assertions can be more useful than exact string comparisons. For example, instead of asserting that the assistant displays one precise sentence, you may want to validate that: The response answers the customer’s question. The recommendation respects the selected constraints. The assistant does not claim the payment succeeded before confirmation. The checkout flow still reaches the expected outcome. The assertion needs flexibility, but the workflow still needs objective boundaries. 这篇关于 Endtest 在 AI 驱动表单助手和引导式结账流程中的评测，展示了 AI 断言在何种情况下比精确字符串比较更有用。例如，与其断言助手显示了某一句精确的句子，你可能更想验证：回复是否回答了客户的问题？推荐是否尊重了所选的约束条件？助手是否在确认前没有声称支付成功？结账流程是否依然达到了预期的结果？断言需要灵活性，但工作流仍然需要客观的边界。

Account for the people who will own the tests

考虑测试的实际负责人

A technically impressive agent can still be the wrong product for a team. This comparison of Endtest and Playwright for QA teams without a dedicated SDET highlights a factor that feature matrices often ignore: who will operate the system after implementation? A developer-led team may prefer code, custom fixtures, and complete framework control. A QA-led team may benefit more from readable steps, managed execution, built-in reporting, and an interface where tests can be updated without editing a framework. Neither approach is universally correct. The mistake is choosing an architecture for the team you hope to hire rather than the team you currently have. 一个技术上令人印象深刻的代理，对某个团队来说可能依然是错误的产品。这份针对没有专门 SDET（软件测试开发工程师）的 QA 团队对 Endtest 和 Playwright 的对比，强调了一个功能矩阵经常忽略的因素：实施后谁来操作该系统？开发主导的团队可能更喜欢代码、自定义 fixture 和完整的框架控制。QA 主导的团队可能更受益于可读的步骤、托管执行、内置报告以及无需编辑框架即可更新测试的界面。这两种方法都没有绝对的对错。错误在于，你选择的架构是为你“希望雇佣”的团队准备的，而不是为你“当前拥有”的团队准备的。

AI should compress work, not hide it

AI 应该压缩工作，而不是隐藏工作

At Endtest, features such as the AI Test Creation Agent are intended to accelerate test creation while keeping the resulting test understandable and editable. That distinction matters. The most useful AI systems do not replace the test with a mysterious generated artifact. They compress repetitive work while preserving review, ownership, and control. Before adopting an AI test agent, ask one final question: When it makes a mistake—and eventually it will—how quickly will my team understand what happened? That answer matters far more than the demo. 在 Endtest，诸如“AI 测试创建代理”之类的功能旨在加速测试创建，同时保持生成的测试可理解且可编辑。这种区别至关重要。最有用的 AI 系统不会用神秘的生成产物来取代测试，它们是在保留审查、所有权和控制权的同时，压缩重复性工作。在采用 AI 测试代理之前，请问最后一个问题：当它犯错时（它终究会犯错），我的团队能多快理解发生了什么？这个答案比演示重要得多。