Your Test Suite Isn't Slow. It's Accumulating Decisions
Your Test Suite Isn’t Slow. It’s Accumulating Decisions
你的测试套件并不慢,它只是在不断累积决策
Most browser test suites do not collapse in one dramatic moment. They get a little slower on Monday. A little noisier on Tuesday. Someone adds a retry on Wednesday. By Friday, the pipeline takes 28 minutes, three tests fail for reasons nobody can reproduce, and the team has quietly learned to merge anyway. That is how reliability debt works. It rarely looks urgent while you are creating it. 大多数浏览器测试套件并不会在某个戏剧性的时刻瞬间崩溃。它们可能在周一变慢一点,周二变得更不稳定一点,周三有人加了一个重试机制。到了周五,流水线运行需要 28 分钟,三个测试因为没人能复现的原因失败了,而团队已经默默学会了直接合并代码。这就是可靠性债务的运作方式。当你制造这些债务时,它看起来很少是紧急的。
The common explanation is that browser tests are inherently flaky. That explanation is convenient because it makes the problem feel unavoidable. But many of the failures we call “flakiness” are really the accumulated result of dozens of small technical decisions: an animation that behaves differently in CI; a feature flag that changes the DOM after the test starts; a performance threshold that ignores natural variance; more parallel workers than the environment can actually support; an assertion that checks implementation details instead of user-visible outcomes. 常见的解释是浏览器测试天生就不稳定(flaky)。这种解释很方便,因为它让问题看起来不可避免。但许多我们称之为“不稳定”的失败,实际上是数十个微小技术决策累积的结果:在 CI 环境中表现不同的动画;在测试开始后改变 DOM 的功能开关(feature flag);忽略自然波动的性能阈值;超过环境实际承载能力的并行工作进程;以及检查实现细节而非用户可见结果的断言。
The test suite is not betraying you. It is reporting the architecture you gave it. 测试套件并没有背叛你,它只是在如实反映你赋予它的架构。
The environment is part of the product
环境是产品的一部分
A test that passes locally and fails in CI is often treated as a tooling problem. Sometimes it is. More often, the two environments are not equivalent. One subtle example is CSS motion. A developer machine may use normal motion preferences while a CI browser reports reduced motion, or vice versa. That can change transition duration, animation timing, element visibility, and even which branch of a component renders. The result is a test that appears to fail randomly even though it is responding consistently to different inputs. 一个在本地通过但在 CI 中失败的测试,通常被视为工具问题。有时确实如此,但更多时候,这两个环境并不等同。一个微妙的例子是 CSS 动画偏好。开发者的机器可能使用正常的动画设置,而 CI 浏览器可能报告为“减少动画”,反之亦然。这会改变过渡持续时间、动画时序、元素可见性,甚至组件渲染的分支。结果就是测试看起来是随机失败的,尽管它实际上是在对不同的输入做出一致的响应。
This article on why browser tests fail when CSS motion preferences differ between local and CI environments is a good reminder that browser configuration is test data. The same principle applies to locale, timezone, colour scheme, viewport size, available fonts, GPU behaviour, network conditions, and feature flags. Teams often spend hours debugging the final assertion when the real difference was established before the first line of the test ran. A useful rule is this: If an environment setting can change the user experience, make it explicit in the test configuration. Do not depend on whatever default the laptop, container, or hosted runner happens to provide. 这篇文章探讨了当本地和 CI 环境的 CSS 动画偏好不同时,浏览器测试为何会失败,这提醒我们:浏览器配置本身就是测试数据。同样的原则也适用于区域设置、时区、配色方案、视口大小、可用字体、GPU 行为、网络状况和功能开关。团队经常花费数小时调试最后的断言,而真正的差异在测试运行的第一行代码之前就已经确定了。一个有用的规则是:如果某个环境设置会改变用户体验,请在测试配置中明确指定它。不要依赖笔记本电脑、容器或托管运行器提供的任何默认值。
Feature flags create multiple applications
功能开关创造了多个应用程序
A feature flag is not just a Boolean variable. It creates another version of your product. Five independent flags can theoretically create 32 combinations. Most teams do not test all of them, nor should they. But many teams also fail to define which combinations matter. That is where browser tests become confusing. The same test name may execute against different UI structures depending on rollout state, account assignment, cached configuration, or timing. 功能开关不仅仅是一个布尔变量,它创造了你产品的另一个版本。五个独立的开关理论上可以产生 32 种组合。大多数团队不会测试所有组合,也不应该这样做。但许多团队也没有定义哪些组合是重要的。这就是浏览器测试变得令人困惑的地方。同一个测试名称可能会根据发布状态、账户分配、缓存配置或时机,在不同的 UI 结构上执行。
The practical guide on debugging frontend tests that fail after feature flag changes highlights the first thing to verify: what state did the application actually render? That question sounds obvious. Yet many test reports preserve screenshots and logs without preserving the active flag set. When a failure is tied to gradual rollout logic, capture at least: the evaluated flag values; the user or account segment; the application version; the relevant API response; the DOM or screenshot at the moment the branch appeared. Without that context, you are not debugging a test. You are reconstructing a missing environment. 关于调试功能开关变更后失败的前端测试的实用指南强调了第一件要核实的事:应用程序实际渲染的状态是什么?这个问题听起来很显而易见。然而,许多测试报告保留了截图和日志,却没保留当时激活的开关集合。当失败与灰度发布逻辑相关时,至少要捕获:评估后的开关值、用户或账户细分、应用程序版本、相关的 API 响应,以及分支出现那一刻的 DOM 或截图。没有这些上下文,你就不在调试测试,而是在重建一个缺失的环境。
Modern UI patterns require different assertions
现代 UI 模式需要不同的断言方式
React Server Actions and optimistic interfaces make applications feel faster by showing the expected result before the server confirms it. That is good product design. It also creates several states that browser tests can accidentally confuse: the original state; the optimistic state; the confirmed state; the rollback state after an error. A test that sees the optimistic update and immediately passes may miss a server failure. A test that waits only for a network request may ignore a rollback bug. A test that asserts every transitional DOM detail becomes brittle whenever the implementation changes. React Server Actions 和乐观 UI 界面通过在服务器确认之前显示预期结果,让应用程序感觉更快。这是很好的产品设计,但也创造了浏览器测试容易混淆的几种状态:原始状态、乐观状态、确认状态以及错误后的回滚状态。一个看到乐观更新就立即通过的测试可能会漏掉服务器错误;一个只等待网络请求的测试可能会忽略回滚 Bug;一个断言每一个过渡性 DOM 细节的测试,在实现发生任何变化时都会变得脆弱。
The article on testing React Server Actions and optimistic UI offers a more durable approach: assert the state transitions that matter to the user, not every intermediate implementation detail. For a “save” action, that might mean: the user sees the immediate optimistic update; the server request succeeds; the state remains correct after a reload; an error produces a clear rollback or recovery path. This is a broader lesson. Browser automation should validate product promises. The closer your assertions are to internal mechanics, the more maintenance you purchase. 关于测试 React Server Actions 和乐观 UI 的文章提供了一种更持久的方法:断言对用户重要的状态转换,而不是每一个中间实现细节。对于“保存”操作,这意味着:用户看到即时的乐观更新;服务器请求成功;重新加载后状态保持正确;错误产生清晰的回滚或恢复路径。这是一个更广泛的教训:浏览器自动化应该验证产品承诺。你的断言越接近内部机制,你所购买的维护成本就越高。
Parallelism has a ceiling
并行化有其上限
When a suite becomes slow, the reflex is to add more workers. That works—until it does not. Parallel test execution competes for CPU, memory, browser processes, network bandwidth, database connections, test accounts, and shared environments. Once one of those resources saturates, adding workers can make the suite slower rather than faster. This framework for understanding why CI test suites get slower when parallelism increases is worth using before paying for larger runners or increasing concurrency again. 当测试套件变慢时,本能反应是增加工作进程。这在一定程度上有效,直到它失效为止。并行测试执行会竞争 CPU、内存、浏览器进程、网络带宽、数据库连接、测试账户和共享环境。一旦其中任何一个资源饱和,增加工作进程反而会让套件变慢。在为更大的运行器付费或再次增加并发之前,值得参考这个关于“为什么并行度增加时 CI 测试套件反而变慢”的框架。
The important metric is not the number of workers. It is throughput. Run a simple experiment: execute the same representative group with 2 workers; repeat with 4, 8, and 16; record total duration, failure rate, CPU, memory, and external-service latency; stop increasing concurrency when throughput stops improving reliably. Most teams have an efficient range, not an efficient maximum. And beware of shared state. Two tests using the same account, inbox, cart, project, or database record are not truly independent. Parallelism merely makes the collision occur sooner. 重要的指标不是工作进程的数量,而是吞吐量。做一个简单的实验:用 2 个进程执行同一组代表性测试;重复 4、8 和 16 个进程;记录总时长、失败率、CPU、内存和外部服务延迟;当吞吐量不再稳定提升时,停止增加并发。大多数团队都有一个高效区间,而不是一个高效最大值。此外,要警惕共享状态。两个使用相同账户、收件箱、购物车、项目或数据库记录的测试并不是真正独立的。并行化只会让冲突发生得更快。
Dynamic interfaces punish vague synchronization
动态界面会惩罚模糊的同步逻辑
Tables, filters, and infinite scrolling are common sources of false confidence because the first visible state looks complete before the application is finished. A filter click can trigger debouncing, mu… 表格、过滤器和无限滚动是虚假信心的常见来源,因为在应用程序完成加载之前,第一个可见状态看起来就已经完整了。点击过滤器可能会触发防抖(debouncing)、mu…