Testing the claim: a degraded-link matrix as a required CI gate
Testing the claim: a degraded-link matrix as a required CI gate
验证声明：将降级链路矩阵作为 CI 的必要门禁
This is a writeup of building a required CI gate for degraded-network behavior. The system under test is a robotics fleet substrate, but the finding applies to anyone shaping networks in CI. Ganglion exists to reach robots on networks nobody controls. Warehouse Wi-Fi, carrier CGNAT, a hospital VLAN, a customer firewall that was configured once in 2019 and has not been touched since. 这是一篇关于构建针对网络降级行为的 CI（持续集成）必要门禁的总结。被测系统是一个机器人车队底层架构，但其结论适用于任何在 CI 中进行网络整形（network shaping）的场景。Ganglion 的存在是为了连接那些处于不可控网络环境下的机器人，例如仓库 Wi-Fi、运营商 CGNAT、医院 VLAN，或是 2019 年配置后就再未动过的客户防火墙。
Until this week that claim was a sentence on a website. CI ran on clean loopback, everything was green, and the failure modes that actually matter in the field were the exact ones the test suite could never produce. That is now a required gate. Every push to main runs the full deploy, invoke and verify round trip over the relay against five shaped network profiles, and all five have to pass before anything merges. 直到本周，这些说法还仅仅是网站上的一行文字。CI 在干净的本地回环（loopback）上运行，一切显示正常，但现场真正关键的故障模式，恰恰是测试套件永远无法复现的。现在，这已成为一道必要门禁。每次推送到主分支（main）时，都会通过中继器运行完整的部署、调用和验证往返测试，并针对五种网络整形配置进行测试，所有五项必须全部通过才能合并代码。
I build Ganglion, so treat the enthusiasm accordingly. The part worth your time is not that it went green. It is what I got wrong on the way there. The five profiles: 我开发了 Ganglion，所以请带着审视的眼光看待我的热情。真正值得你关注的不是测试通过了，而是我在实现过程中犯的错误。这五种配置如下：
- clean: baseline, no shaping. If this one fails, something else is broken.
- lossy: packet loss with light reordering.
- high-latency: 250ms round trip.
- asymmetric: plentiful downlink, starved uplink. This is the one nobody tests and the one teleop actually dies on, because control acknowledgements go the starved direction.
- nat-relay: endpoints with no route to each other at all, forcing hole punching to fail and relay fallback to carry the session.
- clean（干净）：基准测试，无整形。如果这一项失败，说明其他地方坏了。
- lossy（丢包）：伴随轻微乱序的丢包。
- high-latency（高延迟）：250ms 往返延迟。
- asymmetric（非对称）：下行充足，上行受限。这是没人测试但远程操作（teleop）最容易挂掉的场景，因为控制确认信息需要通过受限的上行链路。
- nat-relay（NAT 中继）：端点之间完全没有路由，强制打洞失败并回退到中继模式来承载会话。
The last two are the ones I care about. Loss and latency are what people imagine a bad network is. Asymmetry and no-direct-route are what a bad network usually is. 最后两种是我最关心的。丢包和延迟是人们想象中“糟糕网络”的样子，而实际上，非对称和无直接路由才是糟糕网络的常态。
What I got wrong
我犯的错误
The original design assumed you can pin netem’s seed and get a repeatable lossy run. Two profiles: a pinned-seed one that gates the build, and a nastier randomized one that runs nightly and is allowed to fail. You cannot pin netem’s seed. Its loss and jitter draw from the kernel RNG and there is no seed parameter to set. A “deterministic lossy netem profile” is not a thing that exists. 最初的设计假设可以固定 netem 的种子（seed）以获得可重复的丢包运行。我设计了两个配置：一个固定种子的配置作为构建门禁，另一个更恶劣的随机配置作为夜间运行且允许失败的任务。但你无法固定 netem 的种子。它的丢包和抖动源自内核随机数生成器（RNG），且没有可设置的种子参数。“确定性的丢包 netem 配置”根本不存在。
This matters more than a wrong detail usually would, because the whole design rested on it. A gate that fails randomly is worse than no gate at all. It does not catch regressions, it teaches everyone to re-run the job until it passes, and after a month nobody reads red as meaning anything. 这比一般的细节错误更严重，因为整个设计都建立在它之上。一个随机失败的门禁比没有门禁更糟糕。它无法捕获回归问题，反而教会大家不断重试直到通过，一个月后，没人会再把“红色（失败）”当回事。
What actually reproduces
真正可复现的方案
The fix was to build the gate out of only the mechanisms that reproduce exactly, and to be honest that this is a smaller set than netem advertises: 修复方案是仅使用能精确复现的机制来构建门禁，并坦诚这比 netem 宣传的范围要小：
- Fixed netem delay: Constant delay, zero jitter. Deterministic.
- tbf rate caps: Bandwidth ceilings are a token bucket, not a distribution. Deterministic.
- iptables -m statistic —mode nth for loss: This drops precisely every Nth packet rather than N percent on average. Every 33rd packet is roughly three percent loss, and critically it is the same three percent every run.
- Route blocking for the nat-relay case: Either a route exists or it does not.
- 固定 netem 延迟：恒定延迟，零抖动。确定性。
- tbf 速率限制：带宽上限是令牌桶算法，而非概率分布。确定性。
- iptables -m statistic —mode nth 实现丢包：这会精确丢弃每第 N 个包，而不是平均丢弃 N%。每 33 个包丢一个大约是 3% 的丢包率，关键在于每次运行丢的都是那同一个 3%。
- NAT 中继的路由阻塞：路由要么存在，要么不存在。
Randomized netem still runs. It just runs nightly, in a separate non-blocking job, and a failure opens an issue instead of stopping a merge. The nightly job generates its parameters deterministically from a recorded seed, so a failing run can be replayed. Worth being precise about what that buys, because it is easy to oversell: replaying reproduces the impairment distribution, not the packet-level draw. You get the same shape of bad network, not the same individual dropped packets. 随机 netem 依然保留。它在夜间作为独立的非阻塞任务运行，失败时只会开启一个 Issue 而不会阻止合并。夜间任务通过记录的种子确定性地生成参数，因此失败的运行可以被重放。需要明确这带来的价值，因为很容易夸大其词：重放复现的是损伤分布，而不是数据包级别的抽取。你得到的是同样形态的糟糕网络，而不是完全相同的丢包序列。
Riding the harness that already existed
复用现有的测试框架
The obvious implementation is a fresh veth pair or a pair of network namespaces built specifically for shaping. I did not do that, and I think the reason generalizes. There was already an end-to-end dispatch harness running the real deploy, invoke and verify round trip through the relay. Building a second rig would have given me a network test that did not exercise the actual product path, which is the failure mode where CI is green and the thing still breaks. 最显而易见的实现方式是创建一对新的 veth 或专门用于整形的网络命名空间。我没有这样做，我认为这个理由具有普适性。现有的端到端调度框架已经能运行真实的部署、调用和验证往返测试。构建第二套环境会产生一个无法覆盖实际产品路径的网络测试，这正是 CI 显示正常但产品依然崩溃的故障模式。
So the matrix rides the existing harness. Shaping is applied inside the robot and operator containers before the agent starts. netem inside Docker was already proven green in CI by an existing mobile-CGNAT scenario, so the risky part was already de-risked. One rig, one new axis. 因此，矩阵测试直接复用现有的框架。整形配置在代理启动前应用于机器人和操作员容器内部。Docker 内部的 netem 在现有的移动 CGNAT 场景中已被证明在 CI 中是稳定的，所以风险部分已经规避。一套环境，增加一个维度。
Record what you shaped
记录你的整形配置
Every run writes a JSON artifact: mode, seed, the exact shaping commands issued, duration, and result. This came out of a conversation on ROS Discourse where someone building replay tooling for robot fleets made the point better than I would have. Injected faults that are not recorded produce failures you can see and cannot get back. The fault has to be part of the run’s record, or a genuine bug becomes a flaky ghost and gets closed as unreproducible. Shape at the qdisc, record what you shaped. The two compose. 每次运行都会写入一个 JSON 工件：模式、种子、执行的具体整形命令、持续时间和结果。这源于 ROS 论坛上的一次讨论，有人在构建机器人车队重放工具时提出了一个比我更好的观点：未被记录的注入故障会产生你可见但无法追溯的失败。故障必须成为运行记录的一部分，否则真正的 Bug 就会变成难以捉摸的幽灵，最终被以“无法复现”为由关闭。在 qdisc 层整形，记录你的整形配置，两者缺一不可。
Keeping CI cheap
保持 CI 低成本
A five-profile matrix is an easy way to quadruple your CI bill. What kept it small: 五种配置的矩阵很容易让你的 CI 账单翻四倍。以下是控制成本的方法：
- One required-gate job, not five parallel ones: The profiles run sequentially inside it.
- One build, reused across all five: Compiling five times to test networking is pure waste.
- Main-push only: Not every push to every branch.
- Early exit on docs-only pushes.
- 一个必要门禁任务，而非五个并行任务：配置在任务内顺序执行。
- 一次构建，五次复用：为了测试网络而编译五次纯属浪费。
- 仅限主分支推送：不是每个分支的每次推送都测。
- 文档类推送提前退出。
That lands around 15 to 25 minutes per push. The nightly chaos run is scheduled separately and blocks nothing. 这使得每次推送的耗时在 15 到 25 分钟左右。夜间的混沌测试单独调度，不会阻塞任何流程。
The de-flaking pass, which is the part everyone skips
去抖动（De-flaking）环节，这是每个人都会跳过的部分
First full run was not clean, and the failure was not in the shaping. The nat-relay profile had a DNS race: containers with no route between them were resolving each other inconsistently at startup, and the test failed in a way that looked like a shaping bug. Fixed with static compose IPs. The gate profiles also retry once and attach compose logs on failure. That is a deliberate concession. A gate that is right in principle and flaky in practice gets disabled within two weeks, and then you have neither. 第一次完整运行并不顺利，失败原因不在整形上。NAT 中继配置存在 DNS 竞争问题：容器在启动时无法解析彼此，导致测试失败，看起来像整形 Bug。通过静态 Compose IP 解决了这个问题。门禁配置还允许重试一次，并在失败时附加 Compose 日志。这是一个刻意的妥协。一个原则上正确但实践中不稳定的门禁，两周内就会被禁用，到时候你就什么都没有了。
Where it landed
最终结果
Five of five deterministic profiles passing on main. Clean, lossy, high-latency, asymmetric, nat-relay. The honest framing is not… 主分支上五种确定性配置全部通过：Clean、Lossy、High-latency、Asymmetric、NAT-relay。诚实的表述是……