The mean means nothing

平均值毫无意义

I was recently trying to validate some performance improvements related to lld at $DAYJOB and it was a little frustrating to see the improvements in our benchmarks but not in the live-production dashboards. Having come from a background working on web-services, I was used to looking at individual time-series dashboards, sometimes over a few percentiles, and I was expecing to see some noticeable change but the data seemed too noisy to make any conclusions. 最近，我在工作中试图验证与 lld 相关的性能改进，但令人沮丧的是，基准测试显示有提升，但在实时生产环境的仪表板上却看不出来。由于我有从事 Web 服务开发的背景，我习惯于查看单独的时间序列仪表板，有时会关注几个百分位点，我本以为会看到明显的变化，但数据似乎太嘈杂，无法得出任何结论。

Turns out a colleague had also faced similar issues when trying to evaluate build-speed improvements. There are lots of variables that can affect the build: cold-cache, incremental, local, remote, etc. and the build times can vary wildly depending on the state of the system and the workload. She ended up leveraging a cummulative distribution function (CDF) to visualize the data and it was a revelation to me. 后来发现，一位同事在评估构建速度改进时也遇到了类似的问题。有很多变量会影响构建：冷缓存、增量构建、本地构建、远程构建等，而且构建时间会根据系统状态和工作负载的不同而产生巨大差异。她最终利用累积分布函数 (CDF) 来可视化数据，这对我来说是一个启发。

This led me to explore a few other different ways to visualize data, in addition to the CDF, and how a single image or statistic is often not enough to tell the whole story. This post will walk through a single synthetic dataset and show how different visualizations can tell different stories about the same data. The goal is to convince you to look at your data and not just summarize it with a single number. 这促使我除了 CDF 之外，还探索了其他几种可视化数据的方法，并意识到单一的图像或统计数据往往不足以说明全部情况。本文将通过一个合成数据集，展示不同的可视化方式如何讲述关于同一数据的不同故事。我的目标是说服你深入观察数据，而不仅仅是用一个数字来概括它。

Everything below comes from one synthetic dataset with a fixed seed. The full script can be found in this gist. It is a single file with a nix-shell shebang, so you can reproduce every figure exactly as long as you are using nix. Note I leveraged AI to help generate the data and charts in this post for the story. If that bugs you, sorry. 🤷 以下所有内容均来自一个具有固定种子的合成数据集。完整的脚本可以在这个 gist 中找到。它是一个带有 nix-shell shebang 的单文件，只要你使用 nix，就可以完全复现每一个图表。请注意，我利用 AI 帮助生成了本文中的数据和图表以辅助叙述。如果这让你感到困扰，抱歉。🤷

The rollout that “made it worse”

导致“情况变糟”的发布

Here is the setup: we operate a typical web-service and we rolled out a new caching tier over a week, hoping to cut request latency. The change is fully deployed, and the latency dashboard that plots the mean looks like this: Mean latency went up, from 112 ms to 122 ms. ☹️ A SEV is cut, we revert the change and write the postmortem. Right? 🤔 场景如下：我们运营着一个典型的 Web 服务，并在过去一周内推出了一个新的缓存层，希望以此降低请求延迟。变更已完全部署，绘制平均值的延迟仪表板显示：平均延迟从 112 毫秒上升到了 122 毫秒。☹️ 于是触发了严重事故 (SEV)，我们回滚了变更并撰写了事后总结。是这样吗？🤔

One number, four stories

一个数字，四个故事

It’s often good practice especially for web-services to look at various percentiles, especially the tail end of the distribution like the p95 and p99. 对于 Web 服务而言，查看各种百分位点（尤其是 p95 和 p99 等分布尾部）通常是一种良好的实践。

statistic	before	after	change
mean	112 ms	122 ms	+9%
p50 (median)	99 ms	54 ms	−46%
p95	224 ms	454 ms	+103%
p99	309 ms	678 ms	+119%

Now we have a problem, and the problem is that everyone is right. The mean says the change is a mild regression. The median (p50), says the change is a big-win, the typical request got nearly twice as fast. The p99 says it’s a SEV, the worst requests more than doubling. The mean and the median, computed from the very same numbers, point in opposite directions. Engineers are often taught to be data-oriented, but often it’s easy to cherry-pick the statistic that supports your argument. 现在我们遇到了一个问题，问题在于每个人说的都对。平均值显示这次变更是轻微的倒退。中位数 (p50) 显示这是一个巨大的胜利，典型请求的速度快了近一倍。p99 则显示这是一次严重事故，最慢的请求耗时增加了一倍多。由同一组数字计算出的平均值和中位数，却指向了截然相反的方向。工程师通常被教导要以数据为导向，但往往很容易挑选出支持自己论点的统计数据。

Look at the shape

观察形状

The next basic thing you can do with a distribution is plot its shape. Here are the two latency distributions, before and after, as densities: There it is. 🤓☝️ The “before” is one tidy hump. The “after” is two humps. This already explains the earlier contradiction but it’s a bit tricky to visualize correctly. 对于分布，你可以做的下一件基本事情是绘制其形状。这是变更前后两个延迟分布的密度图：看，就是这样。🤓☝️ “变更前”是一个整齐的驼峰。“变更后”则是两个驼峰。这已经解释了之前的矛盾，但要正确地可视化它有点棘手。

The shape depends on a smoothing parameter we chose, the two fills muddy each other where they overlap, and it’s genuinely hard to read a percentile off it. I can see there are two populations; I can’t easily see where the median went. 形状取决于我们选择的平滑参数，两个填充区域在重叠处会相互混淆，而且确实很难从中读出百分位点。我能看出有两个群体，但我无法轻易看出中位数在哪里。

The best chart you’re not using

你没在用但最好的图表

The cumulative distribution function (CDF) answers one question for every percentile at once: what fraction of requests came in at or below x milliseconds? CDFs are an extremely easy way to visualize multiple percentiles in a single chart. Depending on the curve, we can understand how the request latency is distributed across the entire population. 累积分布函数 (CDF) 可以同时回答每个百分位点的一个问题：有多少比例的请求是在 x 毫秒或更短时间内完成的？CDF 是一种在单张图表中可视化多个百分位点的极其简单的方法。通过曲线，我们可以了解请求延迟在整个群体中是如何分布的。

I found it incredibly useful to then plot the “before” and “after” CDFs on the same chart to see how they compare. You can then visualize the shifts at various percentiles and understand how the change affected the entire population. In our story, the “after” curve has shifted left for request latencies below 140ms. That means more requests are finishing faster than before. The “after” curve is higher than the “before” curve to the right of 140ms which means more requests are finishing slower than before. The two curves cross at ~140ms which is the tipping point where the change goes from being a win to a loss. 我发现将“变更前”和“变更后”的 CDF 绘制在同一张图表上进行比较非常有用。你可以直观地看到各个百分位点的偏移，并了解这次变更如何影响了整个群体。在我们的故事中，“变更后”的曲线在 140 毫秒以下的请求延迟处向左移动了。这意味着更多的请求比以前完成得更快。而在 140 毫秒右侧，“变更后”的曲线高于“变更前”，这意味着更多的请求比以前完成得更慢。两条曲线在约 140 毫秒处交叉，这是变更从“胜利”转为“失败”的临界点。

Tip: Two CDFs that cross are the unmistakable signature of a change that no single percentile can summarize, because the sign of the effect depends on which percentile you ask. 提示： 两条交叉的 CDF 曲线是某种变更的明确标志，没有任何单一的百分位点可以概括它，因为效果的正负取决于你关注的是哪个百分位点。

Who won, and by how much

谁赢了，赢了多少

The CDF tells us that the effect changes sign: faster or slower. The obvious next question is by how much, at each point in the distribution. We can plot for every percentile p, the after-latency minus the before-latency, known as the shift function. Below the zero line the change is faster; above it, slower. We can visualize the magnitude of the change at each percentile. CDF 告诉我们效果的正负发生了变化：变快或变慢。接下来的显而易见的问题是，在分布的每个点上，变化幅度是多少。我们可以为每个百分位点 p 绘制“变更后延迟减去变更前延迟”的图，这被称为偏移函数。零线以下表示变快，以上表示变慢。我们可以直观地看到每个百分位点上的变化幅度。

The regression was there all along

倒退一直都在

So far we have looked at two frozen snapshots, before and after. Rollouts though are often not instant. In this story, we rolled out the new caching tier over a week, ramping from 0% to 100% of traffic. What did each day look like? Stack one distribution per day and you get a ridgeline: 到目前为止，我们只看了变更前后的两个静态快照。然而，发布往往不是瞬间完成的。在这个故事中，我们在一周内推出了新的缓存层，流量从 0% 逐步增加到 100%。每一天的情况如何？将每天的分布堆叠起来，你就会得到一个山脊图 (ridgeline)：

We can now visualize the regression emerging over time. We can see the main peak (the fast requests) sliding left as the rollout progresses, and a second peak (the slow requests) emerging on the right. The median is dropping, but the slow requests are quietly growing in number and latency. The x-axis here is logarithmic. Latency is roughly lognormal, and on a linear axis the fast peak is a tall spike next to an invisible smear; the log axis is what lets both humps read as humps. 现在我们可以直观地看到倒退随时间显现的过程。我们可以看到随着发布的推进，主峰（快速请求）向左滑动，而第二个峰（慢速请求）在右侧出现。中位数在下降，但慢速请求的数量和延迟却在悄悄增加。这里的 x 轴是对数轴。延迟大致呈对数正态分布，在线性轴上，快速峰值是一个高尖刺，旁边是一片不可见的模糊区域；对数轴让两个驼峰都能清晰地呈现出来。

We can do something similar, squeezed into a single grid, as a heatmap. One column per day, colour for how much traffic lands at each latency: You can sort-of make out a new population emerging faintly. 我们还可以做类似的事情，将其压缩到一个网格中，作为热力图。每一列代表一天，颜色代表落在每个延迟区间的流量大小：你可以隐约看出一个新的群体正在浮现。