I estimated 500 RPS from the System Design book and measured 24. Here's what broke.
本文为原文前 6,000 字符的节选翻译,完整内容请查看原文。
I estimated 500 RPS from the System Design book and measured 24. Here’s what broke.
There’s a chapter everyone who prepared for a system design interview has read: “Back-of-the-envelope estimation” in Alex Xu’s System Design Interview. Divide DAU by 86,400, double it for peak, multiply out the storage for a few years — and you get “1,160 writes per second, 11,600 reads, 36.5 TB”. I could do that arithmetic too. Then I asked myself a simple question: “and will Postgres actually take it?” — and realised I didn’t know. I knew how to compute the load. I had no idea what happens to the database when that load arrives. So instead of guessing, I measured.
每个准备过系统设计面试的人都读过 Alex Xu 所著《系统设计面试》中的“封底估算”一章。将日活跃用户数(DAU)除以 86,400,翻倍作为峰值,再乘以几年的存储需求——你就会得到“每秒 1,160 次写入,11,600 次读取,36.5 TB”的结果。我也会做这些算术。然后我问了自己一个简单的问题:“Postgres 真的能承受吗?”——我意识到我并不知道。我知道如何计算负载,但我完全不知道当负载到来时数据库会发生什么。所以,与其猜测,不如测量。
I took the system from chapter 8 of the same book — the URL shortener — built it in Docker on my laptop, worked out from the book what it should sustain, and pointed a load generator at it. Below: the estimate, the stand, the numbers, and what I learned from them. Spoiler: the estimate missed the measurement not by a few percent but by three orders of magnitude, and that was the most useful thing that happened in all the time I spent with the book.
我采用了书中第 8 章的系统——URL 缩短器——在我的笔记本电脑上用 Docker 构建了它,根据书中的内容计算出它应该能承受的负载,并用负载生成器对其进行了测试。以下是估算、测试环境、数据以及我从中获得的经验。剧透一下:估算结果与实际测量值相差了三个数量级,而非仅仅几个百分点,这成了我阅读这本书期间最有价值的收获。
The estimate, by the book. The URL shortener from chapter 8. Xu’s assumptions: 100 million new links per day, a 10:1 read-to-write ratio, 10 years of retention, about 100 bytes per record. Writes: 100,000,000 / 86,400 ≈ 1,160 per second. Reads: ×10 ≈ 11,600 per second. Records over 10 years: 100 M × 365 × 10 = 365 billion. Storage: 365 billion × 100 bytes = 36.5 TB.
按书中的估算:第 8 章的 URL 缩短器。Xu 的假设是:每天 1 亿个新链接,读写比为 10:1,保留 10 年,每条记录约 100 字节。写入:100,000,000 / 86,400 ≈ 每秒 1,160 次。读取:×10 ≈ 每秒 11,600 次。10 年内的记录数:1 亿 × 365 × 10 = 3,650 亿条。存储空间:3,650 亿 × 100 字节 = 36.5 TB。
But 11,600 reads per second is company scale, not laptop scale. I needed a target I could check at home. I picked 500 read requests per second — roughly 4 million DAU at one click per user per day, with a peak factor of 2. A perfectly ordinary mid-sized service. And I wrote down my expectation: at that RPS, p95 latency should be around 20 ms — a key lookup in a table of a million rows, what could possibly take long. This matters and I’ll insist on it: write the expectation down before you run anything. Otherwise, after the measurement, your brain will helpfully tell you “yeah, that’s about what I thought”.
但每秒 11,600 次读取是公司级的规模,而不是笔记本电脑能承受的。我需要一个可以在家里测试的目标。我选择了每秒 500 次读取请求——大约相当于 400 万 DAU,假设每个用户每天点击一次,峰值系数为 2。这是一个非常普通的中型服务。我写下了我的预期:在该 RPS 下,p95 延迟应该在 20 毫秒左右——在百万行级别的表中进行键查找,怎么可能耗时很长呢?这一点很重要,我坚持认为:在运行任何测试之前,请先写下你的预期。否则,测量结束后,你的大脑会很“贴心”地告诉你:“没错,这和我预想的差不多。”
The stand: Docker Compose — three long-running containers plus k6, which only comes up for the duration of a run: api — a Go service: POST /shorten takes a long URL and returns a short code (base62 of the id), GET /{code} looks it up and returns a 302. Database pool size: 10. postgres — one table, urls(id, code, long_url, created_at). redis — present but off (CACHE=off). It’ll matter later. k6 — the load generator, constant-arrival-rate scenario: exactly N requests per second regardless of whether the server keeps up.
测试环境:Docker Compose——三个长期运行的容器加上 k6(仅在运行期间启动):api——一个 Go 服务:POST /shorten 接收长 URL 并返回短代码(ID 的 base62 编码),GET /{code} 进行查找并返回 302。数据库连接池大小:10。postgres——一张表,urls(id, code, long_url, created_at)。redis——存在但已关闭(CACHE=off)。这在后面很重要。k6——负载生成器,采用恒定到达率场景:无论服务器是否跟得上,每秒精确发送 N 个请求。
I loaded a million rows with COPY — 11.7 seconds, 102 MB on disk. Picked 20 random codes out of them for k6 to hit with GETs. One deliberate detail I’ll state up front so it doesn’t feel like a trick: there is no index on the code column. Only the primary key on id. Why will become clear from the result.
我用 COPY 命令加载了 100 万行数据——耗时 11.7 秒,磁盘占用 102 MB。从中随机挑选了 20 个代码供 k6 进行 GET 请求测试。我先说明一个刻意的细节,以免让人觉得这是在耍花招:code 列上没有索引,只有 id 上的主键。原因将在结果中揭晓。
The measurement: Run: 500 requests per second, 60 seconds. target_rps | actual RPS | p95, ms | p99, ms | error rate. 500 | 24.17 | 22130.61 | 22808.33 | 0.00%. Read the second column again. I asked for 500 requests per second — the stand served 24. The rest didn’t fail; they queued: p95 is 22 seconds. Not milliseconds. Seconds. I expected 20 ms and got 1,100× that.
测量结果:运行:每秒 500 个请求,持续 60 秒。目标 RPS | 实际 RPS | p95 (毫秒) | p99 (毫秒) | 错误率。500 | 24.17 | 22130.61 | 22808.33 | 0.00%。再看一眼第二列。我要求每秒 500 个请求——测试环境只处理了 24 个。其余的并没有失败,而是排队了:p95 是 22 秒。不是毫秒,是秒。我预期是 20 毫秒,结果却是预期的 1,100 倍。
A second run a couple of hours later, with fewer neighbours on the laptop, gave 72 RPS and a p95 of 7.5 seconds. The estimate did not hold. Why — stages 2 and 3. A 3× spread between runs is itself a lesson about measuring on a laptop. But the order of magnitude is the same: asked for 500, got dozens, waited seconds. If this weren’t a lab but production, it would look like this: users click a link, the page spins for twenty seconds, half of them close the tab. Meanwhile the logs show zero errors. Error-rate monitoring is silent. That’s the second lesson: “0% errors” says nothing about whether the system is alive.
几小时后进行了第二次运行,由于笔记本电脑上运行的其他程序较少,得到了 72 RPS 和 7.5 秒的 p95。估算没有成立。原因在后续阶段。两次运行之间 3 倍的差异本身就是关于在笔记本电脑上进行测量的一课。但数量级是一样的:要求 500,得到几十,等待时间以秒计。如果这不是实验室环境而是生产环境,情况会是这样:用户点击链接,页面转圈二十秒,一半用户关闭了标签页。与此同时,日志显示零错误。错误率监控毫无反应。这是第二个教训:“0% 错误”并不能说明系统是否正常运行。
Where it broke: The arithmetic is simple, and it could have been done before the run if you knew where to look. SELECT long_url FROM urls WHERE code = $1 without an index on code is a sequential pass over the whole table: Postgres reads a million rows to return one.
哪里出了问题:算术很简单,如果你知道去哪里看,在运行之前就能算出来。在没有 code 索引的情况下执行 SELECT long_url FROM urls WHERE code = $1,是对整张表进行顺序扫描:Postgres 为了返回一行数据,读取了 100 万行。
Alone, on a warm cache, such a query takes 44 ms. But under load there are ten of them at once (pool size 10), each spawning two more parallel workers, all sharing the same cores: one query stretches to hundreds of milliseconds. Ten connections at 150–400 ms each is a ceiling of a few dozen requests per second. Everything above that waits for a free connection. At 500 incoming per second the queue grows by hundreds of requests every second, and after a minute p95 is queueing time, not query time.
在缓存预热的情况下,单次查询耗时 44 毫秒。但在负载下,同时有 10 个查询(连接池大小为 10),每个查询又产生两个并行工作进程,所有进程共享相同的核心:单个查询被拉长到数百毫秒。10 个连接,每个耗时 150–400 毫秒,每秒的处理上限仅为几十个请求。超过这个上限的所有请求都在等待空闲连接。当每秒有 500 个请求涌入时,队列每秒增加数百个请求,一分钟后,p95 变成了排队时间,而不是查询时间。