Curl Performance

tldr: The live version is here: https://curl.se/perf/ 简述： 实时版本请见：https://curl.se/perf/

How fast is “fast” and is it good enough? Does it run as fast now as it did before or was there a regression? What exactly needs to be fast? How fast is it? These are questions that many projects and products face, and in curl we are no different. Yet, performance testing and comparisons are hard and full of landmines and time-wasting efforts. For many years we have occasionally brought up the idea of a performance test suite for curl only to shut it down again because the challenges seemed hard and no one was volunteering to do this. This week it changed. “快”到底有多快？是否足够好？现在的运行速度是否和以前一样，还是出现了性能倒退？到底什么地方需要提速？速度指标是多少？这些都是许多项目和产品面临的问题，curl 也不例外。然而，性能测试和对比工作非常困难，充满了陷阱和耗时的琐事。多年来，我们偶尔会提出为 curl 建立性能测试套件的想法，但最终都因为挑战太大且无人愿意承担而作罢。本周，情况发生了变化。

Let’s do this

让我们开始吧

I started out trying to find existing projects that host performance results for Open Source projects so that we could just feed our results something else and get great visualizations and data management. I did not find any such. I then took a look at what existing tools there are for this purpose, and most pointers seemed to suggest that Grafana is a popular and maybe even a good solution to build something like this with. But man, that is a complicated machine and it felt more than a little overwhelming just figure out where or how to start with it. I decided to postpone that take as well. 我最初尝试寻找现有的开源项目性能结果托管平台，以便我们可以直接将结果导入其中，从而获得出色的可视化效果和数据管理。但我没找到这样的项目。随后，我研究了现有的相关工具，大多数建议都指向 Grafana，认为它是一个流行且可能不错的解决方案。但天哪，那套系统太复杂了，光是弄清楚从哪里或如何开始就让我感到不知所措。我决定暂时搁置这个方案。

Let me do this

让我自己来做

I decided that instead of trying to do this the best and optimal way – I shouldn’t let perfect be the enemy of good – I would start out by doing the things I know how to do and take it as far as I can one step at a time. Something should be better than nothing. 我决定不再追求最优方案——我不应让“完美”成为“优秀”的敌人——我将从自己擅长的事情做起，一步一个脚印地推进。有总比没有好。

Performance testing needs decently stable system conditions so that repeated runs produce reasonably similar results, when all involved factors remain identical. This is basically impossibly to accomplish using most cloud infrastructure since those are almost always shared with countless other users. At least on the cheap and free tiers we use. We probably need our own dedicated hardware for this, but instead of trying to figure out where to get that and arrange for that, I would start by running performance tests on my own local development machine. I am a single user on this and it has many cores and runs decently fast. It should be good enough to get this going on. 性能测试需要相对稳定的系统环境，以确保在所有相关因素保持不变的情况下，重复运行能产生大致相同的结果。在使用大多数云基础设施时，这几乎是不可能实现的，因为它们几乎总是与其他无数用户共享资源，至少在我们使用的廉价或免费层级上是这样。我们可能需要专用的硬件，但与其费心去寻找和安排硬件，我决定先在自己的本地开发机器上运行性能测试。我是这台机器的唯一用户，它拥有多个核心，运行速度也相当快。这应该足以让项目启动了。

I created a first shell script that updates the curl source code from git, it configures and builds it. Then it runs a bunch of tests, outputs a bunch of data and logs all the output in a single log file. I started out with a few simple tests. How fast does curl download a 100 GB file from localhost, how many allocations and how big allocations does it need for a single HTTP download? 我编写了第一个 shell 脚本，用于从 git 更新 curl 源代码，并进行配置和构建。然后，它会运行一系列测试，输出大量数据，并将所有输出记录在一个日志文件中。我从几个简单的测试开始：curl 从本地主机下载 100 GB 文件需要多快？单次 HTTP 下载需要多少次内存分配以及分配多大空间？

My second script parses all the test log files from the previous builds and generates summaries and graphs for them. To make it possible for humans to see how the performance changes between builds and ideally to automatically detect when something changes more than what should be tolerated. As I am a graph addict already since before, and that journey has taught me a little gnuplot, I decided that even while there probably are much better tools and fancy JavaScript things that could be used, I don’t know them and learning them now is an endeavor I rather avoid. So I stick to what I know and can get results with quickly. 我的第二个脚本负责解析之前构建的所有测试日志文件，并生成摘要和图表。这样人们就能直观地看到构建版本之间的性能变化，理想情况下还能自动检测出性能波动是否超出了可容忍范围。由于我一直是个“图表控”，且之前学过一点 gnuplot，我决定坚持使用自己熟悉的工具，尽管可能存在更好的工具或炫酷的 JavaScript 库，但我并不了解它们，现在去学习它们是我尽量想避免的。所以我坚持使用自己掌握的工具，以便快速获得结果。

A third script is invoked from a crontab every twenty minutes, sets up some variables and invokes the runner script. Once the basics started to work, I showed my curl friends the early versions and I soon created a new git repository for the code. 第三个脚本通过 crontab 每二十分钟调用一次，设置一些变量并启动运行脚本。一旦基础功能开始运作，我向 curl 的伙伴们展示了早期版本，并很快为这些代码创建了一个新的 git 仓库。

It’s live baby

它上线了，伙计们

After a little more poking, I soon made my locally produced performance test summary get packaged and automatically transferred to the curl website after each build, and voila, the first public curl performance tests were live and public. Getting this data available immediate triggered curl developers. It only took hours until we had the first proposed changes to improve some numbers, and soon we had a few merges to that affect. Visibility really helps! 经过一番折腾，我很快实现了将本地生成的性能测试摘要在每次构建后自动打包并传输到 curl 网站。瞧，首个公开的 curl 性能测试上线了。这些数据的公开立即引起了 curl 开发者的关注。仅仅几个小时后，我们就收到了第一批旨在优化性能指标的改进建议，随后很快就合并了相关代码。可见性确实很有帮助！

The performance numbers we get are still varying to a certain degree, partially of course because I still use my machine for my daily development things, but also because most of them do real (localhost) networking and that is by its nature a little… varying. The system builds and runs a new round every twenty minutes and it does that using the latest commits from git. This setup makes it sometimes run many rounds on the same commit and it might also mean that it sometimes updates and get several new commits at once, so it might skip a round for some commits. I might reconsider this design later, but since it is still a twenty minute time window, the number of commits is still limited. 我们获得的性能数据仍然存在一定程度的波动，部分原因当然是我还在用这台机器进行日常开发，但也因为大多数测试涉及真实的（本地主机）网络通信，这本身就带有一定的……不确定性。系统每二十分钟构建并运行一轮，使用的是 git 最新的提交。这种设置意味着有时会在同一个提交上运行多轮，也可能意味着有时会一次性更新多个提交，从而跳过某些提交的测试。我以后可能会重新考虑这个设计，但由于目前的时间窗口只有二十分钟，提交的数量仍然有限。

When the script makes multiple build rounds on the same commit, it accumulates the numbers and for the graph it stores the maximum, the median and the minimum value. It helps show the variation per commit and allows us to cram more into the graphs. It is still early days, but there will be a maximum limit to how many commits that can be displayed in a single graph and still be helpful. 当脚本在同一个提交上进行多轮构建时，它会累积数据，并在图表中存储最大值、中位数和最小值。这有助于展示每个提交的波动情况，并允许我们在图表中展示更多信息。目前还处于早期阶段，但单个图表中可显示的提交数量将有一个上限，以保证图表的有效性。

HTTP/2 parallel download speed through 261 builds spread over 31 build rounds 通过 31 轮构建、261 次构建版本得出的 HTTP/2 并行下载速度

Distribution

分布情况

To help visualize the distribution and data spread per test, I created a separate illustration that shows the minimum, maximum, P25, P75, medium and mean values in a Box-and-Whisker Plot. 为了帮助可视化每次测试的数据分布和离散程度，我创建了一个单独的图表，通过箱线图（Box-and-Whisker Plot）展示了最小值、最大值、P25、P75、中位数和平均值。

A Box-and-Whisker Plot showing the HTTP/2 parallel download speed data distribution. 展示 HTTP/2 并行下载速度数据分布的箱线图。

Changing conditions

变动的环境

An obvious downside with me just storing build logs in files, is that it will not scale up to the millions. I did however decide that I’m not designing this system for that. At least not now. Performance tests are highly specific and dependent on the exact machine it runs on, the exact third party libraries and their versions that are used, the other components involved in the tests, such as the servers, and more. I expect that we will change conditions for the tests every once in a while that makes it hard to compare the current numbers with past numbers. Therefore I think the performance test numbers 我仅仅将构建日志存储在文件中，一个明显的缺点是它无法扩展到百万级规模。但我决定，目前我设计的系统并非为了应对那种规模，至少现在不是。性能测试具有高度的特异性，取决于运行它的具体机器、所使用的第三方库及其版本、测试中涉及的其他组件（如服务器）等。我预计我们会不时更改测试条件，这使得将当前数据与过去的数据进行比较变得困难。因此，我认为性能测试数据……