There’s no reason for software to be slow anymore

软件不再有理由运行缓慢了

The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We’re not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance. 前几天，我看到一条疯传的推文，大意是说那些抱怨大语言模型（LLM）导致代码臃肿缓慢的人，等以后一切都被重写为超优化的汇编代码时，他们就得自食其果了。虽然我们还没到那种想把一切都用汇编编写的地步，但 Nolan Lawson 关于测试的观点——即你现在可以选择自己想要多少 Bug——正在性能领域变得愈发真实，正如我之前不太优雅地提到过的那样。

In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). 针对我上一篇文章中的一条评论，我提到过去那些需要专业技能的性能优化工作，其成本已经下降了几个数量级，现在任何能打出几句话的人都能完成，这意味着你可以进行各种曾经因成本过高而不值得尝试的优化（除非是超大规模或利润极高的项目）。Marc Brooker 对此回应道：“完全同意你的结论。针对特定工作负载而非一类工作负载的动态定制软件，似乎是一个非常可能的发展方向。”（当然，这也伴随着各种有趣的风险和机遇。）

Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality. And Michael Malis has noted There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT compiler. This is the thesis behind pgrust. Databases historically were the hardest piece of software to build and were limited because of that. Now, with AI, we can be more ambitious about the type of software we build. 这让我想起了 FFTW，以及许多古怪的旧式演示场景（demoscene）技术，它们的核心都是在特定问题（通常是特定硬件）上实现极致的速度和极小的体积。例如，我记得有一个演示程序通过将代码本身作为纹理重复使用，从而获得了极佳的缓存局部性。Michael Malis 曾指出，现在流传着一种说法，认为 AI 没用，因为“代码从来都不是难点”。我认为在某些领域确实如此，但在其他领域，编写代码绝对是难点。JIT 编译器就是一个很好的例子。对于许多软件来说，JIT 编译器能显著提升代码运行速度。JIT 编译器的稀缺让我相信，从历史上看，实现一个 JIT 编译器的难度太大，以至于不值得投入。而 LLM 降低了准入门槛，使得编写 JIT 编译器变得容易多了。这就是 pgrust 背后的核心论点。从历史上看，数据库是最难构建的软件，也因此受到了限制。现在，有了 AI，我们可以对构建的软件类型抱有更大的野心。

Optimizing for a class of workload

针对一类工作负载进行优化

Let’s try this out with FRE, the regex engine we built in the last post. Recall that it was created by having an agent loop for a month on improving regex engine performance with access to the rebar regex benchmark suite. This resulted in FRE being heavily overfit to rebar until we warned our agent that we had a holdout benchmark, which caused the agent to generalize the optimizations enough that performance was ok-ish on our holdout. There’s no particular reason to use a “software factory” regex engine that doesn’t beat a well-tested regex engine on holdout benchmarks, but one notable thing about FRE was that the native AOT compiled version did quite well at longer searches. 让我们用上一篇文章中构建的正则表达式引擎 FRE 来尝试一下。回想一下，它是通过让一个智能体循环一个月，利用 rebar 正则表达式基准测试套件来不断改进性能而创建的。这导致 FRE 对 rebar 产生了严重的过拟合，直到我们警告智能体我们还有一个留存（holdout）基准测试，才促使它将优化泛化，从而在留存测试中表现尚可。没有特别的理由去使用一个在留存基准测试中无法击败成熟引擎的“软件工厂”式正则引擎，但 FRE 的一个显著特点是，其原生 AOT 编译版本在长文本搜索中表现相当不错。

We noted that, it stands to reason that one could run the native code compiler in another thread while ripgrep was running its normal matcher and then cut over to the native code when it finished compiling and generally get better performance. Of course this will generally result in worse performance for short queries as we lose a thread to compilation, but I care a lot more about how long ripgrep takes when it runs for many seconds or minutes than when it runs for a few seconds, so I’m ok with that tradeoff. In the same way we could build a regex engine in a few minutes of human time, we can also just try this experiment in a few minutes of human time. I typed a few sentences and an agent went and did the work to allow this to happen (which would be a decent chunk of code surgery for a human) and it ran the benchmark on actual ripgrep queries that come from my codex history. 我们注意到，完全可以在 ripgrep 运行其常规匹配器的同时，在另一个线程中运行原生代码编译器，待编译完成后切换到原生代码，从而获得更好的性能。当然，这通常会导致短查询的性能下降，因为我们损失了一个线程用于编译，但我更关心 ripgrep 在运行数秒或数分钟时的表现，而不是几秒钟的短查询，所以我可以接受这种权衡。就像我们能在几分钟内构建一个正则引擎一样，我们也可以在几分钟内尝试这个实验。我输入了几句话，智能体就完成了实现这一功能的工作（这对人类来说是一项相当大的代码手术），并在我 codex 历史记录中的实际 ripgrep 查询上运行了基准测试。

For longer queries, we see a 2x-4x performance improvement here for a few very simple queries. But most queries are more complex, and when we run on representative holdout queries, for queries where AOT should be enabled, we get about a 7% speedup. Not an earth shattering result, but also not a bad outcome for spending a few minutes typing to codex (and it’s still doing more optimization and will presumably speed things up further). 对于较长的查询，在一些非常简单的查询中，我们看到了 2 到 4 倍的性能提升。但大多数查询更为复杂，当我们运行代表性的留存查询时，对于应该启用 AOT 的查询，我们获得了约 7% 的速度提升。这虽然不是什么惊天动地的结果，但对于只花了区区几分钟输入指令给 codex 来说，这已经是一个不错的结果了（而且它还在进行更多的优化，预计未来会进一步提升速度）。

Build an index? This is arguably a silly thing to do, since if we’re repeatedly searching for text on a computer, the obvious thing to do to speed that up isn’t to write a native code compiler for regex matching, it’s to create an index. But the point here is just that this kind of technical work, which used to take a fair amount of time and expertise, can just be done trivially now. And if we wanted to build a text index, it just so happens that I worked on BitFunnel, the Bing search index that was specialized for constant/fast text ingestion that won Best Paper Award at SIGIR, so I can think of a few experiments to try if we’re going to build a fast local index of our entire machine. 构建索引？这可以说是一件愚蠢的事情，因为如果我们要在电脑上反复搜索文本，加速它的显而易见的方法不是编写一个用于正则匹配的原生代码编译器，而是创建一个索引。但这里的重点是，这种过去需要大量时间和专业知识的技术工作，现在可以轻而易举地完成。碰巧的是，我曾参与过 BitFunnel 的开发，这是 Bing 的搜索索引，专门用于持续/快速的文本摄入，并获得了 SIGIR 的最佳论文奖。因此，如果我们要为整台机器构建一个快速的本地索引，我能想到几个可以尝试的实验。