The AI cost of JavaScript frameworks

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

The AI cost of JavaScript frameworks

One of my pet peeves with the web development community is that so many of the decisions we make are vibes-based. Frameworks are “blazing fast,” tools will “supercharge” your development, the right abstraction will benefit both you and your users (DX and UX all in one!). All of this breathless boasting is usually offered with zero evidence. Alex Russell has called this “frontend’s lost decade”, Baldur Bjarnason has called programming a pop culture, and I’ve tried to counter this trend in my blog with facts and benchmarks (at my best, anyway).

我对网络开发社区最不满的一点是，我们所做的许多决定都是基于“感觉”的。框架被吹捧为“极速”，工具能让开发“如虎添翼”，正确的抽象既能造福开发者（DX）又能造福用户（UX）。所有这些令人窒息的吹嘘通常都缺乏证据支持。Alex Russell 将此称为“前端失去的十年”，Baldur Bjarnason 称编程为一种流行文化，而我则试图在我的博客中用事实和基准测试来反驳这种趋势（至少在我尽力而为的时候）。

For a long time it seemed like our community’s allergy to evidence was driven by the desire for greater DX. In other words, devs just liked the way it felt to type JSX in React, and the arguments promoting React or related tools (“the virtual DOM is fast!”) were largely post-hoc rationalizations to justify those initial gut feelings. (Jonathan Haidt would likely agree with this assessment.)

长期以来，社区对证据的排斥似乎源于对更好开发体验（DX）的追求。换句话说，开发者只是喜欢在 React 中编写 JSX 的感觉，而推广 React 或相关工具的论点（如“虚拟 DOM 很快！”）大多是事后为了证明最初直觉而进行的合理化解释。（Jonathan Haidt 可能会同意这一评估。）

But in the agentic era, when developers are no longer typing code by hand and are thus less constrained by DX, I still see vibes ruling the day (and not just in the sense of “vibe-coding”). Frontends are rewritten because some framework or another is “better for agents.” (Did anyone think to ask the agents?) The resulting code is “faster,” the agents make “fewer mistakes,” etc. Once again, these claims are often made without evidence. Whenever I read one of these accounts, I think to myself, “Where are the benchmarks?”

但在智能体时代，当开发者不再亲手编写代码，从而不再受限于开发体验时，我仍然看到“感觉”主导着一切（不仅仅是在“凭感觉编程”的意义上）。前端被重写，仅仅是因为某个框架“对智能体更友好”。（有人想过问问智能体吗？）结果是代码“更快”了，智能体“犯错更少”了等等。再一次，这些说法往往没有证据支持。每当我读到这些叙述时，我都会想：“基准测试在哪里？”

Well I got curious, so I wrote a benchmark. The structure of this benchmark is pretty straightforward:

出于好奇，我编写了一个基准测试。这个测试的结构非常简单：

The app is fairly simple: you’ve got a list of Pokémon with their types, moves, and data from various games. You can search, filter, etc.:

这个应用相当简单：你有一个宝可梦列表，包含它们的属性、招式以及来自不同游戏的数据。你可以进行搜索、筛选等操作：

I should start off with some caveats: this is not a perfect experiment. This app represents only one holotype – it’s a relatively content-based site with little interactivity, closer to a Wikipedia than a Figma, GMail, etc. That said, I had to choose some app to build, and this one is at least easy to understand at a high level, and relatively straightforward to spec out and test.

我首先要说明一些注意事项：这不是一个完美的实验。这个应用仅代表一种原型——它是一个相对基于内容的网站，交互性较少，更接近维基百科，而不是 Figma 或 GMail 等。话虽如此，我必须选择一个应用来构建，而这个应用至少在宏观层面上易于理解，且在规格说明和测试方面相对简单。

To try to even the playing field, I had each framework/meta-framework use the official MCP or skills for that tool. I wanted to avoid accusations of “this framework is not well-represented in the training data, that’s why we built our MCP, we expect people to use it,” etc. I also didn’t over-specify the exact architecture to use (e.g. SSR vs CSR), nor did I specify that the app should be as performant as possible. I simply gave the spec to Opus 5.5 with high effort and told it to use the latest version of the given framework. Each agent ran in a sandbox and without any knowledge that it would be tested in Lighthouse on performance and accessibility, although it was given the conformance test suite.

为了尽量公平，我让每个框架/元框架使用其官方的 MCP 或工具技能。我希望避免被指责“该框架在训练数据中表现不佳，所以我们构建了 MCP，我们期望人们使用它”等等。我也没有过度指定要使用的确切架构（例如 SSR 与 CSR），也没有要求应用必须达到极致性能。我只是将规格说明交给 Opus 5.5，并要求它使用给定框架的最新版本。每个智能体都在沙盒中运行，且不知道自己会接受 Lighthouse 的性能和可访问性测试，尽管它被提供了合规性测试套件。

Note I also didn’t mandate any CSS methodology, except to avoid using Tailwind, Bootstrap, or any other component framework, and to design based on a set of spec tokens to enforce some visual consistency. I didn’t want to make this test all about CSS, but instead about JS frameworks, CSR vs SSR, SPAs vs MPAs, etc. The resulting app is tested against a conformance suite based on the accessibility tree rather than the raw DOM to avoid mandating any particular HTML structure – one framework uses web components which rely on shadow DOM, so this was required.

请注意，我也没有强制要求任何 CSS 方法，只是避免使用 Tailwind、Bootstrap 或任何其他组件框架，并基于一组规格标记进行设计以确保视觉一致性。我不想让这个测试变成关于 CSS 的测试，而是关于 JS 框架、CSR 与 SSR、SPA 与 MPA 等的测试。最终的应用是根据可访问性树而非原始 DOM 进行合规性测试，以避免强制要求特定的 HTML 结构——其中一个框架使用了依赖于 Shadow DOM 的 Web 组件，因此这是必要的。

One thing I learned from this experiment is that everything I’m measuring is very stochastic. LLMs are random-word generators, so sometimes you get a low dice-roll and an agent goes off on some weird tangent wasting a bunch of tokens, through no fault of the framework or its tools. Lighthouse itself can give inconsistent results as well: Largest Contentful Paint (LCP) may decide that one image is the “largest” element, then pick another one on the next run. It’s hard to have a perfectly fair experiment, but running multiple iterations did help, and I’ve tried to be honest about the variance.

从这次实验中我学到的一点是，我所测量的一切都具有高度的随机性。大语言模型是随机词生成器，所以有时你会遇到“掷骰子”点数较低的情况，智能体可能会跑偏，浪费大量 Token，这并非框架或其工具的过错。Lighthouse 本身也可能给出不一致的结果：最大内容绘制（LCP）可能在一次运行中判定某张图片是“最大”元素，而在下一次运行中又选择另一个。很难做到绝对公平的实验，但多次迭代确实有所帮助，我也尽量诚实地说明了其中的差异。

With all that throat-clearing out of the way, let’s get to the results.

铺垫到此为止，让我们看看结果。

I’m going to break this down by multiple dimensions: cost to build, quality, and maintainability. For each framework/meta-framework, I took the median of 5 runs: in other words, I built the same app 5 times in the same framework. (Yes, this is the kind of zany thing you can do with coding agents these days.)

我将从多个维度进行分析：构建成本、质量和可维护性。对于每个框架/元框架，我取了 5 次运行的中位数：换句话说，我在同一个框架中构建了 5 次相同的应用。（是的，这就是如今使用编码智能体可以做的疯狂事情。）

The exception was Angular, which failed to create a measurable app twice, so I took 3 iterations instead for some metrics. (This is explained below.)

例外的是 Angular，它有两次未能创建出可测量的应用，因此我在某些指标上只取了 3 次迭代。（下文会有解释。）

Note that this quotes the API prices directly from Claude Code and doesn’t actually represent how much money I spent. I’m on a Max plan, so this was just part of my overall $100/month.

请注意，这里引用的 API 价格直接来自 Claude Code，并不代表我实际花费的金额。我使用的是 Max 计划，这只是我每月总共 100 美元费用的一部分。

Note also the variance: React Router’s median is roughly $1 worse than that of Hono+htmx, but the cheapest version of React Router ($2.82) is still cheaper than the most expensive version of Hono+htmx ($3.19). However some reliable patterns do emerge.

还要注意差异：React Router 的中位数比 Hono+htmx 大约贵 1 美元，但 React Router 最便宜的版本（2.82 美元）仍然比 Hono+htmx 最昂贵的版本（3.19 美元）便宜。不过，一些可靠的模式确实显现出来了。

Most clearly, SvelteKit, Gracile+Lit, and Angular are more expensive to build than the other frameworks. For SvelteKit, this may be due to the fact that the agents read the SvelteKit v3 migration guide in every run, likely because it’s too new to be in Opus 5.5’s training data. Similarly for Gracile, the agents didn’t know the framework very well and either cloned the Gracile repo or read its compiled source from node_modules. For Angular it seems to be a function of the total code size (see below).

最明显的是，SvelteKit、Gracile+Lit 和 Angular 的构建成本比其他框架更高。对于 SvelteKit，这可能是因为智能体在每次运行中都会读取 SvelteKit v3 的迁移指南，很可能是因为它太新，不在 Opus 5.5 的训练数据中。同样对于 Gracile，智能体对该框架不太了解，要么克隆了 Gracile 仓库，要么从 node_modules 读取了其编译后的源代码。对于 Angular，这似乎与总代码量有关（见下文）。

I should also mention that not every agent made use of an MCP/skill/Claude Code plugin, nor did every framework have one:

我还应该提到，并非每个智能体都使用了 MCP/技能/Claude Code 插件，也不是每个框架都有这些工具：

For example, TanStack Start’s agent only discovered its skill files in 2/5 runs (they’re shipped in the node_modules of the project). For Nuxt, the MCP was never used by the agents. For Next.js, agents only read the skill file in 1/5 runs and the AGENTS.md in 2/5 runs. This is a wide range of results, which points back to the stochastic nature of LLMs. I still consider this a valid experiment, though, because I doubt many

例如，TanStack Start 的智能体仅在 5 次运行中的 2 次发现了其技能文件（它们随项目的 node_modules 一起发布）。对于 Nuxt，智能体从未使用过 MCP。对于 Next.js，智能体仅在 5 次运行中的 1 次读取了技能文件，在 2 次中读取了 AGENTS.md。这些结果差异巨大，这再次指向了大语言模型的随机性。尽管如此，我仍然认为这是一个有效的实验，因为我怀疑很多人