I built an LLM eval framework from scratch. Here is what I wish I had bought instead.

我从零构建了一个大模型评估框架，这是我希望当初直接购买的部分

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever. Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, “how does this even work?” 有一个周末，我用大约两百行 Python 代码写了一个大模型（LLM）评估框架。演示效果非常棒，我当时觉得自己很聪明。六个月后，同一个框架变成了一团糟：三个不同的评判模型配着三种不同的解析技巧；一个自去年十一月起就无人问津的测试数据集；一个总是报错的 CI（持续集成）门禁，原因仅仅是供应商微调了模型，而不是有人改坏了提示词。接手的第二位工程师合理地问我：“这玩意儿到底是怎么运作的？”

The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again. The one line I wish someone had told me: build the rubric, buy the runner. 框架本身没坏，坏的是那些周末教程里跳过的、占工作量 80% 的部分。这其中的差距就是整个故事的核心，也是如果能重来，我会告诉自己的话。我希望有人早点告诉我的一句话是：构建评估准则（Rubric），购买运行器（Runner）。

Here is the split that took me six months to see. Some parts of an eval setup are yours and only yours. The rubric that decides what “good” means for your product. The dataset built from your real failures. The rules for when a change is bad enough to block a release. Nobody else can write these, because they encode your domain. 这是我花了六个月才看清的划分。评估系统中的某些部分是你独有的：决定你的产品什么是“好”的评估准则；基于你真实失败案例构建的数据集；以及判定变更严重到何种程度需要拦截发布的规则。这些东西别人写不了，因为它们编码了你的业务领域。

The rest is the same at every company. The thing that calls the judge model, parses its answer, retries, and caches. The machinery to run thousands of checks in parallel. The plumbing that scores live traffic. The system that groups failing calls together. Every team rebuilds these, hits the same bugs, and gains nothing by writing them twice. So build the first list. Do not hand-build the second. I rebuilt the second, and it cost me most of a year. 其余部分在每家公司都是一样的：调用评判模型、解析答案、重试和缓存的逻辑；并行运行数千次检查的机制；为实时流量打分的管道；将失败调用归类的系统。每个团队都在重复造这些轮子，踩同样的坑，重复编写这些代码毫无意义。所以，请构建第一类内容，不要手动构建第二类。我重写了第二类，结果耗费了我大半年的时间。

Two questions I now ask about every single piece. Before writing any part of this, I ask two things: Is it specific to me, or generic? A rubric for my domain is specific and worth owning. A retry-and-cache loop around a model call is generic. Everyone writes the same one. Does it compound, or does it rot? A dataset that grows from real production failures compounds. A year in, it is a regression suite no competitor can copy. A hand-built tracing layer rots. The moment a vendor changes something, you are behind and patching. Build the pieces that are specific and compound. Reuse everything else. That single filter would have saved me the whole mess. 我现在对每一部分都会问两个问题。在编写任何代码前，我会问：它是针对我的业务，还是通用的？针对我业务领域的评估准则具有特殊性，值得自己拥有。而模型调用的重试和缓存循环是通用的，每个人写的都一样。它是会产生复利，还是会腐烂？从真实生产失败中不断增长的数据集会产生复利，一年后，它将成为竞争对手无法复制的回归测试套件。而手动构建的追踪层会腐烂，一旦供应商做出改动，你就得疲于奔命地打补丁。构建那些具有特殊性和复利的部分，复用其他所有内容。这一个筛选标准本可以让我免于陷入那场混乱。

What is actually worth building yourself. Four things earned their place, and all four live in the same git repo as the agent code, reviewed in the same pull request as the prompt they score: The rubric. Keep it in git, not a database, so changes show up in diffs. And make it specific. “Rate the quality” is noise. “Score 5 if every claim is supported by the source, 1 if it is largely made up” is a signal you can act on. Broken-down rubrics like that agree with human reviewers noticeably more often. 什么才值得自己构建？有四样东西值得投入，且它们都应与 Agent 代码存放在同一个 Git 仓库中，并与它们所评估的提示词在同一个 Pull Request 中进行评审：评估准则。将其放在 Git 中而非数据库里，这样变更才能在 Diff 中体现。而且要具体，“评价质量”是无效信息，“如果每个主张都有来源支持则评 5 分，如果大部分是编造的则评 1 分”才是可执行的信号。这种细化的准则与人类评审员的意见一致性明显更高。

The dataset. A hundred real cases from production beat two hundred you invented. Pull the hardest failures in, because most of the signal lives in the worst ten percent of calls. Tag each case by area so a change to one feature does not rerun everything. The gating rules. The exact thresholds and policy for blocking a release are yours. The math behind them is not, but the decision of where the bar sits is. Your labels. The human judgement about what a correct score looks like. Only your team can produce that. 数据集。一百个来自生产环境的真实案例胜过两百个你虚构的案例。把最难的失败案例拉进来，因为大部分有效信号都存在于最差的那 10% 的调用中。按领域给每个案例打标签，这样修改一个功能时就不必重跑所有测试。门禁规则。拦截发布的具体阈值和策略属于你。背后的数学逻辑不属于你，但决定门槛设在哪里属于你。你的标签。关于什么是正确评分的人类判断，只有你的团队能产出这些。

What I should not have built. The judge engine is the one that hurt the most, so start there. Every judge model hands back its score in a slightly different shape. One wraps it in tags, another writes “Score: 4,” another buries the number after a paragraph of rambling. My parser caught almost all of it on day one. On day ninety a vendor shipped an update that formatted numbers differently, my parser quietly fell back to a middle score on the cases it missed, and my dashboard showed evals gently trending up. They were not. Middle scores just drag everything toward the average. 我不该构建什么？评判引擎是让我最痛苦的部分，所以从这里说起。每个评判模型返回分数的方式都略有不同。有的用标签包裹，有的写着“Score: 4”，有的把数字埋在一大段废话后面。我的解析器在第一天几乎捕获了所有情况。但在第九十天，供应商发布了一个更新，改变了数字格式，我的解析器在处理遗漏案例时悄悄回退到了中间分值，导致我的仪表盘显示评估结果在稳步上升。其实并没有，中间分值只会把一切都拉向平均水平。

On top of that came drift, where a judge scores the same answer differently this month than last after a model update, so the gate starts failing on prompts nobody touched. And cost, where running a top-tier judge across a big nightly set becomes a real budget line. Getting all of that right is months of work for an output that looks identical at every company. The same goes for running checks at scale, scoring live traffic, and grouping failing calls into themes. All generic, all a long slog, none of it your edge. I would reuse an existing runner for every one of these now. 此外还有“漂移”问题，即模型更新后，评判员对同一个答案的评分与上个月不同，导致门禁在无人改动的提示词上报错。还有成本问题，在大型夜间测试集中运行顶级评判模型会成为一笔沉重的预算开支。把这些做好需要数月工作，而产出在每家公司看起来都一样。大规模运行检查、为实时流量打分、将失败调用归类也是如此。这些都是通用的、繁琐的，且都不是你的核心竞争力。现在，我会为所有这些任务复用现有的运行器。

Gate on the change, not the average. This was my most expensive small mistake. I set the gate to “fail if the average score drops below a line.” The trouble is that a small dataset is noisy. With only thirty test cases, the average wobbles by a few points on its own, just from which cases you picked. So my gate failed on random noise and passed on real regressions, and within a month everyone had learned to ignore it, which is worse than having no gate at all. 基于变更而非平均值设置门禁。这是我犯过最昂贵的小错误。我将门禁设为“如果平均分低于某条线则失败”。问题在于小数据集是有噪声的。只有三十个测试用例时，平均分会因为你选取的用例不同而产生波动。所以我的门禁在随机噪声上报错，却放过了真正的回归错误。一个月内，大家就学会了无视它，这比根本没有门禁还要糟糕。

What actually works is two checks together: A hard floor that catches a catastrophic drop. A comparison against a rolling recent baseline that only fires when the drop is bigger than the normal wobble. The second one is the difference between a gate people trust and a gate people mute. 真正有效的是结合两种检查：一个捕捉灾难性下降的硬底线；一个与近期滚动基准的对比，仅在下降幅度超过正常波动范围时触发。后者是人们信任门禁与人们屏蔽门禁的区别所在。

Run the same check in CI and in production. I kept my checks in CI only, and production drifted past my dataset within a quarter. The fix is to run the exact same rubric in both places: against your test set before release, and against live traffic after. When they disagree, that disagreement is useful. If CI is green but production is failing, your dataset has stopped looking like reality. If CI fails on something production never shows, your dataset is carrying dead weight. Either way you learn something, but only if it is the same check on both sides. 在 CI 和生产环境中运行相同的检查。我只在 CI 中保留检查，结果一个季度内生产环境就偏离了我的数据集。解决方法是在两处运行完全相同的准则：发布前针对测试集，发布后针对实时流量。当它们产生分歧时，这种分歧是有价值的。如果 CI 通过但生产环境报错，说明你的数据集已脱离现实。如果 CI 在生产环境从未出现的场景上报错，说明你的数据集在承载无效负担。无论哪种情况，你都能有所收获，前提是两边必须是同一个检查。

The cost nobody warned me about. The weekend version is genuinely a weekend. The real version is not. Once you are shipping a couple of products, keeping all of this alive is close to one to one and a half full-time engineers. And the cost never shows up when you expect it. It shows up the week a judge model updates and your parser br… 没人警告过我的成本。周末版确实只需要一个周末，但真实版本绝非如此。一旦你开始发布几款产品，维持这一切的运转大约需要 1 到 1.5 名全职工程师。而且成本从不会在你预期的时候出现，它总是在评判模型更新、你的解析器崩溃的那一周出现……