The date format that broke my production AI agent (and the boring fix)
The date format that broke my production AI agent (and the boring fix)
导致我的生产环境 AI Agent 崩溃的日期格式(以及那个无聊的修复方案)
The demo was clean. Forty-three test cases, all passing. The agent took a structured input, processed it, wrote the result downstream. I watched it run a dozen times in staging, everything right. I put it live. Three days later, I found records in the database with dates in 1970.
演示过程非常完美。四十三项测试用例全部通过。Agent 接收结构化输入,进行处理,并将结果写入下游。我在预发布环境中观察了它运行十几次,一切正常。于是我将其上线。三天后,我在数据库中发现了日期为 1970 年的记录。
What the demo hid: The test inputs had date fields formatted consistently. ISO 8601, clean strings, nothing unusual. In production, one upstream system sent dates in a different format. The agent read the field, made a plausible guess about what it meant, and wrote the result. No exception, no warning. Just a wrong date, written confidently, downstream at 3am with no one watching.
演示中隐藏的问题是:测试输入的日期字段格式非常统一。都是 ISO 8601 标准,干净的字符串,没有任何异常。但在生产环境中,某个上游系统发送的日期格式不同。Agent 读取了该字段,对它的含义做出了一个看似合理的猜测,并写入了结果。没有异常,没有警告。只是一个错误的日期,在凌晨三点无人值守时,自信地被写入了下游。
The failure mode was not dramatic. The agent did not hallucinate in the way people imagine. It treated an ambiguous input as an unambiguous one, picked the most likely interpretation, and was wrong. Exactly the kind of quiet, confident wrongness that is hard to catch because nothing breaks. The output looks fine. The data type is correct. The value is just from 1970. I had tested the happy path forty-three times. The bad path happened on day three.
这种故障模式并不剧烈。Agent 并没有像人们想象的那样产生幻觉。它只是将模糊的输入当作明确的输入来处理,选择了最可能的解释,结果却错了。这正是那种安静且自信的错误,因为系统没有崩溃,所以很难被发现。输出看起来没问题,数据类型也正确,只是值变成了 1970 年。我测试了四十三次“理想路径”,但“错误路径”在第三天出现了。
The fix nobody wants to write: I added a validator between the input and the agent, and another one between the agent’s output and the write operation. The first one checks that required fields exist, that date strings match expected formats, that string lengths are within bounds. If anything fails, the item does not reach the agent. It goes to a review queue instead. No agent call, no write, no silent corruption.
没人愿意写的修复方案:我在输入和 Agent 之间添加了一个验证器,又在 Agent 的输出和写入操作之间添加了另一个验证器。第一个验证器检查必要字段是否存在、日期字符串是否符合预期格式、字符串长度是否在范围内。如果任何一项失败,该条目就不会到达 Agent,而是进入审核队列。没有 Agent 调用,没有写入,没有静默的数据损坏。
The second one runs after the agent produces its output. Before anything is written downstream, the schema gets checked. Required fields present. Types correct. Values within expected ranges. If the output fails, same result: review queue, no write. The agent itself did not change. The model did not change. The fix was wrapper logic that most production systems already have for traditional software and that almost no one thinks to add for AI outputs.
第二个验证器在 Agent 产生输出后运行。在写入下游之前,先检查 Schema。必要字段是否存在?类型是否正确?值是否在预期范围内?如果输出检查失败,结果一样:进入审核队列,不进行写入。Agent 本身没有改变,模型也没有改变。这个修复方案本质上是包装逻辑,大多数传统软件的生产系统都已经具备了这种逻辑,但几乎没人想到要为 AI 输出添加它。
At Agent Enterprise (aienterprise.dk), where I run the full agent operation, this is now the default. Every agent that writes to a persistent system has both checks. The agents do not get to decide whether their output is well-formed enough to write. The validator decides. The entire fix took about an hour to write. It is not interesting code.
在 Agent Enterprise (aienterprise.dk),即我负责全面 Agent 运营的地方,这现在已成为默认设置。每个写入持久化系统的 Agent 都具备这两项检查。Agent 无权决定其输出是否格式规范到可以写入,由验证器来决定。整个修复过程只花了一个小时左右。这并不是什么有趣的代码。
Reliability is the whole job: The demo proved the agent could do the task. On clean inputs, on the forty-three cases I had prepared, it was excellent. Production does not send you your forty-three cases. It sends you what the upstream system sends you, on whatever schedule, in whatever format, with whatever fields missing or malformed. The agent that works in a demo and the agent that works at 3am on a bad input are not the same thing.
可靠性就是全部工作:演示证明了 Agent 可以完成任务。在干净的输入和那四十三项我准备好的用例上,它表现出色。但生产环境不会只发送你那四十三项用例。它发送的是上游系统给你的东西,无论什么时间表、什么格式,无论缺少什么字段或格式是否错误。在演示中工作的 Agent 和在凌晨三点处理错误输入的 Agent,根本不是一回事。
The gap between them is not a model problem. It is an engineering problem. The fix was a type check on a date field and a decision about what to do when it fails. A branch in a validator that routes failures to a queue instead of letting them write. That is all. I have since added the same pattern to every agent that writes to anything. Validate the input before it reaches the agent, validate the output before it leaves. Everything the agent touches in between is its domain. Everything outside those bounds is yours.
它们之间的差距不是模型问题,而是工程问题。修复方案只是对日期字段进行类型检查,并决定失败时该怎么做。在验证器中增加一个分支,将失败的请求路由到队列,而不是让它们直接写入。仅此而已。此后,我将同样的模式应用到了每一个涉及写入操作的 Agent 上。在输入到达 Agent 之前进行验证,在输出离开之前进行验证。Agent 在这两者之间接触的一切是它的领域,而这些边界之外的一切,则是你的责任。
The demo is about capability. Production is about what happens when capability meets a case you did not design for. Reliability is the only feature that matters in that second environment, and it is almost never what the demo shows you.
演示展示的是能力。而生产环境展示的是:当能力遇到你未曾预料的情况时会发生什么。在生产环境中,可靠性是唯一重要的特性,而这几乎从来不是演示所能展示给你的。