The model obeys your schema, not your description

模型遵循的是你的 Schema，而非你的描述

Two models. Same prompt, same tool description, same request. One of them returned this: { "kind": "entity", "entityName": "todo", "definition": { "fields": { "title": "text" } } } The other returned this: { "kind": "entity", "name": "todo", "fields": { "title": "text" } } The second one is wrong, and our downstream patcher rejected it with a 422 that told nobody anything useful. What took me a while to work out was why the first model got it right, because the answer turned out to have nothing to do with being smarter. This was May 2026, Opus 4.7 and Sonnet 4.6 at the time. The story generalizes to whatever pair of models you’re holding today.

两个模型，同样的提示词，同样的工具描述，同样的请求。其中一个返回了：{ "kind": "entity", "entityName": "todo", "definition": { "fields": { "title": "text" } } }。另一个返回了：{ "kind": "entity", "name": "todo", "fields": { "title": "text" } }。第二个结果是错误的，我们的下游补丁程序拒绝了它，并返回了一个毫无意义的 422 错误。我花了一段时间才弄明白为什么第一个模型做对了，因为答案与“更聪明”无关。那是 2026 年 5 月，当时使用的是 Opus 4.7 和 Sonnet 4.6。这个故事适用于你今天使用的任何模型组合。

The setup

设置

We have a tool called apply_patches. An LLM reads a user request plus a source file and emits a list of structural change operations: add this entity, replace that handler, remove that metric. Each operation carries a pattern, the canonical object form of the thing being changed. The tool schema for that pattern parameter was, in effect: { "type": "object", "properties": { "kind": { "type": "string" } }, "required": ["kind"] } kind is a string and everything else is whatever. The actual shape lived in the tool description, a paragraph of prose with examples, the way most people write tool definitions. The big model complied anyway. The smaller one didn’t. Both had read the same description.

我们有一个名为 apply_patches 的工具。LLM 读取用户请求和源文件，并输出一系列结构化变更操作：添加此实体、替换该处理程序、移除该指标。每个操作都包含一个模式（pattern），即被变更对象的规范化形式。该模式参数的工具 Schema 实际上是：{ "type": "object", "properties": { "kind": { "type": "string" } }, "required": ["kind"] }。kind 是一个字符串，其他部分则随意。实际的结构存在于工具描述中——那是一段带有示例的散文，也是大多数人编写工具定义的方式。大模型照做了，但小模型没有。它们都阅读了相同的描述。

Why the big model complied

为什么大模型能做到

It had seen the shape before. entityName and definition.fields are our field names, from our framework. To a model with training exposure to that shape, “emit an entity pattern” retrieves a memory. To a model without it, “emit an entity pattern” is a guess from the description text, and if you’re guessing what an entity looks like, { name, fields } is a better guess than the truth. It’s what everyone else’s API would call those things. So this is an exposure gap rather than a capability gap, which matters because you can’t fix an exposure gap by paying for a bigger model. It will show up for any model on any shape that isn’t in its training data, which is to say on your proprietary shapes, indefinitely. The less your schema looks like the rest of the internet, the harder your description has to work. And descriptions are not what the model is validated against. The tool schema is. So we moved the contract into it.

因为它以前见过这种结构。entityName 和 definition.fields 是我们框架中的字段名。对于接触过这种结构训练的模型来说，“输出一个实体模式”是提取记忆。而对于没有接触过的模型，“输出一个实体模式”则是根据描述文本进行的猜测。如果你在猜测一个实体长什么样，{ name, fields } 显然比真实情况更像是一个合理的猜测——毕竟其他人的 API 大多也是这么命名的。因此，这是一个“暴露差距”（exposure gap）而非“能力差距”（capability gap）。这一点很重要，因为你无法通过购买更大的模型来解决暴露差距。对于任何不在其训练数据中的结构（即你的私有结构），任何模型都会无限期地出现这种问题。你的 Schema 越不像互联网上的通用格式，你的描述就越需要发挥作用。但模型验证的依据并非描述，而是工具 Schema。所以，我们将契约移入了 Schema 中。

Tight on the common kinds, loose on the tail

常见类型从严，长尾类型从宽

We have around twenty pattern kinds. Nine of them account for roughly 85% of everything the model emits. The other dozen (relation, workspace, secret, claimKey, systemScope and friends) show up rarely. Writing strict schemas for all twenty would have been a week of work and a permanent maintenance tax, so we didn’t. Each common kind became a discriminated oneOf branch with a real required list: { title: "EntityPattern", properties: { kind: { const: "entity" }, entityName: { type: "string" }, definition: { type: "object" }, }, required: ["kind", "entityName", "definition"], } The long tail got one fallback branch that requires nothing but kind: { title: "OtherPattern", properties: { kind: { type: "string", not: { enum: [ "entity", "requires", "toggleable", "nav", "writeHandler", "queryHandler", "hook", "notification", "metric", ], }, }, }, required: ["kind"], }

我们大约有二十种模式类型。其中九种占了模型输出总量的 85% 左右。其余十几种（如 relation, workspace, secret, claimKey, systemScope 等）很少出现。为所有二十种类型编写严格的 Schema 需要一周的工作量，且会带来永久的维护成本，所以我们没有这样做。每种常见类型都变成了一个带有明确 required 列表的 oneOf 分支：{ title: "EntityPattern", properties: { kind: { const: "entity" }, entityName: { type: "string" }, definition: { type: "object" }, }, required: ["kind", "entityName", "definition"], }。长尾部分则获得了一个仅要求 kind 的回退分支：{ title: "OtherPattern", properties: { kind: { type: "string", not: { enum: [ "entity", "requires", "toggleable", "nav", "writeHandler", "queryHandler", "hook", "notification", "metric", ], }, }, }, required: ["kind"], }

Rare kinds still go through unvalidated at the schema layer, and the runtime patcher catches them. That split, tight on the discriminator values you see constantly and permissive on the ones you don’t, is the part worth stealing. It costs an afternoon instead of a week and it targets the failures you actually get. We did the same for the natural keys that replace and remove operations use, and pinned the per-operation requirements with allOf plus if/then, so the model can’t hand us a replace with nothing to replace: allOf: [ { if: { properties: { op: { const: "replace" } } }, then: { required: ["id", "pattern"] } }, { if: { properties: { op: { const: "add" } } }, then: { required: ["pattern"] } }, { if: { properties: { op: { const: "remove" } } }, then: { required: ["id"] } }, ] Both Anthropic and OpenAI honor oneOf and allOf/if/then in tool input schemas. Most people skip them because the flat version works well enough on whichever model they tested with.

罕见类型在 Schema 层仍不进行验证，由运行时补丁程序捕获。这种“对高频判别值从严，对低频值从宽”的拆分方式非常值得借鉴。它只需一个下午而非一周的时间，且能针对性地解决你实际遇到的失败。我们对 replace 和 remove 操作使用的自然键也做了同样的处理，并使用 allOf 配合 if/then 锁定了每个操作的需求，这样模型就无法给我们发送一个没有目标对象的 replace 操作：allOf: [ { if: { properties: { op: { const: "replace" } } }, then: { required: ["id", "pattern"] } }, { if: { properties: { op: { const: "add" } } }, then: { required: ["pattern"] } }, { if: { properties: { op: { const: "remove" } } }, then: { required: ["id"] } }, ]。Anthropic 和 OpenAI 都支持工具输入 Schema 中的 oneOf 和 allOf/if/then。大多数人跳过这些，是因为扁平化的版本在他们测试的模型上已经“足够好用”了。

Did it work?

有效吗？

The evidence is thinner than I’d like, and it points the right way. We ran three fixtures live, twice, for about $0.18 total. Before the change: two passed, one failed. The failure was the rename-entity case, emitting { kind: "entity", name, fields }. After: three passed, and the fixture that had been failing emitted { kind: "entity", entityName: "todo", definition: { fields: ... } }, byte for byte the shape the big model had been producing all along. Three fixtures is not a benchmark. What convinced me was which failure disappeared and what replaced it. The smaller model stopped inventing field names and started producing the canonical shape, on the exact case that had been failing. The real payoff came later. Once the smaller model could reliably emit the structured shape, it became viable as the default. That’s usually the whole business case for schema work: a tight schema is what makes the cheap model good enough.

证据比我预想的要少，但方向是正确的。我们运行了三个测试用例，各两次，总共花费约 0.18 美元。修改前：两个通过，一个失败。失败的是重命名实体的情况，输出了 { kind: "entity", name, fields }。修改后：三个全部通过，之前失败的那个用例输出了 { kind: "entity", entityName: "todo", definition: { fields: ... } }，这与大模型一直以来输出的结构完全一致。三个测试用例算不上基准测试，但真正说服我的是：消失的失败类型以及取而代之的结果。小模型不再随意发明字段名，而是开始在之前失败的用例上输出规范结构。真正的回报在后面：一旦小模型能可靠地输出结构化数据，它就具备了作为默认模型的资格。这通常就是进行 Schema 工作的全部商业价值所在：严谨的 Schema 是让廉价模型变得“足够好”的关键。

Footgun 1: oneOf is strict XOR

隐患 1：oneOf 是严格的异或 (XOR)

Two weeks later, a code review caught something the tests hadn’t. For replace and remove operations we have a parallel set of variants describing just the natural key. One of them is a singleton fallback: a kind and nothing else. And { kind: "entity" } matched both the entity branch and the fallback branch. oneOf means exactly one. Two matches is a violation. Anthropic tolerated it; OpenAI’s strict mode rejects the schema outright. Same schema, one provider silently fine, the other refusing to run. The fix is the not.enum you saw above, where the fallback explicitly excludes every kind that has its own branch. Worth internalizing if you’re building discriminated unions in JSON Schema: a fallback branch is not automatically disjoint from the specific ones. You have to make it disjoint by hand, and a test helps.

两周后，一次代码审查发现了测试未覆盖的问题。对于 replace 和 remove 操作，我们有一组平行的变体，仅描述自然键。其中一个是单例回退：只有一个 kind。而 { kind: "entity" } 同时匹配了实体分支和回退分支。oneOf 意味着“恰好一个”，匹配两个即为违规。Anthropic 容忍了这一点，但 OpenAI 的严格模式直接拒绝了该 Schema。同样的 Schema，一个提供商默默接受，另一个拒绝运行。解决方法就是上面提到的 not.enum，即回退分支显式排除了所有拥有独立分支的类型。如果你正在 JSON Schema 中构建判别联合（discriminated unions），请记住：回退分支不会自动与特定分支互斥。你必须手动确保它们互斥，而测试会有所帮助。