Structured Output From LLMs: A Retry-Repair Loop Your Parser Never Sees Through

查看原文 / View Original 编辑 / Edit

Structured Output From LLMs: A Retry-Repair Loop Your Parser Never Sees Through

LLM 的结构化输出：解析器永远无法察觉的重试-修复循环

The first time I wired an LLM into a real product feature at Shpper, I did the naive thing: prompt the model to “return JSON”, jsonDecode the response, move on. It worked in the demo. Then it hit real traffic and I started getting FormatException at 2am because the model wrapped its JSON in a json fence, or added a cheerful "Here's the data you asked for!" preamble, or trailed a comma before the closing brace. 当我第一次在 Shpper 将 LLM 接入真实产品功能时，我做了一件很天真的事：提示模型“返回 JSON”，对响应进行 jsonDecode，然后就完事了。这在演示中运行良好。但当它面对真实流量时，我开始在凌晨两点收到 FormatException，因为模型用 json 代码块包裹了 JSON，或者添加了诸如“这是您要的数据！”之类的愉快前言，又或者在右大括号前多加了一个逗号。

A model that’s right 97% of the time is still wrong on thousands of requests a day. Reliable structured output isn’t a prompting trick — it’s a small pipeline, and the last stage is a repair loop your parser never sees through. This is the pattern I reach for every time I need an LLM to hand back a typed object instead of prose: contact extraction, invoice parsing, classification against a fixed label set, turning a messy paragraph into a database row. The shape is identical every time, and once you internalize it you stop firefighting malformed JSON for good. 一个有 97% 准确率的模型，每天仍会在数千次请求中出错。可靠的结构化输出不是靠提示词技巧实现的，它是一个小型流水线，而最后阶段是一个你的解析器永远无法察觉的修复循环。每当我需要 LLM 返回类型化对象而非散文时，我都会使用这种模式：联系人提取、发票解析、针对固定标签集的分类，或者将杂乱的段落转换为数据库行。这种模式每次都一样，一旦你内化了它，你就再也不用为格式错误的 JSON 疲于奔命了。

Why “return JSON” fails in production

为什么“返回 JSON”在生产环境中会失败

The failure modes are boring and relentless, which is exactly why they’re worth naming. Prompt-only JSON breaks in a handful of predictable ways: 这些故障模式既枯燥又无情，正因如此，它们才值得被列举出来。仅靠提示词生成的 JSON 会以几种可预测的方式崩溃：

  • Markdown fences. The model wraps the object in a ```json block, so your raw string starts with backticks, not {.
    • Markdown 代码块。 模型用 ```json 块包裹对象，导致你的原始字符串以反引号开头，而不是 {。
  • Conversational preamble or trailer. “Sure! Here’s the JSON:” gets prepended, or a “Let me know if you need anything else!” gets appended — either way the payload isn’t parseable end to end.
    • 对话式前言或后语。 前面加上“好的！这是 JSON：”，或者后面加上“如果还需要什么请告诉我！”，无论哪种情况，载荷都无法被完整解析。
  • Trailing commas and single quotes. Valid-looking to a human, invalid to a strict JSON parser.
    • 多余的逗号和单引号。 对人类来说看起来没问题，但对严格的 JSON 解析器来说是无效的。
  • Type drift. You asked for a string, you got a number. You asked for an array, you got a comma-joined string. Syntactically fine, semantically wrong.
    • 类型漂移。 你要字符串，它给了数字。你要数组，它给了逗号分隔的字符串。语法上没问题，语义上却错了。
  • Missing or hallucinated fields. The model omits the one field you actually needed, or invents a confidence: 0.9 you never asked for.
    • 字段缺失或幻觉。 模型遗漏了你真正需要的字段，或者编造了一个你从未要求的 confidence: 0.9。

None of these are exotic. They’re the median Tuesday. The mistake is treating them as bugs to squash one by one instead of a class of failures to absorb architecturally. 这些都不是什么稀奇事，而是每周二都会发生的常态。错误的做法是将它们视为需要逐一修复的 Bug，而不是将其视为一类需要在架构层面去吸收的故障。

The three layers, from strongest to weakest

三个层级：从最强到最弱

You have three tools to force structure, and you should reach for them in this order — strongest guarantee first. 你有三种工具来强制结构化，你应该按此顺序使用它们——优先选择最强的保证。

  1. Constrained decoding (JSON mode / schema-enforced output). The provider constrains token sampling so the output is guaranteed to be syntactically valid JSON, and with a supplied JSON Schema, guaranteed to match the shape. This is the strongest guarantee because it operates at the sampling layer, not the prompt layer — the model literally cannot emit a token that would break the grammar.
    • 约束解码（JSON 模式 / Schema 强制输出）。 提供商限制了 Token 采样，从而保证输出在语法上是有效的 JSON；如果提供了 JSON Schema，还能保证符合预期的形状。这是最强的保证，因为它作用于采样层而非提示层——模型在字面上无法输出破坏语法的 Token。
  2. Tool / function calling. You describe a function with a typed parameter schema; the model emits a structured call to it. This is the same constrained-decoding machinery wearing a different hat, and it’s the cleanest fit when the structured object is an action — create_invoice, extract_contact, schedule_meeting. If your data extraction is really “the model deciding to do a thing”, model it as a tool and let the SDK enforce the arguments.
    • 工具/函数调用。 你用类型化参数 Schema 描述一个函数；模型会发出对该函数的结构化调用。这本质上是换了马甲的约束解码机制，当结构化对象是一个动作（如 create_invoice, extract_contact, schedule_meeting）时，这是最合适的方案。如果你的数据提取本质上是“模型决定做某事”，那就把它建模为工具，让 SDK 来强制执行参数。
  3. Prompt-and-pray plus validation. You ask for JSON in the prompt and validate what comes back. This is the weakest layer and, unfortunately, the one you fall back to whenever a provider, model, or gateway doesn’t support the stronger modes. Older models, some open-weights deployments behind a proxy, and certain streaming paths still land you here.
    • 提示词祈祷加验证。 在提示词中要求 JSON 并验证返回结果。这是最弱的一层，不幸的是，每当提供商、模型或网关不支持更强的模式时，你只能退回到这一层。旧模型、代理后的某些开源权重部署以及特定的流式传输路径仍然会让你处于这种境地。

Here’s what people miss: even the strong layers don’t free you from validation. JSON mode guarantees syntactic validity — that you can jsonDecode it. It does not guarantee the model filled in the field you needed, respected your enum, or didn’t drop a null where you require a string. Schema-enforced modes are much better, but a schema can’t express every business rule: this date must be after that date; this array must be non-empty when type == “premium”; this email must belong to a domain you support. So the architecture is always the same three moves: generate as constrained as the provider allows, then validate against your own source of truth, then repair. 人们容易忽略的一点是：即使是强层级也无法让你免于验证。JSON 模式保证了语法有效性——即你可以 jsonDecode 它。但它不能保证模型填好了你需要的字段、遵守了你的枚举，或者在你要求字符串的地方没有填入 null。Schema 强制模式好得多，但 Schema 无法表达所有的业务规则：比如日期必须在某日期之后；当 type == “premium” 时数组不能为空；邮箱必须属于你支持的域名。因此，架构始终是同样的三个步骤：在提供商允许的范围内尽可能约束生成，然后根据你自己的事实来源进行验证，最后进行修复。

Validate against a schema you own

针对你拥有的 Schema 进行验证

Do not hand-roll if (json[‘name’] == null) checks scattered across your codebase. Define the contract once and validate against it. In the Dart-heavy world I live in that means a real model class with a strict parser; on a Node or Python backend I’ll use JSON Schema directly, or a Zod / Pydantic-style validator that doubles as the schema I send to the provider. The parser has one job: turn any input into either a valid typed object or a structured error that describes exactly what’s wrong — because that error is the input to the repair step. A parser that throws a generic “invalid” is useless here; the specificity of the error determines the quality of the repair. 不要在代码库中到处手写 if (json[‘name’] == null) 检查。定义一次契约并针对它进行验证。在我所处的 Dart 环境中，这意味着使用带有严格解析器的真实模型类；在 Node 或 Python 后端，我会直接使用 JSON Schema，或者使用 Zod/Pydantic 风格的验证器，它同时充当了我发送给提供商的 Schema。解析器只有一个任务：将任何输入转换为有效的类型化对象，或者转换为描述具体错误的结构化错误——因为该错误是修复步骤的输入。抛出通用“无效”错误的解析器在这里毫无用处；错误的具体程度决定了修复的质量。

class SchemaError implements Exception {
  final List<String> messages;
  SchemaError(this.messages);
  @override String toString() => 'SchemaError: ${messages.join('; ')}';
}

class ContactRecord {
  final String name;
  final String email;
  final String? company;

  ContactRecord({required this.name, required this.email, this.company});

  /// Returns the record, or throws SchemaError with a machine-useful message list.
  factory ContactRecord.parse(Map<String, dynamic> json) {
    final errors = <String>[];
    final name = json['name'];
    final email = json['email'];

    if (name is! String || name.trim().isEmpty) {
      errors.add('"name" must be a non-empty string');
    }
    if (email is! String || !email.contains('@')) {
      errors.add('"email" must be a valid email address containing "@"');
    }

    if (errors.isNotEmpty) {
      throw SchemaError(errors); // carries the list to the repair loop
    }

    return ContactRecord(
      name: name as String,
      email: email as String,
      company: json['company'] as String?,
    );
  }
}

The load-bearing detail is that SchemaError carries a specific, actionable list — not just “invalid JSON”. Vague errors produce vague repairs. “email” must contain ”@” gets fixed on the next turn; “validation failed” gets you the same broken output again. 关键的细节在于 SchemaError 携带了一个具体的、可操作的列表，而不仅仅是“无效的 JSON”。模糊的错误会导致模糊的修复。“email 必须包含 @”可以在下一轮中被修复；而“验证失败”只会让你再次得到同样的错误输出。

← 返回新闻列表 / Back to news list