A new Go API for JSON

本文为原文前 6,000 字符的节选翻译,完整内容请查看原文。

A new Go API for JSON

The Go Blog A new Go API for JSON Michael Pratt, Joe Tsai, Daniel Martí, Johan Brandhorst-Satzkorn, Roger Peppe, Chris Hines, and Damien Neil 4 November 2026 Introduction JavaScript Object Notation (JSON) is a simple data interchange format, the most popular data format used on the internet, and widely read and written by Go programs. Go has included support for JSON in the standard library since well before Go 1.0. We first we wrote about support for JSON in Go more than 15 years ago! Over time, packages evolve with the needs of their users, and encoding/json is no exception. Last year, we wrote about a new experimental API for JSON designed to bring long-awaited improvements and fixes. After additional feedback and refinement, the new encoding/json/v2 and encoding/json/jsontext packages are stable and generally available in Go 1.27! This post is an updated version of the article introducing the experiment, describing the problems with encoding/json, and an introduction to the new packages.

Go 博客:Go 的全新 JSON API。作者:Michael Pratt, Joe Tsai, Daniel Martí, Johan Brandhorst-Satzkorn, Roger Peppe, Chris Hines, 以及 Damien Neil。2026 年 11 月 4 日。引言:JavaScript 对象表示法 (JSON) 是一种简单的数据交换格式,也是互联网上最流行的数据格式,被 Go 程序广泛读取和写入。自 Go 1.0 发布之前,Go 的标准库就已包含对 JSON 的支持。我们早在 15 年前就首次撰文介绍了 Go 对 JSON 的支持!随着时间的推移,软件包会随着用户需求的发展而演进,encoding/json 也不例外。去年,我们介绍了一个旨在带来期待已久的改进和修复的全新实验性 JSON API。经过进一步的反馈和完善,全新的 encoding/json/v2 和 encoding/json/jsontext 软件包现已稳定,并可在 Go 1.27 中使用!本文是介绍该实验文章的更新版本,描述了 encoding/json 存在的问题,并介绍了这些新软件包。

Problems with encoding/json Overall, encoding/json has held up well. The idea of marshaling and unmarshaling arbitrary Go types with some default representation in JSON, combined with the ability to customize the representation, has proven to be highly flexible. However, in the years since its introduction, various users have identified numerous shortcomings.

encoding/json 的问题:总体而言,encoding/json 的表现一直不错。将任意 Go 类型与 JSON 中的某种默认表示形式进行序列化和反序列化的理念,结合自定义表示形式的能力,已被证明具有高度的灵活性。然而,自其推出以来的几年里,各种用户发现了许多缺陷。

Behavior flaws There are various behavioral flaws in encoding/json: Imprecise handling of JSON syntax: Over the years, JSON has seen increased standardization in order for programs to properly communicate. Generally, decoders have become stricter at rejecting ambiguous inputs, to reduce the chance that two implementations will have different (successful) interpretations of a particular JSON value. encoding/json currently accepts invalid UTF-8, whereas the latest Internet Standard (RFC 8259) requires valid UTF-8. The default behavior should report an error in the presence of invalid UTF-8, instead of introducing silent data corruption, which may cause problems downstream. encoding/json currently accepts objects with duplicate member names. RFC 8259 does not specify how to handle duplicate names, so an implementation is free to choose an arbitrary value, merge the values, discard the values, or report an error. The presence of a duplicate name results in a JSON value without a universally agreed upon meaning. This could be exploited by attackers in security applications and has been exploited before (as in CVE-2017-12635). The default behavior should err on the side of safety and reject duplicate names.

行为缺陷:encoding/json 存在各种行为缺陷:对 JSON 语法的处理不够精确:多年来,为了使程序能够正确通信,JSON 的标准化程度不断提高。通常,解码器在拒绝歧义输入方面变得更加严格,以减少两个实现对特定 JSON 值产生不同(成功)解释的可能性。encoding/json 目前接受无效的 UTF-8,而最新的互联网标准 (RFC 8259) 要求必须是有效的 UTF-8。默认行为应该在出现无效 UTF-8 时报告错误,而不是引入静默的数据损坏,这可能会导致下游出现问题。encoding/json 目前接受具有重复成员名称的对象。RFC 8259 没有规定如何处理重复名称,因此实现可以自由选择任意值、合并值、丢弃值或报告错误。重复名称的存在会导致 JSON 值没有普遍公认的含义。这可能会被安全应用程序中的攻击者利用,并且以前也曾被利用过(如 CVE-2017-12635)。默认行为应偏向安全性,并拒绝重复名称。

Leaking nilness of slices and maps: JSON is often used to communicate with programs using JSON implementations that do not allow null to be unmarshaled into a data type expected to be a JSON array or object. Since encoding/json marshals a nil slice or map as a JSON null, this may lead to errors when unmarshaling by other implementations. A survey indicated that most Go users prefer that nil slices and maps are marshaled as an empty JSON array or object by default. Case-insensitive unmarshaling: When unmarshaling, a JSON object member name is resolved to a Go struct field name using a case-insensitive match. This is a surprising default, a potential security vulnerability, and a performance limitation. Inconsistent calling of methods: Due to an implementation detail, MarshalJSON methods declared on a pointer receiver are inconsistently called by encoding/json. While regarded as a bug, this cannot be fixed as too many applications depend on the current behavior.

切片和映射的 nil 值泄露:JSON 常用于与使用其他 JSON 实现的程序进行通信,而这些实现不允许将 null 反序列化为预期为 JSON 数组或对象的数据类型。由于 encoding/json 将 nil 切片或映射序列化为 JSON null,这可能导致其他实现进行反序列化时出错。一项调查显示,大多数 Go 用户更倾向于默认将 nil 切片和映射序列化为空的 JSON 数组或对象。不区分大小写的反序列化:在反序列化时,JSON 对象成员名称通过不区分大小写的匹配解析为 Go 结构体字段名称。这是一个令人惊讶的默认设置,也是一个潜在的安全漏洞和性能限制。方法调用的不一致:由于实现细节的原因,在指针接收器上声明的 MarshalJSON 方法被 encoding/json 调用时存在不一致。虽然这被视为一个错误,但无法修复,因为太多的应用程序依赖于当前的行为。

API deficiencies The API of encoding/json can be tricky or restrictive: It is difficult to correctly unmarshal from an io.Reader. Users often write json.NewDecoder(r).Decode(v), which fails to reject trailing junk at the end of the input. Options can be set on the Encoder and Decoder types, but cannot be used with the Marshal and Unmarshal functions. Similarly, types implementing the Marshaler and Unmarshaler interfaces cannot make use of the options and there is no way to plumb options down the call stack. For example, the Decoder.DisallowUnknownFields option loses its effect when calling a custom UnmarshalJSON method. The Compact, Indent, and HTMLEscape functions write to a bytes.Buffer instead of something more flexible like a []byte or io.Writer. This limits the usability of those functions.

API 不足:encoding/json 的 API 可能很棘手或具有限制性:从 io.Reader 正确反序列化很困难。用户经常编写 json.NewDecoder(r).Decode(v),但这无法拒绝输入末尾的垃圾数据。选项可以在 Encoder 和 Decoder 类型上设置,但不能与 Marshal 和 Unmarshal 函数一起使用。同样,实现 Marshaler 和 Unmarshaler 接口的类型无法使用这些选项,也没有办法将选项传递到调用堆栈中。例如,Decoder.DisallowUnknownFields 选项在调用自定义 UnmarshalJSON 方法时会失效。Compact、Indent 和 HTMLEscape 函数写入的是 bytes.Buffer,而不是像 []byte 或 io.Writer 那样更灵活的对象。这限制了这些函数的可用性。

Performance limitations Setting aside internal implementation details, the public API commits it to certain performance limitations: MarshalJSON: The MarshalJSON interface method forces the implementation to allocate the returned []byte. Also, the semantics require that encoding/json verify that the result is valid JSON and also to reformat it to match the specified indentation. UnmarshalJSON: The UnmarshalJSON interface method requires that a complete JSON value be provided (without any trailing data). This forces encoding/json to parse the JSON value to be unmarshaled in its entirety to determine where it ends before it can call UnmarshalJSON. Afterwards, the UnmarshalJSON method itself must parse the provided JSON value again. Lack of streaming: Even though the Encoder and Decoder types operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory. The Decoder.Token method for reading individual tokens is allocation-heavy and there is no corresponding API for writing tokens. Furthermore, if the implementation of a MarshalJSON or UnmarshalJSON method recursively calls the Marshal or Unmarshal function, then the performance becomes quadratic.

性能限制:撇开内部实现细节不谈,公共 API 使其受到某些性能限制:MarshalJSON:MarshalJSON 接口方法强制实现分配返回的 []byte。此外,语义要求 encoding/json 验证结果是否为有效的 JSON,并根据指定的缩进重新格式化它。UnmarshalJSON:UnmarshalJSON 接口方法要求提供完整的 JSON 值(不能有任何尾随数据)。这迫使 encoding/json 在调用 UnmarshalJSON 之前,必须完整解析要反序列化的 JSON 值以确定其结束位置。之后,UnmarshalJSON 方法本身必须再次解析提供的 JSON 值。缺乏流式处理:尽管 Encoder 和 Decoder 类型在 io.Writer 或 io.Reader 上运行,但它们会将整个 JSON 值缓存在内存中。用于读取单个标记的 Decoder.Token 方法分配开销很大,并且没有相应的写入标记的 API。此外,如果 MarshalJSON 或 UnmarshalJSON 方法的实现递归调用 Marshal 或 Unmarshal 函数,性能就会变成二次方级别。

Trying to fix encoding/json directly Introducing a new, incompatible major version of a package is a heavy consideration. If possible, we should try to fix the existing package. While it is relatively easy to add new features, it is difficult to change existing features. Unfortunately, these problems are inherent consequences of the existing API, making them practically impossible to fix within the Go 1 compatibility promise. We could in prin

尝试直接修复 encoding/json:引入一个新的、不兼容的软件包主版本是一个重大的考量。如果可能,我们应该尝试修复现有的软件包。虽然添加新功能相对容易,但更改现有功能却很困难。不幸的是,这些问题是现有 API 的固有后果,使得在 Go 1 兼容性承诺内几乎不可能修复它们。我们原则上可以……