5 ARB and ICU edge cases I wish I'd tested earlier

查看原文 / View Original 编辑 / Edit

5 ARB and ICU edge cases I wish I’d tested earlier

我希望早点测试的 5 个 ARB 和 ICU 边界情况

When I first added localization-file conversion, the job looked almost embarrassingly small: parse a file, copy the strings into a map, serialize that map in another format. Then I tried it on real Flutter projects. The converted files parsed correctly, which felt like success, but some of them had quietly lost translator context. Others kept the text and damaged a placeholder. ICU plurals were the most misleading case: a message could be perfectly valid and still be wrong for the target language. These are the five fixtures I now reach for before I trust a converter.

当我最初添加本地化文件转换功能时，这项工作看起来简单得令人尴尬：解析文件，将字符串复制到映射（map）中，然后以另一种格式序列化该映射。然而，当我将其应用到真实的 Flutter 项目中时，问题出现了。转换后的文件虽然解析正确，看起来像是成功了，但其中一些文件悄无声息地丢失了翻译上下文。另一些文件虽然保留了文本，却损坏了占位符。ICU 复数形式是最具误导性的情况：一条消息可能完全合法，但对于目标语言来说却是错误的。在信任一个转换器之前，我现在会先用以下五个测试用例进行验证。

1. @key is not another string

1. @key 并非普通字符串

An ARB message can have a companion metadata entry: ARB 消息可以包含配套的元数据条目：

{
  "welcome": "Welcome, {name}!",
  "@welcome": {
    "description": "Greeting shown after sign-in",
    "placeholders": {
      "name": {
        "type": "String"
      }
    }
  }
}

welcome is the text. @welcome tells Flutter and the translator what that text means. My first parser skipped every key beginning with @. That was convenient because metadata should not appear in the translation list. It also meant the conversion was not lossless. A round trip restored the greeting but forgot its description and the type of name. The model I use internally now looks more like this: welcome 是文本内容，而 @welcome 则告诉 Flutter 和翻译人员该文本的含义。我最初的解析器跳过了所有以 @ 开头的键。这很方便，因为元数据不应该出现在翻译列表中。但这也意味着转换过程并非无损的。一次往返转换（round trip）虽然恢复了问候语，却丢失了描述信息和 name 的类型。我现在内部使用的模型看起来更像这样：

id: "welcome"
value: "Welcome, {name}!"
description: "Greeting shown after sign-in"
placeholders: ["name"]

That does not magically solve export. CSV, for example, may have nowhere to put a placeholder type. The important part is knowing what was lost. Depending on the target format, I can keep metadata in comments, write a sidecar file, or show a warning. I also test @missing without a matching missing message. Orphan metadata looks odd, but it turns up in repositories after keys are renamed or deleted.

这并不能神奇地解决导出问题。例如，CSV 格式可能没有地方存放占位符类型。关键在于要清楚丢失了什么。根据目标格式的不同，我可以将元数据保留在注释中、写入辅助文件（sidecar file）或显示警告。我还会测试没有对应消息的 @missing。孤立的元数据看起来很奇怪，但在键名被重命名或删除后，它们经常会出现在代码库中。

2. A valid plural can still be wrong for the locale

2. 合法的复数形式在特定语言环境下仍可能是错误的

This is enough for most English cardinal plurals: 对于大多数英语基数复数形式，这样写就足够了：

{count, plural, one {# file} other {# files} }

Using the same branches for Russian loses information: 对俄语使用相同的分支会丢失信息：

{count, plural, one {# файл} few {# файла} many {# файлов} other {# файла} }

English normally needs one and other. Russian uses one, few, many, plus the required other fallback. Arabic may use all six named categories: zero, one, two, few, many, and other. So I treat validation as two separate questions: Does the ICU message parse? Does it cover the categories used by this locale? Checking only the first question catches unbalanced braces and a missing other, but it will not tell you that the Russian few branch is absent. 英语通常只需要 one 和 other。俄语则使用 one、few、many，外加必要的 other 回退项。阿拉伯语可能使用全部六个命名类别：zero、one、two、few、many 和 other。因此，我将验证视为两个独立的问题：ICU 消息能否解析？它是否涵盖了该语言环境使用的所有类别？仅检查第一个问题可以捕获括号不匹配和缺少 other 的情况，但它无法告诉你俄语的 few 分支缺失了。

Exact selectors need to survive as well: 精确选择器也必须被保留：

{count, plural, =0 {No files} one {One file} other {# files} }

=0 is an exact-number override. It is not the same thing as the locale category zero, and a converter should not merge the two. Then there is nesting: =0 是精确数字覆盖。它与语言环境类别 zero 不同，转换器不应将两者合并。此外还有嵌套情况：

{gender, select,
  female {{count, plural, one {She has one task} other {She has # tasks}}}
  male {{count, plural, one {He has one task} other {He has # tasks}}}
  other {{count, plural, one {They have one task} other {They have # tasks}}}
}

This was the point where I stopped trying to count braces with regular expressions. 这就是我放弃尝试用正则表达式来计算括号数量的时刻。

3. One value has to satisfy both JSON and ICU

3. 一个值必须同时满足 JSON 和 ICU 的要求

ARB is JSON, but its string values may contain another language: ICU MessageFormat. The converter has to get through both layers without “helping” too much. ARB 是 JSON 格式，但其字符串值可能包含另一种语言：ICU MessageFormat。转换器必须穿透这两层，且不能“过度帮忙”。

{
  "receipt": "Line 1\nLine 2: \"{total}\"",
  "path": "C:\\Exports\\{locale}\\messages.json",
  "emoji": "Saved ✓ 🚀"
}

After JSON parsing, receipt contains a real newline. path contains backslashes and an ICU-style placeholder. The last value is a quick check that the pipeline is genuinely Unicode-safe. The bugs here are small and annoying: \n becomes the visible characters \ and n; an escaped backslash is escaped a second time; UTF-8 is saved with an unexpected byte-order mark; ICU apostrophe quoting changes in one direction of the conversion; Unicode normalization changes a value that looks identical on screen. I compare decoded values, not their JSON spelling. These two strings represent the same character: "\u2713" and "✓". Byte equality would fail that test for no useful reason. On the other hand, a changed non-breaking space can look fine in a diff and still break the UI. JSON 解析后，receipt 包含一个真正的换行符。path 包含反斜杠和一个 ICU 风格的占位符。最后一个值是快速检查流水线是否真正支持 Unicode 的手段。这里的 Bug 很小但很烦人：\n 变成了可见字符 \ 和 n；转义的反斜杠被二次转义；UTF-8 保存时带有意外的字节顺序标记（BOM）；ICU 的撇号引用在转换的某个方向上发生改变；Unicode 规范化改变了一个在屏幕上看起来完全相同的值。我比较的是解码后的值，而不是它们的 JSON 书写形式。这两个字符串代表同一个字符："\u2713" 和 "✓"。字节相等性测试会毫无意义地失败。另一方面，一个被改变的不换行空格在差异对比（diff）中看起来可能没问题，但却会破坏 UI。

4. Not every brace is a placeholder

4. 并非每个括号都是占位符

Localization projects rarely agree on one placeholder dialect: 本地化项目很少在占位符语法上达成一致：

  • Hello, {name}
  • Downloaded %s of %d files
  • You have :count messages
  • Welcome, {{user}}
  • Price: %(amount)s

Unless the user explicitly asks for a syntax migration, I leave those tokens alone. I also compare them as a multiset rather than a set: 除非用户明确要求进行语法迁移，否则我不会改动这些标记。我还会将它们作为多重集（multiset）而非集合（set）进行比较：

  • input: ["{name}", "{count}", "{count}"]
  • output: ["{name}", "{count}", "{count}"]

The repeated {count} matters. The opposite problem is detecting placeholders where there are none: 重复的 {count} 很重要。相反的问题是，在没有占位符的地方检测出占位符：

  • CSS: .button { color: red; }
  • Discount: 20%
  • Object: {"id": 42}

A single broad regular expression will eventually eat ordinary text. Format-aware parsing is better. If that is not possible, I would rather ask which placeholder dialect a file uses than silently guess. ARB gives us one extra consistency check. Tokens in the message should agree with the placeholders object in @key. If the text contains {name} but the metadata only declares count, that deserves a warning. 单一的宽泛正则表达式最终会误伤普通文本。格式感知解析（Format-aware parsing）更好。如果无法做到这一点，我宁愿询问文件使用了哪种占位符语法，也不愿默默猜测。ARB 为我们提供了一个额外的一致性检查：消息中的标记应与 @key 中的 placeholders 对象一致。如果文本包含 {name} 但元数据只声明了 count，这就应该触发警告。

5. “It parses” is a very low bar for a round trip

5. “能解析”对于往返转换来说是一个极低的门槛

Consider this ARB file: 考虑这个 ARB 文件：

{
  "@@locale": "en",
  "save": "Save",
  "@save": {
    "description": "Button label, not a noun"
  }
}

A simple CSV export might be: 一个简单的 CSV 导出可能是：

key,value
save,Save

The CSV is valid. Turning it back into this ARB is also valid: CSV 是合法的。将其转回 ARB 也是合法的：

{
  "save": "Save"
}

But the locale and the translator’s context have disappeared. For a round-trip test, I compare message IDs, decoded values, placeholder counts, ICU selectors, metadata, and locale information. I do not require the same key order, whitespace, or Unicode escape style. When the target format cannot represent something, I want the result described plainly: lossless, lossy (with a reason), or unsupported. Returning a green checkmark merely because the output parses is how quiet data loss reaches a production translation file. The small fixture I keep around This one file covers most of the cases above: 但语言环境和翻译上下文已经消失了。对于往返测试，我会比较消息 ID、解码后的值、占位符数量、ICU 选择器、元数据和语言环境信息。我不要求键的顺序、空格或 Unicode 转义风格完全一致。当目标格式无法表示某些内容时，我希望结果能被清晰地描述：无损、有损（并说明原因）或不支持。仅仅因为输出能被解析就返回一个绿色的勾选标记，正是静默数据丢失进入生产翻译文件的方式。我保留的这个小测试用例涵盖了上述大多数情况：

{
  "@@
← 返回新闻列表 / Back to news list