本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

StateBench: Testing how language models track state changes over long documents

StateBench: Testing how language models track state changes over long documents. We’re introducing StateBench, a benchmark for long-context state tracking—how well language models keep track of information about entities in long documents as they evolve. State tracking underlies tasks from reading code, where variables are assigned and reassigned, to following a story as characters physically move around and objects change hands.

StateBench：测试语言模型如何跟踪长文档中的状态变化。我们推出了 StateBench，这是一个用于长上下文状态跟踪的基准测试，旨在评估语言模型在长文档中跟踪实体演变信息的能力。状态跟踪是许多任务的基础，从阅读代码（变量被不断赋值和重新赋值）到跟随故事发展（角色移动位置、物品易手），都离不开它。

For example, if a mystery novel states that “John gave Mary the knife” right before a murder occurs, it’s important to know that the state of each character has changed. However, not all state-tracking problems are created equal: some require only finding the latest update; harder ones require following a chain of changes, where one missed step can throw off the final answer. For example, a harder variant of our mystery state tracking would be if John gave Mary the item in his backpack, with it previously known that Edward put a knife in John’s backpack.

例如，如果一部悬疑小说在谋杀发生前写道“约翰把刀给了玛丽”，那么了解每个角色的状态已经改变就非常重要。然而，并非所有的状态跟踪问题难度都相同：有些问题只需要找到最新的更新；而较难的问题则需要追踪一系列的变化，其中任何一个步骤的遗漏都可能导致最终答案错误。例如，我们悬疑状态跟踪的一个更难变体是：如果约翰把背包里的物品给了玛丽，而此前已知爱德华曾把一把刀放进约翰的背包里。

Transformers have well-documented theoretical limits on hard variants of state tracking with long dependency chains, helping motivate recurrent and hybrid architectures such as Gated DeltaNet that carry a running memory forward. Existing state-tracking tests tend to focus on the hardest cases, but as these designs become more common, researchers need to know which kinds of state tracking each architecture can handle. We would like to understand the differences between architectures not just on the hardest kinds of state tracking, but also on the simpler, potentially more common kinds.

Transformer 在处理具有长依赖链的复杂状态跟踪变体时，其理论局限性已有充分记录，这促使人们研究循环和混合架构（如 Gated DeltaNet），这些架构能够携带运行内存向前推进。现有的状态跟踪测试往往侧重于最困难的情况，但随着这些设计变得越来越普遍，研究人员需要了解每种架构能够处理哪种类型的状态跟踪。我们希望不仅在最难的状态跟踪任务上，而且在更简单、可能更常见的任务上，理解不同架构之间的差异。

To this end, we developed the StateBench benchmark to capture state-tracking problems of different complexity, which we used to evaluate the differences between language models with different architectures. We find that recurrent and hybrid models can generalize much farther than transformers when trained directly on these tasks, while frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna still break down on harder forms of state tracking as inputs grow longer.

为此，我们开发了 StateBench 基准测试，以捕捉不同复杂性的状态跟踪问题，并利用它来评估不同架构语言模型之间的差异。我们发现，当直接在这些任务上进行训练时，循环和混合模型的泛化能力远超 Transformer；而包括 Gemini 3.8 Flash、Claude Opus 5.5 以及 GPT-6 Sol 和 Luna 在内的前沿模型，随着输入长度的增加，在处理较难的状态跟踪形式时仍然会失效。

Four levels of state tracking: StateBench consists of four difficulty levels that permit increasingly complex state update operations. For programmers, these levels of state tracking complexity can be understood as allowing different types of variable assignment operations: R-trivial problems use immutable assignment. A variable’s value never changes, so the model only needs to recover what it was first set to. In code, this looks like let x = 4; in everyday life, it’s like a birth date—it stays fixed.

四种状态跟踪级别：StateBench 包含四个难度级别，允许进行日益复杂的状态更新操作。对于程序员来说，这些状态跟踪复杂性级别可以理解为允许不同类型的变量赋值操作：R-trivial 问题使用不可变赋值。变量的值永远不会改变，因此模型只需要恢复它最初被设置的值。在代码中，这类似于 let x = 4;；在日常生活中，它就像出生日期一样——是固定不变的。

Aperiodic problems allow reassignment by value. The model needs to find the most recent value and can ignore earlier ones. In code, this looks like x = 4 followed later by x = 2; in everyday life, it’s like a thermostat, where the latest setting determines the current temperature. Soluble problems add self-referential updates. Each update builds on the current value, so the model has to carry changes forward rather than simply find the latest assignment. In code, this looks like x = x + 5; in everyday life, it’s like a game score, where each point adds to the running total.

Aperiodic 问题允许按值重新赋值。模型需要找到最近的值，并可以忽略之前的值。在代码中，这类似于先执行 x = 4 随后执行 x = 2；在日常生活中，它就像恒温器，最新的设置决定了当前的温度。Soluble 问题增加了自引用更新。每次更新都基于当前值，因此模型必须向前传递变化，而不是简单地找到最新的赋值。在代码中，这类似于 x = x + 5；在日常生活中，它就像游戏得分，每一分都会增加到总分中。

General finite problems add assignment by reference. One variable can take on another variable’s current value, creating dependencies across updates. In code, this looks like x = y or swapping two variables. Imagine a magician hiding a ball under one of five cups, then repeatedly swapping their positions. Finding the ball means keeping track of how those swaps change its location. This illustrates the S₅ state-tracking problem: following how successive swaps rearrange five objects. It belongs to the hardest of our four levels and is discussed in previous work on the limits of state tracking.

General finite 问题增加了引用赋值。一个变量可以获取另一个变量的当前值，从而在更新之间产生依赖关系。在代码中，这类似于 x = y 或交换两个变量。想象一个魔术师把球藏在五个杯子中的一个下面，然后反复交换它们的位置。找到球意味着要跟踪这些交换如何改变它的位置。这说明了 S₅ 状态跟踪问题：跟踪连续的交换如何重新排列五个对象。它属于我们四个级别中最难的一个，在之前关于状态跟踪局限性的研究中已有讨论。

These levels can be intuitively understood in terms of variable assignments, but they also represent theoretically motivated distinctions in state tracking rooted in Krohn–Rhodes theory, a branch of algebra that describes how state-tracking problems – which can be viewed as word problems over finite monoids – can be decomposed into atomic primitives. Each level in our hierarchy corresponds to a class of state-tracking problems that can be built out of a different set of primitives.

这些级别可以根据变量赋值直观地理解，但它们也代表了状态跟踪中具有理论依据的区别，这些区别植根于 Krohn–Rhodes 理论。这是代数的一个分支，描述了状态跟踪问题（可以看作有限幺半群上的字问题）如何分解为原子基元。我们层级结构中的每个级别都对应一类状态跟踪问题，这些问题可以由不同的一组基元构建而成。

To compare the four levels fairly, we start with a general finite sequence of variable assignments and derive three easier versions from it, producing four comparable StateBench problems—one at each difficulty level. All four use the same number of variables and distinct values, with assignments appearing at the same points in the sequence; only the type of assignment changes.

为了公平地比较这四个级别，我们从一个通用的有限变量赋值序列开始，并从中推导出三个更简单的版本，从而产生四个可比较的 StateBench 问题——每个难度级别一个。所有四个问题都使用相同数量的变量和不同的值，赋值出现在序列中的相同位置；只有赋值的类型发生了变化。

What we found: Pretrained models. We tested a range of open pretrained models on the r-trivial, aperiodic, and general finite levels of StateBench to see where their state tracking breaks down, and whether different architectures fail where theory predicts. Under standard theoretical assumptions, transformers can handle r-trivial robustly (and soluble and aperiodic to some degree) but are predicted to fail on general problems. In contrast, Gated DeltaNet with a more flexible memory update can, in principle, handle all four levels.

我们的发现：预训练模型。我们在 StateBench 的 r-trivial、aperiodic 和 general finite 级别上测试了一系列开源预训练模型，以观察它们的状态跟踪在何处失效，以及不同架构是否在理论预测的地方失败。在标准的理论假设下，Transformer 可以稳健地处理 r-trivial（以及在一定程度上处理 soluble 和 aperiodic），但预计在 general 问题上会失败。相比之下，具有更灵活内存更新的 Gated DeltaNet 原则上可以处理所有四个级别。

We started with open pretrained models, where the training data and context lengths are fully documented. These included two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), along with smaller models with 275 million and 2.7 billion parameters. All of these models were pretrained on the same data; none had seen contexts longer than 64,000 tokens during training, and we tested most of them on inputs well beyond that length. When we scored each model, almost all performed well on r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Problem format mattered here—th

我们从训练数据和上下文长度有完整记录的开源预训练模型开始。其中包括两个 70 亿参数的模型：Olmo 3（Transformer）和 Olmo Hybrid（增加了 Gated DeltaNet 层），以及参数量分别为 2.75 亿和 27 亿的较小模型。所有这些模型都在相同的数据上进行了预训练；在训练过程中，没有一个模型见过超过 64,000 个 token 的上下文，而我们测试它们时，大多数输入的长度都远超这一限制。当我们对每个模型进行评分时，几乎所有模型在 r-trivial 问题上表现良好，在 aperiodic 问题上随着输入增长准确率下降，而在 general finite 问题上则处于随机猜测水平。问题格式在这里很重要——th