Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does

上下文窗口不知道什么是“仍然正确”的——我构建了一个能够识别有效性的层

TL;DR: I built a working benchmark for this in pure Python. No APIs, no LLMs, just a deterministic setup with real numbers and a runnable repo. 简而言之：我用纯 Python 构建了一个可运行的基准测试。没有 API，没有大模型，只是一个带有真实数据和可运行仓库的确定性设置。

The basic problem is simple. A context window remembers what happened. It does not know whether that information is still valid. 基本问题很简单。上下文窗口能记住发生了什么，但它不知道这些信息是否仍然有效。

So I built two deterministic executors. They do the exact same work. One checks whether a dependency is still valid before acting. The other only finds out after the action fails. 所以我构建了两个确定性执行器。它们执行完全相同的工作。一个在行动前检查依赖项是否仍然有效，另一个只有在行动失败后才会发现问题。

That small difference shows up in the numbers. The second executor does work that was already doomed. When your resource budget is tight, that wasted work is enough to fail the whole task. 这种微小的差异体现在数据上。第二个执行器做了注定失败的工作。当你的资源预算紧张时，这些浪费的工作足以导致整个任务失败。

I got one of my main assumptions wrong along the way. I thought graph shape would drive the wasted work. I ran a 96-configuration sweep, and it disproved my guess. Size was the real driver. 在这个过程中，我的一个主要假设错了。我原以为图的形状会影响浪费的工作量。我运行了 96 种配置的扫描，结果推翻了我的猜测。规模才是真正的驱动因素。

I updated the experiment instead of forcing my original hypothesis to fit the data. The corrected result was more precise and led to the next experiment. 我更新了实验，而不是强行让原始假设去迎合数据。修正后的结果更加精确，并引导我进行了下一个实验。

···

Same price change, two outcomes. The baseline keeps going and only notices at the moment it acts; the validity-aware executor checks first and replans immediately. 同样的价格变动，两种结果。基准执行器继续运行，直到行动时才发现问题；而具备有效性意识的执行器会先检查，并立即重新规划。

This is the diagram I built to show what’s actually different between the two executors in this benchmark, without any numbers yet. A flight price is $420, then a plan gets made around that price, then the price jumps to $610. From there the diagram splits into two branches. The baseline branch just keeps executing like nothing happened. It doesn’t have any way of checking whether its assumptions are still good, so it walks straight into the stale plan and only discovers the problem when it actually tries to act and the action fails. By then it’s already burned a step it didn’t need to. The validity-aware branch does one extra thing before acting: it checks whether the price it’s relying on is still valid. Since it isn’t, it catches that immediately and goes straight to replanning, without ever touching the doomed action. I drew this before writing a single benchmark number, because I wanted the mechanism to be obvious on its own before I started asking anyone to trust a chart. Everything after this image is just putting numbers on the gap between these two paths. 这是我绘制的图表，用于展示该基准测试中两个执行器的实际区别（暂未包含数据）。机票价格为 420 美元，随后围绕该价格制定了计划，接着价格跳涨至 610 美元。从这里开始，图表分为两个分支。基准分支像什么都没发生一样继续执行。它没有办法检查其假设是否仍然有效，因此它直接执行过时的计划，只有在尝试行动且行动失败时才发现问题。到那时，它已经浪费了一个不必要的步骤。具备有效性意识的分支在行动前多做了一件事：检查它所依赖的价格是否仍然有效。由于价格已变，它立即捕捉到这一点并直接重新规划，根本不会触碰那个注定失败的行动。我在写下任何基准数据之前就画好了这张图，因为我希望在让任何人相信图表之前，机制本身就足够直观。这张图之后的所有内容，只是为了给这两条路径之间的差距量化数据。

A context window can remember everything about the past while giving an agent the wrong picture of the present. 上下文窗口可以记住关于过去的一切，同时却给智能体提供关于现在的错误图景。

This is easy to miss because it does not look like a memory failure. Nothing was forgotten. No text was cut off. No logs showed a missing record. The fact that drove the wrong decision was right there in the window. It just stopped being true three minutes ago, and nothing in the baseline executor was checking for that change. 这一点很容易被忽略，因为它看起来不像内存故障。没有任何东西被遗忘，没有文本被截断，也没有日志显示记录丢失。导致错误决策的事实就在窗口里。它只是在三分钟前不再正确了，而基准执行器中没有任何机制在检查这种变化。

Take a basic case: 以一个基本案例为例： 10:00: Flight A costs $420. 10:00：航班 A 价格 420 美元。 10:01: The plan is to book Flight A. 10:01：计划预订航班 A。 10:03: Flight A jumps to $610. 10:03：航班 A 价格跳涨至 610 美元。 10:04: The system still books based on the $420 plan. 10:04：系统仍然基于 420 美元的计划进行预订。

If you have ever debugged an agent that confidently followed a plan built on a broken assumption, this is one failure mode you may have encountered. It is not context loss. It is context that stays around long after it stops being valid. 如果你曾经调试过一个自信地遵循基于错误假设的计划的智能体，这可能是你遇到过的一种故障模式。这不是上下文丢失，而是上下文在失效后依然长期存在。

I spent the last few weeks building a benchmark to measure what this actually costs in wasted steps and failed tasks. I also wanted to see if tracking validity, instead of just keeping facts in memory, fixes the problem. 过去几周我一直在构建一个基准测试，以衡量这在浪费步骤和失败任务方面带来的实际成本。我还想看看，追踪有效性（而不是仅仅将事实保存在内存中）是否能解决这个问题。

To be clear about the title: “knows when context goes stale” does not mean the system predicts the future. It simply means the system checks if a fact is still valid right before an action relies on it. It re-verifies the assumption instead of moving forward blindly. 明确一下标题的含义：“知道上下文何时过时”并不意味着系统能预测未来。它仅仅意味着系统在行动依赖某个事实之前，会检查该事实是否仍然有效。它重新验证了假设，而不是盲目地向前推进。

···

Presence Is Not the Same as Validity

存在并不等同于有效

Most talk about context windows focuses on space. People worry about having too much context, not enough context, or loading it in the wrong order. Those are real problems. But they are not the problem here. This issue is all about time. 关于上下文窗口的大多数讨论都集中在空间上。人们担心上下文太多、太少，或者加载顺序错误。这些都是真实存在的问题，但不是这里的问题。这个问题完全关乎时间。

A context window is basically a transcript. It records what happened in order. A transcript tells an agent about the past. A validity layer tells the agent if that information is still safe to use right now. A normal context window cannot do that second job. It just was not built for it. 上下文窗口本质上是一份记录。它按顺序记录了发生的事情。记录告诉智能体关于过去的信息。而有效性层告诉智能体这些信息现在是否仍然可以安全使用。普通的上下文窗口无法完成第二项工作，它天生就不是为此而设计的。

To make this concrete, I gave every fact in the benchmark one of four states instead of a simple true or false: 为了具体化，我在基准测试中为每个事实赋予了四种状态之一，而不是简单的真或假：

• ACTIVE: Current evidence supports it. ACTIVE（活跃）：当前证据支持它。
• STALE: It was true once, but newer data exists. STALE（过时）：它曾经是真的，但现在有了更新的数据。
• SUPERSEDED: A newer observation completely replaced it. SUPERSEDED（被取代）：更新的观察结果完全取代了它。
• UNKNOWN: There is not enough evidence to say either way. UNKNOWN（未知）：没有足够的证据来判断真假。

That fourth state matters a lot. Keeping UNKNOWN separate from an outright failure gives the executor a third choice. It can verify the fact before acting or making a new plan. This lets the benchmark measure the real cost of uncertainty. It stops treating every doubt as a hard failure, which ended up being one of the most interesting results. 第四种状态非常重要。将 UNKNOWN 与彻底的失败区分开来，给了执行器第三种选择。它可以在行动或制定新计划之前验证事实。这使得基准测试能够衡量不确定性的真实成本。它不再将每一个疑点都视为硬性失败，这最终成为了最有趣的结果之一。

There is another distinction worth calling out because it drives the more complex failures in this benchmark. We need to separate factual invalidity from operational invalidity. 还有另一个值得指出的区别，因为它导致了该基准测试中更复杂的故障。我们需要将“事实性失效”与“操作性失效”区分开来。

A fact is factually invalid when it simply becomes false. The flight price is a clean case. It was $420, and now it is not. That is easy to spot. It is what most people picture when they think of stale data. 当一个事实变得不真实时，它就是“事实性失效”。机票价格就是一个清晰的例子。它曾经是 420 美元，现在不是了。这很容易发现。这也是大多数人在想到过时数据时所想到的情况。

A fact is operationally invalid when it is still true but can no longer support a decision. Imagine a database holds exactly 10,000 records. That number has not changed. But the database itself just went offline. The record count did not become false. It just became completely useless. Truth and usability are different things. That distinction matters here because a fact can remain true while the dependency that makes it usable has failed. 当一个事实仍然真实但无法再支持决策时，它就是“操作性失效”。想象一个数据库正好有 10,000 条记录。这个数字没有变，但数据库本身离线了。记录数量并没有变假，它只是变得完全没用了。真实性和可用性是两码事。这种区别在这里很重要，因为一个事实可能依然真实，但使其可用的依赖项却已经失效了。

This is why a validity system cannot just slap a timestamp on every log entry. A timestamp only tells you how old something is. It does not tell you if the system that made it usable is still working. An agent that only checks timestamps will happily act on a fresh record count from a broken database. Nothing about the count’s timestamp changed. The actual break happened somewhere else in the dependency. 这就是为什么有效性系统不能仅仅在每个日志条目上贴个时间戳。时间戳只能告诉你某样东西有多旧，它不能告诉你使其可用的系统是否仍在工作。一个只检查时间戳的智能体，会乐意基于一个已损坏数据库中“新鲜”的记录数量进行操作。记录的时间戳没有任何变化，但实际的故障发生在依赖链的其他地方。