The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models
Computer Science > Machine Learning arXiv:2610.08816 (cs) [Submitted on 24 Sep 2026] Title: The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models Authors: Yuheng Song.
计算机科学 > 机器学习 arXiv:2610.08816 (cs) [提交于 2026 年 9 月 24 日] 标题：长记忆的代价：序列模型中的状态、上下文与稳定性复杂度 作者：宋宇恒（音译）。
Abstract: Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately? We study this question directly in forecasting risk. For algebraically decaying predictive memory, we prove matching upper and lower approximation bounds for exponential and finite-state modes.
摘要：长程时间依赖性为序列模型提出了一个资源问题：对于给定的预测记忆定律，为了实现准确预测，需要多少状态、上下文或动力学临界性？我们直接在预测风险中研究了这个问题。对于代数衰减的预测记忆，我们证明了指数模式和有限状态模式的匹配上下近似界。
The best $r$-mode forecast error decays as $e^{-\Theta(\sqrt r)}$, so reaching forecast error $\tau$ needs $r=\Theta(\log^2(1/\tau))$ states or modes. Earlier curse-of-memory results establish broad limitations of stable recurrent models under different approximation notions; here both sides match for one canonical predictive target in forecast risk, which fixes the optimal resource exponent for that target.
最佳 $r$ 模式预测误差以 $e^{-\Theta(\sqrt r)}$ 的速度衰减，因此达到预测误差 $\tau$ 需要 $r=\Theta(\log^2(1/\tau))$ 个状态或模式。早期的“记忆诅咒”结果在不同的近似概念下确立了稳定循环模型的广泛局限性；在此，我们针对预测风险中的一个规范预测目标使双方匹配，从而确定了该目标的最佳资源指数。
We then show that genuine fractional long memory changes the geometry itself. In particular, forecast error is measured after fractional integration, prediction from a finite context of length $L$ has an exact $1/L$ leading order, and a fixed fractional strength $d$ keeps the square-log state-complexity law. Near the short-memory boundary, we identify the relevant $d^2$ and $d^4$ scales and give a uniform constructive law in the intermediate regime.
随后我们证明，真正的分数阶长记忆改变了几何结构本身。具体而言，预测误差是在分数阶积分后测量的，从长度为 $L$ 的有限上下文进行预测具有精确的 $1/L$ 主导阶，且固定的分数阶强度 $d$ 保持了平方对数状态复杂度定律。在短记忆边界附近，我们确定了相关的 $d^2$ 和 $d^4$ 尺度，并给出了中间区域的统一构造定律。
For nonlinear contextual recurrences with uniformly contractive state dynamics, we derive an exponential first-chaos envelope and an explicit necessary condition that relates forecast accuracy to the contraction margin. Vanishing forecasting error on an algebraic target forces the recurrence quantitatively toward criticality, a condition that is necessary and not by itself sufficient.
对于具有一致收缩状态动力学的非线性上下文循环，我们推导出了一个指数级的第一混沌包络，以及一个将预测准确度与收缩裕度相关联的显式必要条件。在代数目标上消除预测误差会从定量上迫使循环趋向临界状态，这是一个必要条件，但其本身并不充分。
Finite-sample Kullback—Leibler calculations further connect the predictive geometry to statistical information. Theorem-matched experiments with contractive state-space, gated recurrent, and attention models reproduce the state and stability predictions.
有限样本 Kullback—Leibler 计算进一步将预测几何与统计信息联系起来。通过收缩状态空间模型、门控循环模型和注意力模型进行的定理匹配实验，重现了关于状态和稳定性的预测。