Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States
Computer Science > Machine Learning arXiv:2610.20637 (cs) [Submitted on 2 Oct 2026] Title: Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States Authors: Zicong Huang.
计算机科学 > 机器学习 arXiv:2610.20637 (cs) [提交于 2026 年 10 月 2 日] 标题：扩展注意力：一种通过泰勒级数实现具有固定循环状态数量的线性时间 Transformer 的路径 作者：Zicong Huang。
Abstract: We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one outer product per token and independent of context length $N$.
摘要：我们将注意力的指数核展开为 $n$ 项的截断泰勒级数，并将每一项分解为张量幂内积，这使得因果历史被压缩为精确的 $n$ 个循环状态，每个状态在每个 token 处通过一个秩一外积进行更新，且与上下文长度 $N$ 无关。
The recursion is lossless with respect to the truncated kernel ($2.2 \times 10^{-16}$ in float64), and the states are additive across chunks ($5.7 \times 10^{-14}$), so prefill admits a parallel scan.
该递归相对于截断核是无损的（float64 下为 $2.2 \times 10^{-16}$），且状态在分块间具有可加性（$5.7 \times 10^{-14}$），因此预填充（prefill）支持并行扫描。
Measured exponents are $t \propto N^{2.28}$ for softmax against $N^{0.91}$ for states, with a crossover at $N \approx 728$ and $20.8\times$ at $N = 8192$; per-token decoding latency is constant at $7.3,\mu$s against $N^{1.04}$ for softmax, giving $76.0\times$ at $N = 65536$.
测量得到的指数显示，softmax 的时间复杂度为 $t \propto N^{2.28}$，而状态方法为 $N^{0.91}$，在 $N \approx 728$ 时出现交叉点，在 $N = 8192$ 时快 20.8 倍；每个 token 的解码延迟在 $7.3,\mu$s 时保持恒定，而 softmax 为 $N^{1.04}$，在 $N = 65536$ 时快 76.0 倍。
Accuracy improves exponentially in $n$ while cost grows combinatorially, since the $m$-th state is $\binom{d+m-1}{m}$ wide---a bound confirmed by SVD---so the optimum is second order, $n = 3$, at $6.5 \times 10^{-4}$ error and $9{,}945$ elements per head.
准确率随 $n$ 指数级提升，而成本呈组合级增长，因为第 $m$ 个状态的宽度为 $\binom{d+m-1}{m}$（该界限已通过 SVD 证实），因此最优解为二阶，$n = 3$，误差为 $6.5 \times 10^{-4}$，每个头包含 $9{,}945$ 个元素。
Query and key normalization is a prerequisite: without it the argument exceeds the convergence radius and adding terms stops helping.
查询（Query）和键（Key）归一化是先决条件：若无此步骤，参数将超出收敛半径，增加项数将不再有帮助。