An Input-Dependent Readout Phase for Resonant Sequence Models

Sequence models with a fixed-size state avoid the quadratic cost of attention, but face a design choice. An input-dependent state transition gives content-dependent memory at the cost of FFT-based training, while a fixed transition keeps the O(T log T) convolutional path but reads memory the same way for every input.

具有固定大小状态的序列模型避免了注意力机制的二次方成本，但面临着设计选择。输入依赖的状态转换提供了内容相关的记忆，但代价是需要基于 FFT 的训练；而固定转换则保留了 O(T log T) 的卷积路径，但对每个输入都以相同的方式读取记忆。

ResonatorLM, a recent model of the second kind, reads a bank of damped complex oscillators through a fixed learned phase per head. We make that readout phase a function of the current token, which adds input dependence to the read only, keeps the convolution exact, and costs 0.2 percent more parameters.

ResonatorLM 是最近出现的一种此类模型，它通过每个头部的固定学习相位来读取一组阻尼复振荡器。我们将该读取相位设为当前标记的函数，这仅在读取过程中增加了输入依赖性，保持了卷积的精确性，且参数成本仅增加了 0.2%。

An analysis of the untrained bank shows that phase does encode elapsed time, but an ordinary token falls below the background within 2-8 steps and no choice of damping extends linear reconstruction beyond about 16 tokens.

对未经训练的振荡器组进行的分析表明，相位确实编码了经过的时间，但普通标记在 2-8 步内就会降至背景水平以下，且任何阻尼选择都无法将线性重构扩展到约 16 个标记之外。

In 6M-parameter character-level models on WikiText-2, the input-dependent phase matches the original at the training length and removes its failure beyond it: trained on 256 tokens and tested on 1024, perplexity falls from 12.81 to 3.76. We trace the failure to long-memory heads that accumulate more history than training exposed them to. The learned phase tracks the current character rather than distance and does not improve retrieval.

在 WikiText-2 上进行的 600 万参数字符级模型实验中，输入依赖相位在训练长度内与原始模型表现相当，并消除了超出该长度后的失效问题：在 256 个标记上训练并在 1024 个标记上测试时，困惑度从 12.81 降至 3.76。我们将这种失效归因于长记忆头部，它们积累的历史信息超过了训练期间所暴露的范围。学习到的相位跟踪的是当前字符而非距离，且并未改善检索效果。