baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity
baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity
Computer Science > Computation and Language arXiv:2610.30154 (cs) [Submitted on 23 Aug 2026] Title: baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity Authors: Linke Zhang.
计算机科学 > 计算与语言 arXiv:2610.30154 (cs) [提交于 2026 年 8 月 23 日] 标题：baseVultra：具有重分配容量的紧凑型非线性递归。作者：Linke Zhang。
Abstract: Nonlinear recurrent networks remain useful for streaming sequence modeling, yet gated cell design is still dominated by LSTM and GRU. We study whether recurrent computation can be reallocated away from gate transforms and toward candidate generation. We propose baseVultra, a compact recurrent cell that concentrates the dense recurrent budget on a single H x H candidate-generation matrix, while replacing recurrent gate matrices with O(H) diagonal interactions.
摘要：非线性递归网络在流式序列建模中仍然非常有用，但门控单元设计仍由 LSTM 和 GRU 主导。我们研究了递归计算是否可以从门变换中重新分配，转向候选生成。我们提出了 baseVultra，这是一种紧凑的递归单元，它将密集的递归预算集中在单个 H x H 候选生成矩阵上，同时用 O(H) 对角交互替换递归门矩阵。
The candidate update combines this dense branch with two low-rank auxiliary branches, an EMA cache, and learnable output blending; the gates additionally receive lightweight second-order access to h_{t-2} through diagonal terms. On TinyShakespeare character-level language modeling, baseVultra achieves a validation loss of 1.594 in the compact one-layer 200K setting, compared with 1.604 for a parameter-matched LSTM and 1.616 for a parameter-matched GRU, while its dense recurrent matrix-vector computation is about 54% of GRU under matched hidden size for this configuration.
候选更新将此密集分支与两个低秩辅助分支、一个 EMA 缓存和可学习的输出混合相结合；门控单元还通过对角项获得了对 h_{t-2} 的轻量级二阶访问。在 TinyShakespeare 字符级语言建模中，baseVultra 在紧凑的单层 200K 设置下实现了 1.594 的验证损失，相比之下，参数匹配的 LSTM 为 1.604，参数匹配的 GRU 为 1.616，而在此配置的匹配隐藏层大小下，其密集递归矩阵向量计算量约为 GRU 的 54%。
On text8, a roughly 45x larger benchmark, baseVultra is competitive with full-gate cells (1.854 bpc vs. 1.849 for GRU and 1.837 for LSTM). On formal-language probes, the largest gains appear under constrained hidden-state capacity (0.991 vs. 0.761 GRU and 0.691 LSTM OOD2 accuracy on Counter at H=16) and vanish as hidden size grows. These results characterize where and when candidate-focused recurrent capacity allocation helps: the benefit is concentrated in compact regimes, while full-gate cells remain competitive at scale.
在规模大出约 45 倍的基准测试 text8 上，baseVultra 与全门控单元具有竞争力（1.854 bpc，而 GRU 为 1.849，LSTM 为 1.837）。在形式语言探测中，最大的增益出现在受限的隐藏状态容量下（在 H=16 时，Counter 任务的 OOD2 准确率为 0.991，而 GRU 为 0.761，LSTM 为 0.691），并随着隐藏层大小的增加而消失。这些结果刻画了以候选为中心的递归容量分配在何处以及何时有效：其优势集中在紧凑的方案中，而全门控单元在大规模下仍保持竞争力。