Wiola 13M: A Gated Spiral Attention Architecture for Parameter Efficient Small Language Models

Wiola 13M：一种用于参数高效型小语言模型的门控螺旋注意力架构

Small language models in the ten to one hundred million parameter range are attractive for on-device inference, rapid experimentation, and controlled scientific study, yet most of them reuse the standard transformer block without adaptation to the small scale regime. 参数量在千万到一亿级别的小型语言模型在端侧推理、快速实验和受控科学研究方面极具吸引力，然而大多数模型仍沿用标准的 Transformer 模块，并未针对小规模场景进行适配。

We present Wiola, a decoder-only language model whose novelty is concentrated in three drop-in components of every layer. 我们提出了 Wiola，这是一种仅解码器（decoder-only）的语言模型，其创新点集中在每一层的三个即插即用组件中。

First, Spiral Rotary Positional Encoding perturbs the standard rotary frequencies by a slowly growing per-dimension factor so that phase trajectories fan outward, improving long-range discrimination while adding no parameters. 首先，“螺旋旋转位置编码”（Spiral Rotary Positional Encoding）通过一个随维度缓慢增长的因子来扰动标准的旋转频率，使相位轨迹向外发散，从而在不增加参数的情况下提高了长距离辨别能力。

Second, Gated Spiral Attention introduces a per-head, content-adaptive scalar gate derived from a causal cumulative statistic of the query stream, providing an implicit and differentiable form of soft head selection at negligible cost. 其次，“门控螺旋注意力”（Gated Spiral Attention）引入了一种基于查询流因果累积统计量的逐头（per-head）、内容自适应标量门控，以极低的成本提供了一种隐式且可微的软头选择机制。

Third, the Butterfly feed-forward block replaces the conventional expansion layer with a multiplicative interaction and an intra-block bypass path, matching the parameter count of a four-times gated linear unit block while improving gradient flow in shallow stacks. 第三，“蝴蝶前馈块”（Butterfly feed-forward block）用乘法交互和块内旁路路径取代了传统的扩展层，在匹配四倍门控线性单元（GLU）块参数量的同时，改善了浅层堆叠中的梯度流动。

We formalize each component, derive exact parameter and computation budgets, and prove that the gated attention admits an exact and numerically verified equivalence between full sequence training and cached autoregressive decoding, so that no approximation is introduced at inference time. 我们对每个组件进行了形式化定义，推导了精确的参数和计算预算，并证明了该门控注意力机制在全序列训练和缓存自回归解码之间存在精确且经数值验证的等价性，从而确保在推理阶段不会引入任何近似误差。

We also describe a fully reproducible training and evaluation protocol on a standard tiny story corpus. The reference implementation is released as an open-source package with weights ready for publishing support. 我们还描述了一套基于标准微型故事语料库（tiny story corpus）的完全可复现的训练与评估协议。参考实现已作为开源包发布，并提供支持权重发布的配套功能。