TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

TinyCeNN-LM：基于 CeNN 启发式细胞循环层的预训练注意力机制质量门控转换

Abstract: Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a \emph{quality-gated post-training conversion} framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation.

摘要： 在预训练语言模型中替换注意力机制是一个兼容性问题：一个看似合理的替代方案可能会改变后续层所预期的表征。TinyCeNN-LM 引入了一种“质量门控训练后转换”框架，利用受 CeNN（细胞神经网络）启发的细胞循环层，结合了有界局部处理、紧凑循环记忆、路由、融合以及“接受或回滚”验证机制。

Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative $\Delta\mathrm{NLL}=+0.01209$, while layer 3 is rejected despite acceptable NLL because representation fidelity fails.

研究探讨了三种实现方式：集成记忆（Integrated Memory）、记忆融合（MemoryFusion）以及 PDelta3-GDN2-CLVR+Local32。严格的 PDelta3 转换仅在表征和负对数似然（NLL）标准通过固定阈值时才接受该层。在 SmolLM2-135M 模型上，第 0-2 层被接受，累积 $\Delta\mathrm{NLL}=+0.01209$；而第 3 层尽管 NLL 指标合格，但因表征保真度未达标而被拒绝。

On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final $\Delta\mathrm{NLL}=+0.02073$. Integrated Memory keeps perplexity within $-0.07%$ to $+0.93%$ while reducing total cache by up to $6.01%$. A sampled 200-item downstream sanity check gives $28.5%$—$32.0%$ overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.

在 Qwen3.5-0.8B 模型上，全注意力层 3、7 和 11 被接受，最终 $\Delta\mathrm{NLL}=+0.02073$。集成记忆方法将困惑度（perplexity）保持在 $-0.07%$ 到 $+0.93%$ 的范围内，同时将总缓存减少了高达 $6.01%$。针对转换后的 Qwen 版本进行的 200 项下游抽样完整性检查显示，整体准确率为 $28.5%$—$32.0%$。研究结果支持采用保守的、质量门控的结构转换，而非盲目地进行全局注意力替换或加速。