Convolution for Large Language Models

大语言模型的卷积研究

Abstract: Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size.

摘要： 大语言模型（LLMs）主要依赖于 Transformer 架构，其中自注意力机制提供了全局的 Token 交互，但并未显式地编码自然语言的局部性。我们研究了轻量级深度卷积（depthwise convolutions）是否能在不显著增加模型规模的情况下，提供这种局部归纳偏置。

Our macro-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention. A subsequent micro-level study favors a residual depthwise convolution with kernel size $k=3$, without additional normalization or activation.

我们的宏观消融实验对比了 Qwen3 Transformer 块中 17 个不同位置的卷积效果，发现当卷积应用于注意力机制之前的投影查询（Query）、键（Key）和值（Value）时，效果最佳。随后的微观研究表明，采用核大小为 $k=3$ 的残差深度卷积，且不额外增加归一化或激活函数，表现最为理想。

Across Qwen3 models and several pre-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than $0.01%$ parameters. A representation-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context. These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions.

在 Qwen3 模型和多种预训练数据规模下，该设计在七个下游基准测试中提升了平均准确率，同时参数增加量不到 $0.01%$。表征层面的案例研究进一步表明，卷积使得重复的 Token ID 对其直接上下文更加敏感。这些结果支持将深度卷积作为自注意力机制的轻量级补充，用于建模短距离的 Token 交互。