EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

EntropyMoE：面向无分词器大语言模型的熵感知稀疏专家路由

Abstract: Recent byte-level large language models (LLMs) have made tokenizer-free modeling increasingly competitive by grouping bytes into dynamically sized patches. However, existing byte-patch architectures still apply the same dense feed-forward computation to every patch. This uniform computation cannot adapt model capacity to variations in patch semantics and granularity.

摘要： 近期的字节级大语言模型（LLMs）通过将字节分组为动态大小的补丁（patches），使得无分词器（tokenizer-free）建模的竞争力日益增强。然而，现有的字节补丁架构仍对每个补丁应用相同的密集前馈计算。这种统一的计算方式无法根据补丁语义和粒度的变化来调整模型容量。

We address this limitation with EntropyMoE, a Mixture-of-Experts (MoE) architecture designed for dynamic byte patches. EntropyMoE replaces the dense feed-forward modules in the global patch Transformer with Top-K expert layers. Each dynamic patch serves as the basic unit of expert routing, and its byte coverage determines its contribution to workload accounting.

我们通过 EntropyMoE 解决了这一局限性，这是一种专为动态字节补丁设计的混合专家（MoE）架构。EntropyMoE 将全局补丁 Transformer 中的密集前馈模块替换为 Top-K 专家层。每个动态补丁作为专家路由的基本单元，其字节覆盖范围决定了其在工作负载核算中的贡献。

The router selects experts directly from patch entropy, using the same granularity signal that underlies dynamic patch construction to organize sparse computation. Patch entropy and length jointly define the feature space for regulating expert specialization.

路由选择器直接根据补丁熵来选择专家，利用与动态补丁构建相同的粒度信号来组织稀疏计算。补丁熵和长度共同定义了用于调节专家专业化的特征空间。

Experiments show that EntropyMoE achieves the lowest held-out bits-per-byte among matched dense and sparse baselines while maintaining comparable downstream accuracy. These results establish patch entropy as an effective routing coordinate for sparse conditional computation and extend Mixture-of-Experts modeling beyond tokenizer-based representations.

实验表明，EntropyMoE 在匹配的密集和稀疏基准模型中实现了最低的留出集每字节比特数（bits-per-byte），同时保持了相当的下游任务准确率。这些结果确立了补丁熵作为稀疏条件计算的有效路由坐标，并将混合专家建模的应用范围扩展到了基于分词器的表示之外。