NeoMME: an efficient Multimodal-native and Multilingual Encoder

NeoMME：一种高效的多模态原生多语言编码器

TL;DR We introduce NeoMME, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective. 简而言之，我们推出了 NeoMME，这是一个包含 2.6 亿和 8 亿参数的多语言多模态编码器系列。与许多生成式视觉语言模型不同，NeoMME 不使用独立的预训练视觉塔或因果语言模型。它通过单个双向 Transformer 同时处理文本标记和原始图像块，并使用掩码离散扩散目标从头开始训练整个模型。

We fine-tuned NeoMME for visual document retrieval using ColPali’s page-image approach. NeoMME-Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT’s throughput. 我们使用 ColPali 的页面图像方法对 NeoMME 进行了视觉文档检索微调。NeoMME-Retriever 可在一次前向传播中返回密集和后期交互嵌入。两种模型尺寸均位于 ViDoRe v3 的 nDCG@10 与模型尺寸的帕累托前沿。在 NVIDIA L40S GPU 上，以 2048×2048 的匹配图像输入尺寸计算，260M 模型每秒可编码约 51 页，吞吐量约为 ColModernVBERT 的两倍。

Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10. NeoMME is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license. 分层标记池化和非对称量化将每页的后期交互索引存储从约 1.5 MB 减少到 6 kB（缩小了 255 倍），同时保留了超过 95% 的基准 nDCG@10 性能。NeoMME 现已在 Hugging Face Transformers 中可用。我们根据 Apache 2.0 许可证发布了所有模型检查点。

Why another multimodal encoder? Many recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model’s input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture. 为什么要开发另一种多模态编码器？最近许多视觉文档检索器都是从预训练的生成式视觉语言模型改编而来的。一个单独预训练的视觉编码器产生视觉特征，通过投影仪映射到语言模型的输入空间，然后由因果解码器处理组合后的图像和文本表示。检索、分类和标记任务并不需要自回归生成文本，因此它们不需要因果解码器，也不需要该架构带来的参数和计算开销。

ModernBERT brought efficient architecture and training improvements to bidirectional encoders. For visual document retrieval, ModernVBERT applied a bidirectional ModernBERT-style text encoder while retaining a separate pretrained SigLIP2 vision tower. We wanted to push this even further by designing and training a multimodal encoder without having to carry over the parameter and compute overhead of a VLM. ModernBERT 为双向编码器带来了高效的架构和训练改进。在视觉文档检索方面，ModernVBERT 应用了双向 ModernBERT 风格的文本编码器，同时保留了独立的预训练 SigLIP2 视觉塔。我们希望通过设计和训练一种无需承担 VLM 参数和计算开销的多模态编码器，将这一理念进一步推向极致。

NeoMME (pronounced “nee-oh-me”, IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal foundation encoder that generates vector representations for input text and/or images using a single Transformer encoder. It is not based on an existing pretrained vision tower, text encoder, or text decoder. Unlike dual-tower and VLM encoders, NeoMME processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or a pretrained text encoder or decoder. Images and text use the same computational path, so NeoMME can more easily support pretraining, fine-tuning, parallelization, and serving across both modalities. NeoMME（发音为 “nee-oh-me”，IPA /ˈniː.oʊ.mi/）是一个多语言、多模态基础编码器，它使用单个 Transformer 编码器为输入文本和/或图像生成向量表示。它不基于现有的预训练视觉塔、文本编码器或文本解码器。与双塔和 VLM 编码器不同，NeoMME 在一个双向 Transformer 中处理图像块和文本标记，无需预训练的视觉塔或预训练的文本编码器/解码器。图像和文本使用相同的计算路径，因此 NeoMME 可以更轻松地支持跨两种模态的预训练、微调、并行化和服务。

NeoMME encoder backbone: One Transformer for images and text. NeoMME comes in two sizes, 260M and 800M. Both variants share the same architecture: NeoMME 编码器主干：用于图像和文本的单一 Transformer。NeoMME 有 2.6 亿和 8 亿两种尺寸，两种变体共享相同的架构：

Native multimodal inputs: text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small MLP. Both enter the same Transformer encoder.
原生多模态输入： 文本输入使用因子分解的标记嵌入，而图像被划分为非重叠的 32×32 网格块，并通过小型 MLP 进行投影。两者都进入同一个 Transformer 编码器。
Dynamic image resolution: images keep their aspect ratio and size. This allows the model to use more tokens on a high-resolution, information-dense document page than on a smaller image with less content.
动态图像分辨率： 图像保持其纵横比和尺寸。这使得模型能够在高分辨率、信息密集的文档页面上使用比在内容较少的较小图像上更多的标记。
Long bidirectional context: both models have a context length of 16,384 tokens (enough for up to two standard 3840×2160 4K UHD images). Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention.
长双向上下文： 两种模型的上下文长度均为 16,384 个标记（足以容纳最多两张标准的 3840×2160 4K UHD 图像）。大多数层使用对称滑动窗口注意力机制，而每隔六层以及最后一层使用全局注意力机制。
A modern encoder stack: NeoMME uses recent encoder improvements such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs, among others.
现代编码器堆栈： NeoMME 使用了近期的编码器改进技术，例如分组查询注意力（GQA）、查询-键归一化、门控注意力、2D 旋转位置嵌入和平方 ReLU MLP 等。
Multilingual text: we trained a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.
多语言文本： 我们从头开始训练了一个 BPE 分词器，词汇量为 13.1 万个标记，涵盖多语言文本、代码、数学和机器生成的图像转录内容。

Learning from images through masked text: We pretrain NeoMME from scratch as a discrete masked-diffusion text denoiser. For each text-only example, we sample a corruption rate uniformly between 0 and 1. Each eligible text token is then independently masked at that rate. Multimodal examples use corruption rates between 0.3 and 1. The image patches remain visible while NeoMME reconstructs masked text. 通过掩码文本从图像中学习：我们从头开始将 NeoMME 预训练为离散掩码扩散文本去噪器。对于每个纯文本示例，我们从 0 到 1 之间均匀采样一个损坏率，然后以该比率独立掩码每个符合条件的文本标记。多模态示例使用 0.3 到 1 之间的损坏率。在 NeoMME 重建掩码文本时，图像块保持可见。

With light masking, the model can often recover a missing word from the surrounding text alone. For example, “cat” is a plausible completion of “The [MASK] sat on the mat,” even without an image. But high masking forces the model to learn image-grounded descriptions with little to no signal from the non-masked input text tokens. Higher text corruption removes language-only shortcuts and encourages NeoMME to use visible image evidence. 在轻度掩码下，模型通常仅凭周围文本就能恢复缺失的单词。例如，即使没有图像，“cat”也是“The [MASK] sat on the mat”的一个合理补全。但高比例掩码迫使模型在非掩码输入文本标记信号极少甚至没有的情况下，学习基于图像的描述。更高的文本损坏率消除了纯语言捷径，并鼓励 NeoMME 使用可见的图像证据。

Pretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples. This text budget is relatively small compared with ModernBERT’s 2 trillion training token budget. Hence, we chose the NorMuon optimizer to improve data efficiency during training. 预训练混合了多语言文本、代码、数学、自然图像和文档图像。每个模型处理约 5240 亿个打包输入标记，其中包括来自纯文本示例的 2900 亿个标记。与 ModernBERT 的 2 万亿训练标记预算相比，这个文本预算相对较小。因此，我们选择了 NorMuon 优化器来提高训练过程中的数据效率。

NeoMME-Retriever: To get a meaningful downstream evaluation of the backbone, we fine-tune NeoMME for visual document retrieval using the page-image methodology introduced by ColPali. While traditional text-based retrieval consists of retrieving text chunks, NeoMME-Retriever ranks document page screenshots and bypasses all the preprocessing OCR steps necessary to extract text from PDFs. Treating the pages as images preserves layout, charts, tables, font type and size, and other visual clues that cannot be captured even by a perfect OCR. NeoMME-Retriever：为了对主干模型进行有意义的下游评估，我们使用 ColPali 引入的页面图像方法对 NeoMME 进行了视觉文档检索微调。传统的基于文本的检索涉及检索文本块，而 NeoMME-Retriever 则对文档页面截图进行排序，绕过了从 PDF 中提取文本所需的所有预处理 OCR 步骤。将页面视为图像可以保留布局、图表、表格、字体类型和大小以及其他视觉线索，这些线索即使是完美的 OCR 也无法捕捉到。