LFM2.5-Encoders for Fast Long-Context Inference on CPU

查看原文 / View Original 编辑 / Edit

LFM2.5-Encoders for Fast Long-Context Inference on CPU

LFM2.5-Encoders：实现 CPU 上的快速长上下文推理

Today, we release two new encoder models on Hugging Face: LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. They match the quality of larger models but stay fast as inputs get longer. This means you can run document-scale jobs on the hardware you already have, even on CPU. 今天，我们在 Hugging Face 上发布了两款全新的编码器模型：LFM2.5-Encoder-230M 和 LFM2.5-Encoder-350M。它们在保持与大型模型同等质量的同时，随着输入长度的增加依然能保持高速运行。这意味着你可以在现有的硬件上（甚至是 CPU 上）运行文档规模的处理任务。

Here’s what you get: 以下是它们的核心优势：

  • Strong for their size: match or beat larger encoders on GLUE, SuperGLUE, and multilingual tasks.
  • 体积小巧但性能强劲： 在 GLUE、SuperGLUE 和多语言任务中，表现持平甚至超越了更大的编码器模型。
  • 8,192-token context with latency that grows slowly as inputs get longer.
  • 8,192 token 上下文： 且延迟随输入长度增加而缓慢增长。
  • Fast on CPU: about 3.7× faster than ModernBERT-base at long context.
  • CPU 运行速度快： 在长上下文任务中，比 ModernBERT-base 快约 3.7 倍。

With these, you can build intent routers, policy linters, PII detectors, and text classifiers that run cheaply, all day. See the live demos below. 利用这些模型，你可以构建意图路由器、策略检查器、PII（个人身份信息）检测器和文本分类器，并以低成本全天候运行。请查看下方的实时演示。

Why we built a general-purpose encoder

为什么我们要构建通用编码器

Last month we released LFM2.5-Retrievers, built for multilingual search. LFM2.5-Encoders come from the same family but serve a broader purpose. They’re pre-trained with a masked-language objective, so you can fine-tune them for classification, token-level tasks, and search alike. Search is just one thing an encoder enables. That’s why we built a general-purpose model instead of reusing the retrievers. 上个月，我们发布了专为多语言搜索设计的 LFM2.5-Retrievers。LFM2.5-Encoders 属于同一系列，但用途更为广泛。它们通过掩码语言模型（Masked-language objective）进行预训练，因此你可以针对分类、Token 级任务以及搜索任务进行微调。搜索只是编码器支持的功能之一，这就是为什么我们构建了通用模型，而不是直接复用检索器。

Encoders power many modern production NLP applications: classifiers, intent routers, safety filters. These jobs run all day, usually on CPU, on ever-longer inputs. BERT established this class of model, and recently ModernBERT pushed its accuracy, speed, and context further. LFM2.5-Encoders take the next step on the LFM2 architecture, where cost grows slowly as inputs grow. 编码器为许多现代生产级 NLP 应用提供支持：分类器、意图路由器、安全过滤器等。这些任务通常在 CPU 上全天候运行，且处理的输入越来越长。BERT 开创了此类模型，而最近的 ModernBERT 进一步提升了其准确性、速度和上下文能力。LFM2.5-Encoders 在 LFM2 架构上更进一步，实现了随输入增长而成本缓慢增加的特性。

How the encoders are built

编码器是如何构建的

We initialize the encoders from their respective LFM2 decoder backbones: LFM2.5-230M and LFM2.5-350M. Then we turn each causal decoder into a bidirectional encoder with a few changes: 我们从各自的 LFM2 解码器主干（LFM2.5-230M 和 LFM2.5-350M）初始化编码器。然后，通过以下几项更改将因果解码器转换为双向编码器：

  • Bidirectional attention mask: each token now sees the tokens on both sides, not just the ones before it.
  • 双向注意力掩码： 每个 Token 现在可以看到两侧的 Token，而不仅仅是它之前的 Token。
  • Non-causal short convolutions: we pad them symmetrically so each token’s convolution mixes in its neighbors on both sides.
  • 非因果短卷积： 我们对其进行对称填充，使每个 Token 的卷积都能融合两侧的邻居信息。
  • Masked language modeling: we mask 30% of the tokens during training.
  • 掩码语言建模： 我们在训练过程中掩盖了 30% 的 Token。

We train both models in two stages: 我们分两个阶段训练这两个模型：

  1. General language competence: a short-context masked-language objective on a large web corpus at a 1,024-token context.
  2. 通用语言能力： 在大规模网络语料库上进行 1,024 token 上下文的短上下文掩码语言训练。
  3. Long-context adaptation: extending context to 8,192 tokens on the full data mix, strengthening factual, legal, and multilingual competence.
  4. 长上下文适配： 在完整数据混合集上将上下文扩展至 8,192 token，增强事实、法律和多语言能力。

Benchmark Results

基准测试结果

We fine-tune each model fully on every task and report the resulting score. Across the table, that’s 14 models on 17 tasks pulled from GLUE, SuperGLUE, and multilingual classification. We report the mean across five held-out seeds, so the numbers are stable run to run. The full framework and raw results are open-sourced. 我们对每个模型在所有任务上进行完全微调并报告最终得分。表中涵盖了来自 GLUE、SuperGLUE 和多语言分类任务的 17 个任务上的 14 个模型。我们报告了五个留出种子（held-out seeds）的平均值，以确保结果在多次运行中保持稳定。完整的框架和原始结果已开源。

LFM2.5-Encoder-350M ranks fourth of the 14 models. The three ahead of it are all larger, including a 3.5B model nearly 10 times its size. LFM2.5-Encoder-230M beats ModernBERT-base and every EuroBERT model, while being smaller than most of them. Both also score well above our own LFM2.5-Retrievers here. LFM2.5-Encoder-350M 在 14 个模型中排名第四。排在它前面的三个模型体积都更大，其中包括一个体积几乎是其 10 倍的 3.5B 模型。LFM2.5-Encoder-230M 击败了 ModernBERT-base 和所有 EuroBERT 模型，且体积比它们中的大多数都要小。两款模型在此处的得分也远高于我们自己的 LFM2.5-Retrievers。

Inference speed on CPU and GPU

CPU 和 GPU 上的推理速度

Our encoders inherit the LFM2 backbone’s fast inference. Since both our encoders and ModernBERT support an 8,192-token context, we measure speed across the full range. Our encoders show their biggest edge on CPU. 我们的编码器继承了 LFM2 主干的快速推理能力。由于我们的编码器和 ModernBERT 都支持 8,192 token 的上下文，我们测量了全范围内的速度。我们的编码器在 CPU 上展现出了最大的优势。

Here, LFM2.5-Encoder-230M is the fastest at every sequence length (even faster than the smaller ModernBERT-base for short inputs). With increasing input length, throughput decreases sharply for ModernBERT, while our LFM2.5-Encoders rise into the mid-range before tapering. At 8,192 tokens, ModernBERT-base takes over a minute and a half per forward pass versus about 28s for LFM2.5-Encoder-230M. This is about 3.7x faster. 在这里，LFM2.5-Encoder-230M 在所有序列长度下都是最快的（对于短输入，甚至比更小的 ModernBERT-base 还要快）。随着输入长度的增加，ModernBERT 的吞吐量急剧下降，而我们的 LFM2.5-Encoders 在下降前会先升至中等水平。在 8,192 token 时，ModernBERT-base 的单次前向传播需要超过一分半钟，而 LFM2.5-Encoder-230M 仅需约 28 秒。这大约快了 3.7 倍。

For developers, that means you can scan or classify a full contract, transcript, or long support thread in under 30 seconds on a laptop CPU. On GPU, a similar pattern holds with a smaller margin: ModernBERT-base leads below ~1K tokens on the Apple GPU. Our encoders take the lead from about 2K tokens. This shows that for long inputs, LFM2.5-Encoders are the faster choice, and if you’re running on CPU, dramatically so. 对于开发者而言，这意味着你可以在笔记本电脑的 CPU 上，在 30 秒内扫描或分类一份完整的合同、转录稿或长篇支持对话。在 GPU 上，情况类似但差距较小：在 Apple GPU 上，ModernBERT-base 在 1K token 以下领先，而我们的编码器从 2K token 左右开始领先。这表明对于长输入，LFM2.5-Encoders 是更快的选择，如果是在 CPU 上运行，优势则更为显著。

LFM2.5-Encoder demos

LFM2.5-Encoder 演示

We built the demos below from fine-tuned LFM2.5-Encoders. Each one runs in a CPU-only Hugging Face space: 我们基于微调后的 LFM2.5-Encoders 构建了以下演示。每个演示都在仅限 CPU 的 Hugging Face Space 中运行：

  • Zero-shot prompt routing: define your own routing lanes as free text. The model scores the whole prompt against every lane in one pass.
  • 零样本提示路由： 以自由文本定义你自己的路由通道。模型在一次传递中根据每个通道对整个提示进行评分。
  • Zero-shot policy linting: check text against your company’s rules, written as free text. It scores every token against every rule in one pass.
  • 零样本策略检查： 根据以自由文本编写的公司规则检查文本。它在一次传递中根据每条规则对每个 Token 进行评分。
  • Spell checking: correct misspellings token by token.
  • 拼写检查： 逐个 Token 纠正拼写错误。
  • PII detection: spot and remove 40 kinds of personal information across 16 languages.
  • PII 检测： 在 16 种语言中识别并删除 40 种个人信息。
  • Masked-diffusion text generation (bonus): run the encoder as a chatbot that generates text by iteratively unmasking instead of left to right.
  • 掩码扩散文本生成（彩蛋）： 将编码器作为聊天机器人运行，通过迭代去掩码（而非从左到右）来生成文本。

How to use and fine-tune LFM2.5-Encoders

如何使用和微调 LFM2.5-Encoders

Reach for an LFM2.5-Encoder when you have a high-volume understanding task, such as classification, routing, extraction, or scoring, that runs constantly and has to stay cheap and fast. For jobs like these, a fine-tuned encoder is smaller, faster, and far cheaper to run than a generative LLM, and it fits on the CPUs you already have. 当你需要处理高并发的理解任务（如分类、路由、提取或评分），且要求任务持续运行、成本低廉且速度快时，请选择 LFM2.5-Encoder。对于这类工作，微调后的编码器比生成式大模型（LLM）体积更小、速度更快、运行成本更低，并且完全可以在你现有的 CPU 上运行。

Between the two encoder sizes: 关于两种编码器尺寸的选择：

  • LFM2.5-Encoder-350M: choose it when accuracy matters most.
  • LFM2.5-Encoder-350M： 当准确性最重要时选择它。
  • LFM2.5-Encoder-230M: choose it for tighter hardware or higher throughput.
  • LFM2.5-Encoder-230M： 当硬件资源受限或需要更高吞吐量时选择它。

You can start in a few lines. Load a model with transformers. Then run it directly for masked-token prediction, or attach your own head and fine-tune it for your task. 你只需几行代码即可开始。使用 transformers 加载模型，然后直接运行它进行掩码 Token 预测，或者添加你自己的头部（head）并针对你的任务进行微调。

Load and run the model 加载并运行模型

Install the latest version of transformers: 安装最新版本的 transformers： pip install -U transformers

Run masked-token prediction: 运行掩码 Token 预测：

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model_id = "LiquidAI/LFM2.5-Encoder-230M" # or "LiquidAI/LFM2.5-Encoder-350M"
← 返回新闻列表 / Back to news list