Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
使用 Sentence Transformers 的多向量（后期交互）嵌入模型
Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v6.0 update, it gains a fourth model type: MultiVectorEncoder, for ColBERT-style late interaction retrieval. Sentence Transformers 是一个用于使用和训练嵌入（embedding）及重排序（reranker）模型的 Python 库，适用于检索增强生成（RAG）、语义搜索等应用。在 v6.0 更新中，它新增了第四种模型类型：MultiVectorEncoder，用于 ColBERT 风格的后期交互（late interaction）检索。
Any PyLate checkpoint and any Stanford-NLP ColBERT checkpoint loads straight into it, and colpali-engine models for visual document retrieval can be used too, through the same familiar API you already use for dense, sparse, and reranker models. 任何 PyLate 检查点和 Stanford-NLP ColBERT 检查点都可以直接加载到其中，用于视觉文档检索的 colpali-engine 模型也可以通过你已经熟悉的、用于密集（dense）、稀疏（sparse）和重排序模型的 API 来使用。
Where a regular embedding model compresses a whole text into one vector, a multi-vector model keeps one vector per token and scores query against document with the MaxSim operator. That preserves token-level matching information that a single vector has to average away, which usually means stronger retrieval at the cost of a bigger index. 常规嵌入模型将整段文本压缩为一个向量，而多向量模型则为每个 token 保留一个向量，并使用 MaxSim 算子对查询和文档进行评分。这保留了单向量模型必须通过平均化而丢失的 token 级匹配信息，通常意味着更强的检索能力，但代价是索引体积更大。
It’s also the state of the art for visual document retrieval, where a text query is matched against page images directly, with no OCR step in between. In this blogpost, we’ll show you how to use these models: loading the various checkpoint formats, encoding and scoring, plugging them into a search stack, running them on page images, and keeping the index affordable. 这也是视觉文档检索领域的最先进技术，它无需 OCR 步骤，直接将文本查询与页面图像进行匹配。在本篇博文中，我们将展示如何使用这些模型：加载各种检查点格式、编码与评分、将其接入搜索栈、在页面图像上运行，以及如何保持索引的经济性。
What are Multi-Vector Models?
什么是多向量模型？
A dense embedding model reads a text and returns a single fixed-size vector. Everything the model noticed has to fit in those 384, 768, or 1024 numbers, and similarity is one dot product between two such summaries. This works remarkably well, but the compression is lossy in a specific way: a rare entity, an exact identifier, or one crucial clause in a long passage all have to compete for room in the same vector. 密集嵌入模型读取文本并返回单个固定大小的向量。模型捕捉到的所有信息都必须塞进这 384、768 或 1024 个数字中，相似度计算则是两个此类摘要之间的点积。这种方法效果显著，但压缩是有损的：罕见实体、精确标识符或长段落中的关键从句，都必须在同一个向量中争夺空间。
A multi-vector model (also called a late-interaction or ColBERT-style model, after the ColBERT paper) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector. 多向量模型（也称为后期交互或 ColBERT 风格模型，源自 ColBERT 论文）跳过了这种压缩。它运行相同的 Transformer，但不是将 token 嵌入池化为一个向量，而是将每个 token 嵌入投影到一个较小的维度（通常为 128）并保留所有这些向量。一个 9 个 token 的文档会变成一个 9x128 的矩阵，而不是 1x128 的向量。
The interaction between query and document is then deferred until scoring time, which is where the name “late interaction” comes from. A cross-encoder interacts early: both texts go through the model together, which is accurate but leaves nothing to precompute, since every document has to be re-encoded for each new query. 查询和文档之间的交互被推迟到评分阶段，这就是“后期交互”名称的由来。交叉编码器（cross-encoder）进行早期交互：两个文本一起通过模型，这虽然准确，但无法进行预计算，因为每个新查询都必须重新编码所有文档。
A bi-encoder, which is what the dense embedding model above is, barely interacts at all (one dot product between two finished summaries), and that is exactly what lets you encode a collection once and query it fast. Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token, which leaves far more room for the two to interact. 双编码器（bi-encoder，即上述的密集嵌入模型）几乎不进行交互（仅在两个完成的摘要之间进行一次点积），这正是让你能够一次性编码集合并快速查询的原因。后期交互则介于两者之间：文档仍然独立编码并可离线索引，但在评分时会将每个查询 token 与每个文档 token 进行比较，这为两者交互留下了更大的空间。
The MaxSim Operator
MaxSim 算子
Scoring uses MaxSim: for each query token, take its highest similarity against any document token, then sum those maxima across the query. 评分使用 MaxSim：对于每个查询 token，取其与任何文档 token 的最高相似度，然后将这些最大值在整个查询中求和。
$$\text{MaxSim}(Q, D) = \sum_{Q_i \in Q} \max_{D_j \in D} Q_i \cdot D_j$$
Because the token embeddings are L2-normalized, each of those dot products is a cosine similarity in [-1, 1], so the whole sum lands within [-num_query_tokens, num_query_tokens]. You can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document supports the query overall. 由于 token 嵌入经过了 L2 归一化，每个点积都是 [-1, 1] 范围内的余弦相似度，因此总和落在 [-查询 token 数量, 查询 token 数量] 之间。你可以将该算子理解为一种软对齐：每个查询 token 都指向最能解释它的那个文档 token，而得分则代表文档对查询的整体支持程度。
The alignment doesn’t have to be lexical, since the token embeddings are contextualized. Encode “Where do penguins live?” against “Penguins inhabit Antarctica.” with lightonai/mLateOn and the query token live finds its best match on inhabit at 0.94, a word it shares no characters with! 这种对齐不必是词汇上的，因为 token 嵌入是上下文相关的。使用 lightonai/mLateOn 将“Where do penguins live?”与“Penguins inhabit Antarctica.”进行编码，查询 token “live”会在“inhabit”上找到 0.94 的最佳匹配，尽管它们没有任何共同字符！