How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code
How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code
Hugging Face Inference Endpoints、Jobs 和 Buckets 如何驱动 Papers with Code 的搜索功能
3 months ago, we started a revival of Papers with Code (see also the announcement tweet). Its goal is to make open AI research accessible and digestible, so that people can easily find the artifacts related to a paper, find state-of-the-art (SOTA) across the various domains of AI, share interesting research and build on top of each other’s work. In other words, its goal is to power the wave of research that leads to the next Transformer.
三个月前，我们启动了 Papers with Code 的复兴计划（参见公告推文）。其目标是让开放的 AI 研究变得易于获取和理解，以便人们能够轻松找到与论文相关的工件，发现 AI 各个领域的最先进技术（SOTA），分享有趣的研究，并在此基础上进行构建。换句话说，它的目标是推动引领下一个 Transformer 的研究浪潮。
Of course, making AI research accessible requires a powerful search engine, so that humans and agents can quickly find relevant and related work, either through the website or the pwc search CLI command, which agents can use via the Skill. It’s important to note that searching for research is not quite the same as searching for regular text. A useful paper search engine should find an exact title or arXiv identifier, but it should also understand a query such as “small language models for code generation” even when those words do not appear together in a paper. It needs to recognize that “the original BERT paper” is a navigational request, tolerate an incomplete title or typos, and still respond quickly when a model service is cold or temporarily unavailable.
当然，要让 AI 研究变得易于获取，需要一个强大的搜索引擎，以便人类和智能体（Agents）能够快速找到相关工作，无论是通过网站还是 pwc search 命令行工具（智能体可以通过 Skill 使用该工具）。需要注意的是，搜索研究论文与搜索普通文本并不完全相同。一个有用的论文搜索引擎不仅应该能找到确切的标题或 arXiv 标识符，还应该能够理解诸如“用于代码生成的小型语言模型”这样的查询，即使这些词并没有在论文中同时出现。它需要识别出“原始 BERT 论文”是一个导航性请求，能够容忍不完整的标题或拼写错误，并且在模型服务处于冷启动或暂时不可用时依然能快速响应。
For Papers with Code, we built this as a hybrid search system. This is also based on our prior experience at ML6, where we developed RAG-based systems for clients. It turned out that hybrid search typically outperforms keyword- and vector-based search systems, as it combines the best of both worlds. Keyword search finds exact mentions, whereas vector search finds more fuzzy, semantically similar terms. Note that rerankers (also called cross-encoders) can further improve the results, although they also come with additional overhead and latency.
对于 Papers with Code，我们将其构建为一个混合搜索系统。这也基于我们在 ML6 的过往经验，当时我们为客户开发了基于 RAG 的系统。事实证明，混合搜索通常优于单纯的关键词搜索和向量搜索系统，因为它结合了两者的优点。关键词搜索可以找到精确的提及，而向量搜索可以找到更模糊、语义相似的术语。请注意，重排序器（也称为交叉编码器）可以进一步改善结果，尽管它们也会带来额外的开销和延迟。
Papers with Code relies on a PostgreSQL database, hence its full-text search capabilities provide a fast lexical baseline. For dense embeddings, pgvector is used to add semantic recall, and the reciprocal rank fusion (RRF) algorithm combines the two. Three Hugging Face services are used for the dense embeddings: Hugging Face Jobs gives us burstable GPU compute for embedding the paper corpus. Hugging Face Storage Buckets provides the durable handoff between our database, experiments, and Jobs. Hugging Face Inference Endpoints serves low-latency embeddings for live queries and incremental updates.
Papers with Code 依赖于 PostgreSQL 数据库，因此其全文搜索功能提供了一个快速的词法基准。对于稠密向量嵌入，我们使用 pgvector 来增加语义召回，并使用倒数排名融合（RRF）算法将两者结合起来。我们使用了三项 Hugging Face 服务来实现稠密嵌入：Hugging Face Jobs 为我们提供了可突发的 GPU 计算能力，用于对论文语料库进行嵌入；Hugging Face Storage Buckets 在我们的数据库、实验和 Jobs 之间提供了持久的数据中转；Hugging Face Inference Endpoints 则为实时查询和增量更新提供低延迟的嵌入服务。
Today, the system maintains embeddings for more than 110,000 current papers sourced from arXiv and Daily Papers. This post explains the architecture, the design decisions behind it, and the lessons we learned while taking it to production.
目前，该系统维护着超过 110,000 篇来自 arXiv 和 Daily Papers 的最新论文的嵌入向量。本文将解释其架构、背后的设计决策，以及我们在将其投入生产过程中学到的经验教训。
TL;DR We deliberately split search into an offline corpus build and an online search service. The expensive, throughput-oriented work runs as Jobs. Durable artifacts live in a Bucket. Only the small query-embedding step sits on the request path, behind a protected Inference Endpoint, to power the online search. If that endpoint is cold, busy, or unhealthy, search immediately falls back to full-text retrieval. This separation makes the system both powerful and fast.
简而言之：我们特意将搜索拆分为离线语料库构建和在线搜索服务。昂贵的、面向吞吐量的工作作为 Jobs 运行。持久化的工件存储在 Bucket 中。只有小型的查询嵌入步骤位于请求路径上，部署在受保护的 Inference Endpoint 之后，以驱动在线搜索。如果该端点处于冷启动、繁忙或不健康状态，搜索会立即回退到全文检索。这种分离使系统既强大又快速。
Start with a strict embedding contract
从严格的嵌入契约开始
Embedding pipelines often fail in subtle ways: a model revision changes, query and document prompts are mixed up, vectors are truncated differently, or an updated abstract no longer matches its stored vector. We avoid this by treating the embedding format as a versioned API. Every paper is encoded as: normalized title + "\n\n" + normalized abstract.
嵌入流水线经常会以微妙的方式失败：模型版本发生变化、查询和文档提示词混淆、向量截断方式不同，或者更新后的摘要不再与其存储的向量匹配。我们通过将嵌入格式视为版本化 API 来避免这种情况。每篇论文的编码格式为：标准化标题 + "\n\n" + 标准化摘要。
For each vector generation, we record: the model repository and exact revision; the output dimension; the input-format version; whether the input is a query or a document; the normalization method; a content hash for the source title and abstract. Our production generation uses Qwen/Qwen3-Embedding-0.6B, pinned to an exact revision, with 256-dimensional L2-normalized vectors. We selected the model with help from the MTEB leaderboard, the go-to benchmark for comparing embedding models.
对于每次向量生成，我们都会记录：模型仓库及确切版本；输出维度；输入格式版本；输入是查询还是文档；归一化方法；以及源标题和摘要的内容哈希值。我们的生产环境生成使用了 Qwen/Qwen3-Embedding-0.6B，锁定在特定版本，并使用 256 维的 L2 归一化向量。我们在 MTEB 排行榜（比较嵌入模型的首选基准）的帮助下选择了该模型。
Note that newer embedding models like Qwen3 allow for 2 new features: one can specify a dynamic embedding size, which allows to trade-off quality with speed/storage costs. Qwen models call this “MRL” which is short for Matryoshka Representation Learning. You can learn all about it here. We chose an embedding size of 256 to make the search fast. One can provide an instruction prompt. Qwen embedding models support a document prompt (which we use to embed the papers) and live searches use their query prompt (to embed the user query). This contract follows an embedding from export, through GPU inference, into PostgreSQL, and finally into online retrieval.
请注意，像 Qwen3 这样较新的嵌入模型支持两个新特性：一是支持指定动态嵌入大小，从而在质量与速度/存储成本之间进行权衡。Qwen 模型将其称为“MRL”，即 Matryoshka Representation Learning（套娃表示学习）。你可以在此处了解详情。我们选择了 256 的嵌入大小以加快搜索速度。二是支持提供指令提示词。Qwen 嵌入模型支持文档提示词（我们用它来嵌入论文），而实时搜索则使用查询提示词（用于嵌入用户查询）。这一契约贯穿了从导出、GPU 推理、存入 PostgreSQL，最后到在线检索的整个嵌入过程。
Jobs turn a database snapshot into a vector corpus
Jobs 将数据库快照转换为向量语料库
Full-corpus embedding is a classic batch workload. It needs a GPU for a relatively short period, benefits from high throughput, and should not consume resources between runs. Hugging Face Jobs fits that shape well: a Job is defined by a command, a hardware flavor, and optionally a Docker image, and can run uv scripts with their dependencies declared inline.
全语料库嵌入是一个经典的批处理工作负载。它需要在相对较短的时间内使用 GPU，受益于高吞吐量，并且在运行间隙不应消耗资源。Hugging Face Jobs 非常适合这种模式：一个 Job 由命令、硬件规格和可选的 Docker 镜像定义，并且可以运行内联声明依赖项的 uv 脚本。
Our corpus build starts by exporting the latest version of every paper from a repeatable-read PostgreSQL snapshot. The exporter streams rows rather than loading the catalog into memory, writes bounded JSONL shards, and creates a manifest containing row counts and SHA-256 checksums. We sync that immutable run directory to a private Storage Bucket and mount the Bucket directly (using hf-mount) into an l4x1 Job (an NVIDIA L4 GPU, which has 24GB of VRAM).
我们的语料库构建始于从可重复读（repeatable-read）的 PostgreSQL 快照中导出每篇论文的最新版本。导出器采用流式处理行数据，而不是将整个目录加载到内存中，它会写入有界的 JSONL 分片，并创建一个包含行数和 SHA-256 校验和的清单。我们将该不可变的运行目录同步到私有的 Storage Bucket，并使用 hf-mount 将其直接挂载到 l4x1 Job（配备 24GB 显存的 NVIDIA L4 GPU）中。