Unranked, systemd, crawls

Posted: 2026-07-22 发布日期：2026年7月22日

There’s been some changes to Marginalia Search: Marginalia Search 最近进行了一些调整：

• Migrated to systemd from docker
• 从 Docker 迁移到了 systemd
• New unranked query endpoint
• 新增了无排名（unranked）查询端点
• Wide domains got their own index for faster crawls
• 为大型域名建立了独立索引，以实现更快的抓取速度

In brief, this has removed a lot of operational headaches, reduced expensive queries leaving more computational power for the rest, and cut crawl times in half. 简而言之，这些改动消除了许多运维上的头疼问题，减少了高开销查询，从而为其他任务留出了更多计算资源，并将抓取时间缩短了一半。

Let’s tackle them in some order. 让我们按顺序逐一探讨。

We have docker at home

我们家里有 Docker（意指：我们自己实现了类似 Docker 的功能）

The system has been migrated off docker compose, and onto bare systemd in production. This has been successful, and solved a number of issues. 生产环境系统已从 Docker Compose 迁移到了纯粹的 systemd。这次迁移非常成功，并解决了一系列问题。

There are a few constraints that demand a non-standard deployment for Marginalia. Marginalia 的一些特殊限制要求采用非标准的部署方式。

NUMA (non-uniform memory architecture). NUMA（非统一内存访问架构）。

The production server has two CPUs, each with their own RAM bank, and while they can reach over into each other’s memory, this comes at a cost. In many scenarios this isn’t a big deal, but for indexes and databases, generally bottlenecked by RAM bandwidth, this is far from ideal. 生产服务器拥有两个 CPU，每个 CPU 都有各自的内存库。虽然它们可以访问对方的内存，但这会带来性能损耗。在许多场景下这不算什么大问题，但对于通常受限于内存带宽的索引和数据库而言，这远非理想状态。

Process lifecycles. 进程生命周期。

A search engine is anything but stateless, some processes are fairly heavy and run for weeks, some services are very slow to start, some services need to restart relatively often. This all but demands that the search engine is cut up into different pieces. Not necessarily microservices, but at least services. 搜索引擎绝非无状态，有些进程非常繁重且运行数周，有些服务启动缓慢，还有些服务需要频繁重启。这几乎要求搜索引擎必须被拆分成不同的部分。不一定是微服务，但至少应该是独立的服务。

IP addresses. IP 地址。

The crawler (and crawler-adjacent processes) run off about a dozen public IP addresses, from the same host. This is done via ipvlan using network namespaces. Linux has many advanced capabilities for giving processes their own virtual network stack, which can be wired together with virtual switches and virtual patch cables. There are to my knowledge no tools for managing this I would consider good, it’s all either too low level, or abstracting away key functionality, or suffering from multiple sources of truths leading to jank when they inevitably differ. 爬虫（及相关进程）在同一台主机上使用十几个公共 IP 地址运行。这是通过使用网络命名空间（network namespaces）的 ipvlan 实现的。Linux 拥有许多高级功能，可以为进程提供独立的虚拟网络栈，并通过虚拟交换机和虚拟跳线连接起来。据我所知，目前没有我认为好用的管理工具；它们要么过于底层，要么抽象掉了关键功能，要么因为存在多个事实来源（sources of truth），在它们不可避免地产生差异时导致系统不稳定。

Docker can deal with all of these constraints, but at the expense of considerable jank. Docker 可以处理所有这些限制，但代价是相当多的不稳定因素。

Docker’s networking model in particular, while it supports ipvlan, doesn’t map onto it particularly well, if you have a finite subnet you pretty much have to have spare free IPs to be able to reliably restart services. It further doesn’t let you set firewall rules for the ipvlan interface, which means that anything you put there better not bind on the public IP, or its ass is going to be hanging out in full view. This is in no way a limitation of ipvlans, but purely a docker problem. 特别是 Docker 的网络模型，虽然支持 ipvlan，但映射得并不好。如果你有一个有限的子网，你几乎必须预留空闲 IP 才能可靠地重启服务。此外，它不允许你为 ipvlan 接口设置防火墙规则，这意味着你放在那里的任何东西最好不要绑定在公共 IP 上，否则它将完全暴露在外。这绝不是 ipvlan 的局限性，纯粹是 Docker 的问题。

This is made worse by the fact that there is inadequate tooling for letting the processes in the container know which network interface is public, so you have to kinda guess based on dowsing rods, RFC 1918, and looking at tea leaves. This is no way to live. 更糟糕的是，缺乏足够的工具让容器内的进程知道哪个网络接口是公共的，所以你只能靠“占卜”、RFC 1918 标准和“看茶叶渣”来猜测。这根本没法活。

There is no magic in docker, it’s just cgroups and namespaces all the way down, and you can fully replicate what it’s doing yourself, if you really want to torment yourself using syscalls or shellscripts, or if you want a middle ground using systemd. Docker 并没有什么魔法，它底层全是 cgroups 和命名空间。如果你真的想通过系统调用或 shell 脚本来折磨自己，或者想通过 systemd 寻找一个折中方案，你完全可以自己复刻它所做的一切。

If docker’s problem is that there is a mismatch between the docker model of the system and the system itself, systemd represents a more raw approach, where a lot more of the configuration is in your hands, while still offering many capabilities you’d want for a more serious deployment, such as health checks, automatic restarts that can be configured. 如果说 Docker 的问题在于其系统模型与实际系统之间存在错位，那么 systemd 则代表了一种更原始的方法。它将更多的配置权交到你手中，同时仍然提供了严肃部署所需的许多功能，例如健康检查和可配置的自动重启。

Setting up a million .service:s is pretty tedious, but drop-ins help. There isn’t a ton to say here, other than after some fiddling, it works very well, and feels a lot more stable than docker ever did. Cutting JIB out of builds means builds now take 2-3 seconds for the most part. 设置一百万个 .service 文件确实很繁琐，但使用 drop-in 配置会有所帮助。这里没什么好说的，除了经过一番折腾后，它运行得非常好，而且感觉比 Docker 稳定得多。从构建中剔除 JIB 意味着现在的构建过程大部分只需 2-3 秒。

Deployments are faster, there’s less jank in service discovery because “containers” retain the same internal and external IP, the whole operation feels considerably less floaty. Good upgrade. 部署速度更快了，服务发现中的不稳定因素也减少了，因为“容器”保留了相同的内部和外部 IP，整个操作感觉不再那么“飘忽不定”。这是一次很好的升级。

There are many opinions about systemd, and I think for most desktop-type systems, it’s incredibly overengineered and kind of a pain to work with. Though for something like this, its design is well motivated, and it really shines. 关于 systemd 有很多不同的看法，我认为对于大多数桌面系统来说，它确实过度设计了，用起来很痛苦。但对于像这样的项目，它的设计动机非常充分，而且确实表现出色。

Unranked Queries

无排名查询

The index now allows unranked query execution! 索引现在支持无排名查询执行了！

The query execution pipeline used in the search engine is fairly computationally expensive, and does things that aren’t strictly necessary in many queries. 搜索引擎中使用的查询执行流水线计算开销相当大，并且在许多查询中执行了一些并非绝对必要的操作。

If you’re for example just looking for backlinks, a pure intersection of the terms ‘site:foo.com links:bar.com’ is enough, there’s nothing to rank anything there, but things working the way they did, threads were allocated to ranking none the less, term positions were attempted to be retrieved, all a bunch of done that needed not to be done. 例如，如果你只是在寻找反向链接，那么对“site:foo.com links:bar.com”这些词进行纯粹的交集运算就足够了，这里根本不需要排名。但在之前的机制下，线程仍然会被分配去进行排名计算，并尝试检索词项位置，这些都是完全没必要做的额外工作。

So I added a path for unranked queries that just do dumb term intersections and a bare minimum of anything else. 所以我为无排名查询添加了一条路径，它只进行简单的词项交集运算，并将其他操作降至最低。

The bonus of this is that with some fiddling and the construction of a cursor that can map to multiple index partitions, unlike the ranked path, unranked queries can be exhaustively retrieved with minimal additional work. 这样做的好处是，通过一些调整并构建一个可以映射到多个索引分区的游标（cursor），与排名路径不同，无排名查询可以用极少的额外工作实现穷举检索。

Keeping track of the position across multiple partitions to allow exhaustive retrieval took some thinking, but I designed a cursor that looks like this, that can be passed to the clients: 为了实现穷举检索，跟踪跨多个分区的位置需要一些思考，但我设计了一个如下所示的游标，可以传递给客户端：

28mbshfptkj.6ijoop7xty.43nivb3bn1.817ucldxy2x.90.12ria2fmp7a.32cydk0dei2t.73wrhi4igjr.53lb1o4iof7

Each period-delimited part starts with a single character mapping to an index partition, and then a string of alphanumerical values that’s a base 36-encoded document id, the document id to resume from on that partition. It’s short enough to pass on a query string or along an API query, which is really all that matters. 每个以句点分隔的部分都以一个映射到索引分区的字符开头，后面跟着一串字母数字值，这是经过 36 进制编码的文档 ID，表示在该分区上从哪个文档 ID 继续检索。它足够短，可以放在查询字符串或 API 查询中传递，这才是最重要的。

Implementing this allows up to half of the query load to go to the new endpoint, though many of these queries are driven by bots and scrapers traversing the /site viewer so the exact percentage varies quite significantly. 实现这一功能使得多达一半的查询负载可以转向新的端点，尽管其中许多查询是由遍历 /site 查看器的机器人和爬虫驱动的，因此确切的百分比会有很大波动。

The execution time of the new unranked queries usually sits in the ~5ms range, which is at least an order of magnitude below a full query. A big part of the win is that the unranked queries are single threaded, which frees up many threads from the execution pool to do more meaningful work. 新的无排名查询执行时间通常在 5 毫秒左右，这至少比完整查询快一个数量级。这一改进的一大优势在于无排名查询是单线程的，这从执行池中释放了许多线程去处理更有意义的工作。

Faster crawls

更快的抓取

The crawler’s ru… 爬虫的 ru…（原文中断）