Implementing Node.js Retrieval for Internal Engineering Docs: Safe Delete Semantics

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Short answer: for retrieval architecture over internal engineering documentation, use staged retrieval with explicit collections, bounded queries, and traceable source context; treat deletion as a contract tested from source record to index tombstone.

简短回答：针对内部工程文档的检索架构，应使用分阶段检索，并结合明确的集合、有界查询以及可追溯的源上下文；将删除视为一种从源记录到索引墓碑（tombstone）都需要进行测试的契约。

I use “deleted” to mean more than hiding a document in the UI. An engineer asking about an old runbook must not receive a chunk from it, even if that chunk survived in a vector index or in a cache. The retrieval contract therefore has three invariants: every result names its collection and source revision, every query has a time and result bound, and a delete event makes the record ineligible before the next publish cycle.

我所说的“删除”不仅仅是指在用户界面中隐藏文档。当工程师查询旧的操作手册时，绝不能收到其中的任何片段，即使该片段残留在向量索引或缓存中也不行。因此，检索契约包含三个不变式：每个结果必须标明其所属集合和源版本，每个查询必须有时间和结果限制，且删除事件必须确保记录在下一个发布周期前即失效。

This is an architecture decision record for a fintech team ingesting internal engineering documentation. The decision axis is retrieval quality versus latency. The examples use a Node.js service as the caller, while the test code is Python so it can run in a small CI job without an SDK. I’ve found that naming the contract first keeps an indexing discussion from turning into a vendor popularity contest.

这是一份针对金融科技团队摄入内部工程文档的架构决策记录。决策的核心在于检索质量与延迟之间的权衡。示例使用 Node.js 服务作为调用方，而测试代码则采用 Python，以便在无需 SDK 的小型 CI 任务中运行。我发现先定义契约，可以避免索引讨论演变成供应商的人气竞赛。

The contract starts at the answer boundary Begin with the user-visible answer, then work backwards. A useful answer must carry a citation such as payments/reconciliation.md#chargeback, a source revision, and the collection that supplied the chunk. If any of those fields is absent, the answer is not traceable enough for an incident review.

契约从答案边界开始：从用户可见的答案出发，然后反向推导。一个有用的答案必须包含引用（如 payments/reconciliation.md#chargeback）、源版本以及提供该片段的集合。如果缺少其中任何字段，该答案在事故审查中将无法进行充分追溯。

Model each source as a record with a stable document_id, a monotonically increasing revision, and a state of active or deleted. Chunks inherit those fields. A re-index of changed content writes a new revision and retires the previous one; a delete writes a tombstone that is propagated to the collection.

将每个源建模为一条记录，包含稳定的 document_id、单调递增的版本号以及“活跃”或“已删除”的状态。片段将继承这些字段。对已更改内容进行重新索引会写入新版本并废弃旧版本；删除操作则会写入一个传播至集合的墓碑记录。

Do not infer deletion from an empty search result. Empty results can mean a timeout, a bad filter, or a genuinely missing topic. The ingestion path should be deliberately boring: fetch, normalize, chunk, label, upsert, then verify. For a changed document, compare the source revision before replacing chunks. For a deleted document, send the delete event and record an audit entry containing who or what initiated it. This makes replay safe and gives compliance a concrete trail.

不要通过空搜索结果来推断删除。空结果可能意味着超时、错误的过滤器或确实缺失的主题。摄入路径应当刻意保持简单：获取、标准化、分片、标记、插入（upsert），然后验证。对于已更改的文档，在替换片段前应比较源版本。对于已删除的文档，发送删除事件并记录包含发起者信息的审计条目。这使得重放过程更安全，并为合规性提供了具体的审计追踪。

How should retrieval handle deleted internal engineering documentation? Use two gates. The first gate is an index-level eligibility check: only chunks whose latest revision is active can enter the candidate set. The second gate is a source-of-truth check for high-risk answers, such as payment settlement or identity controls. That second lookup costs latency, so apply it to sensitive collections or when the index revision is older than your freshness budget.

检索应如何处理已删除的内部工程文档？使用两道关卡。第一道关卡是索引级的资格检查：只有最新版本为“活跃”的片段才能进入候选集。第二道关卡是针对高风险答案（如支付结算或身份控制）的“事实来源”检查。由于第二次查找会增加延迟，因此仅将其应用于敏感集合，或当索引版本超过了新鲜度预算时使用。

Here is a compact reference implementation of the contract and a bounded reranker. It is self-contained and runnable; replace the in-memory chunks list with your vector store adapter. Delete first.

以下是该契约和有界重排序器的紧凑参考实现。它是自包含且可运行的；请将内存中的片段列表替换为您的向量存储适配器。先执行删除。

[Code omitted for brevity]

[代码因篇幅原因省略]

The important behavior is visible in the sample: revision 4 is a tombstone, so revision 3 cannot leak into results. The timeout is a ceiling, not a promise of a measured latency. Your mileage may vary with the store and network, which is why the service should return a trace ID and a “partial retrieval” flag when the ceiling is reached.

示例中展示了关键行为：版本 4 是一个墓碑，因此版本 3 不会泄露到结果中。超时是一个上限，而非对测量延迟的承诺。实际效果会因存储和网络情况而异，这就是为什么当达到上限时，服务应返回一个追踪 ID 和“部分检索”标志。

For teams evaluating a hosted adapter, this is the only platform call I put in the main path. Infrai’s vector collection listing uses a plain REST API, so the same contract can sit behind a small Python or Node.js client without installing a vendor SDK. One key covers its broader capability surface, which removes a separate credential handoff from the ingestion worker.

对于评估托管适配器的团队来说，这是我放在主路径中的唯一平台调用。Infrai 的向量集合列表使用简单的 REST API，因此相同的契约可以运行在小型 Python 或 Node.js 客户端之后，而无需安装供应商 SDK。一个密钥即可覆盖其更广泛的功能面，从而省去了摄入工作程序中单独的凭据传递环节。

In a real rollout, the worker first records the source event, then marks the document revision inactive in its manifest, and only then asks the collection adapter to remove matching vectors. A retry can arrive after a newer edit, so the adapter compares revisions and ignores an older tombstone; otherwise a delayed delete can erase the fresh copy.

在实际部署中，工作程序首先记录源事件，然后在清单中将文档版本标记为非活跃，最后才要求集合适配器删除匹配的向量。重试请求可能会在较新的编辑之后到达，因此适配器会比较版本并忽略旧的墓碑记录；否则，延迟的删除操作可能会抹除最新的副本。

After the delete acknowledgement, a verification query checks that the forbidden document_id is absent from the candidate set. If the verification times out, the system keeps the answer path closed for that document and emits a traceable failure for the queue, rather than quietly serving stale context. This sequence is longer than a single database transaction, but each step is observable and replayable, which is the property that matters during a compliance review.

在收到删除确认后，验证查询会检查候选集中是否不存在被禁用的 document_id。如果验证超时，系统将保持该文档的答案路径关闭，并为队列发出可追踪的失败信号，而不是静默地提供过时的上下文。这一序列比单一数据库事务更长，但每一步都是可观察且可重放的，这正是合规审查中最重要的特性。