The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You

查看原文 / View Original 编辑 / Edit

The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You

静默向量污染漏洞：为什么你的并发嵌入（Embeddings）可能在欺骗你

TL;DR: If you run concurrent inference (e.g., via OpenVINO AsyncInferQueue or custom threading) for text/code embeddings, your tests might show 0 exceptions and 0 errors, while silently returning embeddings belonging to other inputs in the batch. Here is how we caught a subtle race condition using a cosine-similarity contamination test. 简而言之： 如果你为文本/代码嵌入运行并发推理（例如通过 OpenVINO 的 AsyncInferQueue 或自定义线程），你的测试可能会显示 0 异常和 0 错误，但却静默地返回了批处理中属于其他输入的嵌入向量。以下是我们如何利用余弦相似度污染测试捕获这一隐蔽竞态条件的过程。

The Setup

环境设置

We use OpenVINO with an INT8 quantized E5-small model for in-process code embedding. To maximize throughput on multi-core CPUs, we set up an asynchronous inference queue (AsyncInferQueue) with jobs=4. In standard unit testing, everything looked pristine: 我们使用 OpenVINO 和 INT8 量化的 E5-small 模型进行进程内代码嵌入。为了最大化多核 CPU 上的吞吐量，我们设置了一个 jobs=4 的异步推理队列（AsyncInferQueue）。在标准单元测试中，一切看起来都很完美：

  • All infer jobs completed with exit code 0
  • No None values or zero-filled tensors were returned
  • Latency and throughput were great (~37 chunks/sec on Ryzen 5600)
  • 所有推理任务均以退出代码 0 完成
  • 没有返回 None 值或全零张量
  • 延迟和吞吐量表现优异（在 Ryzen 5600 上约为 37 个分块/秒）

However, during end-to-end RAG retrieval tests, we noticed weird semantic anomalies: searching for authentication logic would occasionally return chunks related to database migrations or UI components with unreasonably high confidence. 然而，在端到端的 RAG 检索测试中，我们注意到了一些奇怪的语义异常：搜索身份验证逻辑时，偶尔会以极高的置信度返回与数据库迁移或 UI 组件相关的分块。

The Bug: Silent Contamination

漏洞：静默污染

The root cause was a subtle race condition in callback/userdata mapping inside the async wrapper. Because the inputs were processed concurrently across multiple execution streams, a shared user-data context wasn’t strictly isolated per inference request. When Request A (auth.py) and Request B (payment.py) were scheduled back-to-back: 根本原因是异步包装器内部回调/用户数据映射中存在一个隐蔽的竞态条件。由于输入是在多个执行流中并发处理的，共享的用户数据上下文并未针对每个推理请求进行严格隔离。当请求 A (auth.py) 和请求 B (payment.py) 被连续调度时：

  • Both requests succeeded without throwing exceptions
  • The output tensor for Request A was mapped to the metadata/chunk wrapper of Request B
  • The resulting vector was syntactically valid and non-zero, but it represented the wrong text input
  • 两个请求均成功执行且未抛出异常
  • 请求 A 的输出张量被映射到了请求 B 的元数据/分块包装器中
  • 生成的向量在语法上是有效的且非零，但它代表了错误的文本输入

Standard assertion tests like assert output_vector is not None or assert output_vector.shape == (384,) passed 100% of the time. The pipeline was silently corrupting the vector store. 像 assert output_vector is not None 或 assert output_vector.shape == (384,) 这样的标准断言测试 100% 通过。该流水线正在静默地破坏向量数据库。

The Code

代码实现

# Before fix: shared results dict across concurrent calls
self._ov_results = {}

def _callback(request, userdata):
    # BUG: userdata is a global index (0, 1, 2, ...)
    # Two concurrent calls reuse the same indices!
    self._ov_results[userdata] = request.get_tensor().data
# 修复前：并发调用间共享结果字典
self._ov_results = {}

def _callback(request, userdata):
    # 漏洞：userdata 是一个全局索引 (0, 1, 2, ...)
    # 两个并发调用重用了相同的索引！
    self._ov_results[userdata] = request.get_tensor().data

The problem: userdata was a simple integer counter (0, 1, 2, …) that reset between embed_batch calls. When two calls overlapped, they wrote to the same dictionary keys. 问题在于：userdata 是一个简单的整数计数器（0, 1, 2, …），在每次 embed_batch 调用之间重置。当两个调用重叠时，它们会写入相同的字典键。

The Solution: Cosine Contamination Testing

解决方案：余弦污染测试

To catch this reliably in CI, we wrote an explicit cross-contamination test designed for concurrent embedding queues. 为了在 CI 中可靠地捕获此问题，我们编写了一个专门针对并发嵌入队列的显式交叉污染测试。

The Test Logic

测试逻辑

# 1. Semantically distinct inputs
samples = {
    "auth": "def authenticate_user(username, password_hash): return verify_jwt(token)",
    "sql": "SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user_id WHERE o.status = 'active'",
    # ...
}

# 2. Sequential baseline (ground truth)
baseline_vectors = {}
for key, text in samples.items():
    baseline_vectors[key] = await embedder.embed_single_sync(text)

# 3. High-concurrency stress test
# ... (randomized queue order)
async_results = await asyncio.gather(*async_tasks)

# 4. Verify identity & cross-isolation via Cosine Similarity
for expected_key, async_vec in zip(keys_order, async_results):
    # Self-similarity must be ~1.0
    # Cross-similarity must remain low (<0.6)
# 1. 语义上截然不同的输入
samples = {
    "auth": "def authenticate_user(username, password_hash): return verify_jwt(token)",
    "sql": "SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user_id WHERE o.status = 'active'",
    # ...
}

# 2. 顺序基准（真实值）
baseline_vectors = {}
for key, text in samples.items():
    baseline_vectors[key] = await embedder.embed_single_sync(text)

# 3. 高并发压力测试
# ... (随机化队列顺序)
async_results = await asyncio.gather(*async_tasks)

# 4. 通过余弦相似度验证身份和交叉隔离
for expected_key, async_vec in zip(keys_order, async_results):
    # 自相似度必须约为 1.0
    # 交叉相似度必须保持在低位 (<0.6)

Benchmark Results

基准测试结果

MetricUnpatchedPatchedExpected
Self-similarity (auth↔auth)0.34 ❌0.99 ✅>0.98
Cross-similarity (auth↔sql)0.98 ❌0.32 ✅<0.6

The unpatched queue showed 0 exceptions while vectors were completely swapped. 未修复的队列显示 0 异常，但向量却被完全调换了。

The Fix

修复方案

The fix was simple once we understood the problem: 一旦理解了问题所在，修复就变得很简单：

# After fix: isolated results per call
self._ov_results = {}

def _callback(request, userdata):
    # FIX: userdata is now (index, local_results_dict)
    # Each embed_batch call creates its own dict
    index, local_dict = userdata
    local_dict[index] = request.get_tensor().data
# 修复后：每个调用结果隔离
self._ov_results = {}

def _callback(request, userdata):
    # 修复：userdata 现在是 (index, local_results_dict)
    # 每个 embed_batch 调用都会创建自己的字典
    index, local_dict = userdata
    local_dict[index] = request.get_tensor().data

Each embed_batch call now creates its own isolated dictionary. The callback writes to the call-specific dict, not a shared global. No locks needed — complete isolation by design. 现在，每个 embed_batch 调用都会创建自己独立的字典。回调函数写入的是特定于调用的字典，而不是共享的全局字典。无需锁机制——通过设计实现了完全隔离。

Key Takeaways

关键要点

  • 0 exceptions ≠ Correctness. Silent data corruption doesn’t throw errors.

  • Valid shape ≠ Valid embedding. A (384,) tensor with non-zero floats can represent the wrong input.

  • Write cross-contamination tests. If you use async inference queues or multi-threading for vector generation, verify that Vector X actually belongs to Input X under concurrent load.

  • Cosine similarity is your friend. A simple similarity check between concurrent outputs and sequential baselines catches contamination that no other test detects.

  • 0 异常 ≠ 正确性。 静默数据损坏不会抛出错误。

  • 有效的形状 ≠ 有效的嵌入。 一个带有非零浮点数的 (384,) 张量可能代表了错误的输入。

  • 编写交叉污染测试。 如果你使用异步推理队列或多线程生成向量，请务必验证在并发负载下，向量 X 是否确实属于输入 X。

  • 余弦相似度是你的好帮手。 在并发输出和顺序基准之间进行简单的相似度检查，可以捕获其他任何测试都无法检测到的污染问题。

← 返回新闻列表 / Back to news list