Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Multimodal CoLRAG-TF：针对复杂 PDF 的三元组过滤检索系统
Abstract: Retrieval-augmented generation (RAG) over heterogeneous PDF collections remains challenging due to multimodal content, domain-specific terminology, and the need for multi-hop reasoning across dispersed evidence. We present Multimodal CoLRAG-TF, a four-axis fusion architecture that integrates dense text embeddings, BM25 keyword matching, knowledge-graph triple filtering, and image-based similarity for robust retrieval over complex documents.
摘要： 由于多模态内容、特定领域术语以及跨分散证据进行多跳推理的需求，针对异构 PDF 集合的检索增强生成（RAG）仍然极具挑战性。我们提出了 Multimodal CoLRAG-TF，这是一种四轴融合架构，集成了密集文本嵌入、BM25 关键词匹配、知识图谱三元组过滤和基于图像的相似度计算，从而实现了对复杂文档的稳健检索。
Our system constructs a multimodal index of 2,403 blocks extracted from 43 Japanese disaster lesson PDFs, supported by a hybrid OCR pipeline and LLM-based caption generation. To enhance compositional reasoning, we extract 11,414 OpenIE triples and index them with FAISS, enabling sub-second triple lookup and hierarchical propagation of relevance signals.
我们的系统构建了一个多模态索引，包含从 43 份日本灾害经验教训 PDF 中提取的 2,403 个数据块，并由混合 OCR 流水线和基于大语言模型（LLM）的标题生成技术提供支持。为了增强组合推理能力，我们提取了 11,414 个 OpenIE 三元组并使用 FAISS 进行索引，实现了亚秒级的三元组查找和相关性信号的层级传播。
A HippoRAG2-inspired coarse-to-fine retriever (volume $\to$ chapter $\to$ block) narrows the search space before final fusion scoring. Bayesian optimization over fusion weights reveals that the triple axis must dominate ($\alpha_\text{triple} = 0.44$) to counteract lexical bias and sustain multi-hop retrieval quality.
受 HippoRAG2 启发，我们采用了一种由粗到细的检索器（卷 $\to$ 章节 $\to$ 数据块），在最终融合评分前缩小搜索空间。通过对融合权重进行贝叶斯优化发现，三元组轴必须占据主导地位（$\alpha_\text{triple} = 0.44$），以抵消词汇偏差并维持多跳检索的质量。
Evaluated on a 457-pair benchmark, Multimodal CoLRAG-TF achieves a Retrieval Recall of 0.9909 and a 71.6% improvement in multi-hop answer similarity over single-hop queries. An image-to-lesson pipeline using a vision LLM further demonstrates the applicability of the approach to visual inputs. These results show that triple-filtered multimodal fusion is essential for structured reasoning over noisy, heterogeneous PDFs and provides a general framework applicable beyond the disaster domain.
在 457 对基准测试中，Multimodal CoLRAG-TF 的检索召回率达到了 0.9909，且在多跳答案相似度上比单跳查询提升了 71.6%。使用视觉大模型（Vision LLM）的“图像到经验教训”流水线进一步证明了该方法在视觉输入上的适用性。这些结果表明，三元组过滤的多模态融合对于在嘈杂、异构的 PDF 上进行结构化推理至关重要，并提供了一个可应用于灾害领域之外的通用框架。