From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings
From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings
从像素到键值对：噪声文档环境下基于大语言模型的键值提取综合基准测试
Large language models (LLMs) are increasingly used for structured information extraction from documents, yet their behavior under realistic OCR noise remains poorly understood. 大语言模型（LLMs）正越来越多地被用于从文档中提取结构化信息，然而，它们在真实 OCR 噪声环境下的表现仍缺乏深入研究。
We present a systematic benchmark of open-source instruction-tuned LLMs for key-value pair (KVP) extraction under both clean-text and noisy OCR conditions. 我们针对开源指令微调大语言模型，在纯文本和噪声 OCR 条件下进行键值对（KVP）提取的系统性基准测试。
We evaluate representative decoder-only models (Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek) on the FUNSD, CORD, and SROIE benchmarks using both Gold-text annotations and OCR outputs from PaddleOCR, EasyOCR, and Tesseract. 我们评估了具有代表性的仅解码器（decoder-only）模型（Gemma、Mistral、Qwen2.5、LLaMA 3 和 DeepSeek），测试集涵盖 FUNSD、CORD 和 SROIE，并分别使用了黄金标准文本标注以及来自 PaddleOCR、EasyOCR 和 Tesseract 的 OCR 输出结果。
A unified evaluation protocol isolates the effects of input quality, model design, and prompting under consistent conditions. 通过统一的评估协议，我们在一致的条件下隔离了输入质量、模型设计和提示词（prompting）对结果的影响。
The results show that modern LLMs act as strong semantic extractors when high-quality text is available, in some cases approaching supervised layout-aware systems. 结果表明，当提供高质量文本时，现代大语言模型表现出强大的语义提取能力，在某些情况下甚至接近监督式布局感知系统。
Under OCR noise, however, performance degrades substantially and performance gaps between models narrow as input corruption increases. 然而，在存在 OCR 噪声的情况下，模型性能会大幅下降，且随着输入损坏程度的加剧，不同模型之间的性能差距会缩小。
Across all datasets, extraction performance is governed by two factors: semantic reasoning over text and preservation of textual fidelity under OCR noise. 在所有数据集中，提取性能主要受两个因素制约：对文本的语义推理能力，以及在 OCR 噪声下对文本保真度的维持能力。
While larger models improve results on clean text, these gains diminish under noisy inputs, where OCR quality becomes the dominant factor. 虽然更大的模型在纯文本上表现更好，但这些优势在噪声输入下会减弱，此时 OCR 质量成为决定性因素。
We also identify recurring failure modes, including key-value misalignment, hallucination, and numeric corruption. 我们还识别出了一些反复出现的失败模式，包括键值错位、幻觉以及数字损坏。
Our findings highlight the gap between clean-text evaluation and real-world deployment, emphasizing the need to jointly improve OCR quality, structural reasoning, and LLM-based semantic modeling. 我们的研究结果突显了纯文本评估与实际部署之间的差距，并强调了协同提升 OCR 质量、结构化推理能力以及基于大语言模型的语义建模能力的必要性。