DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification
DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification
DS@GT ARC 在 CheckThat! 2026：基于大语言模型的推理轨迹排序与分组奖励建模的多语言数值声明验证
Abstract: Automated verification of numerical claims is a challenging problem, as it requires both language understanding and quantitative reasoning. This paper describes our system for CLEF 2026 CheckThat! Task 2, which focuses on ranking reasoning traces generated by large language models (LLMs) and predicting a final verdict for numerical claims in English and Arabic.
摘要： 数值声明的自动验证是一个极具挑战性的问题，因为它既需要语言理解能力，也需要定量推理能力。本文介绍了我们为 CLEF 2026 CheckThat! 任务 2 开发的系统，该系统专注于对大语言模型（LLM）生成的推理轨迹进行排序，并对英语和阿拉伯语的数值声明预测最终结论。
We explore two approaches. The first approach fine-tunes an LLM-based verifier using LoRA to score each reasoning trace independently as a binary classification problem, and selects the final verdict using Best-of-N selection. We further experiment with adaptive sub-claim decomposition to break complex claims into simpler parts before verification.
我们探索了两种方法。第一种方法使用 LoRA 对基于 LLM 的验证器进行微调，将每个推理轨迹独立地视为二分类问题进行评分，并利用“Best-of-N”（N选最优）策略选择最终结论。我们还尝试了自适应子声明分解，即在验证前将复杂声明拆解为更简单的部分。
The second approach uses a lightweight TF-IDF reward model with handcrafted numeric and temporal overlap features to score traces, and aggregates scores by verdict group to determine the final prediction. For Arabic, we compare a general multilingual model against AraBERT, a language-specific model pretrained on Arabic text.
第二种方法使用轻量级的 TF-IDF 奖励模型，结合手工设计的数值和时间重叠特征来对轨迹进行评分，并通过结论分组聚合分数来确定最终预测。针对阿拉伯语，我们将通用多语言模型与 AraBERT（一种在阿拉伯语文本上预训练的特定语言模型）进行了对比。
Our results show that the LLM-based approach outperforms the lightweight reward model on most metrics, particularly Recall@5, while the reward-based approach shows stronger performance on the Conflicting class. Sub-claim decomposition did not improve performance, suggesting that claim splitting introduces noise rather than aiding reasoning. For Arabic, AraBERT outperforms the multilingual baseline across most metrics.
实验结果表明，基于 LLM 的方法在大多数指标（尤其是 Recall@5）上优于轻量级奖励模型，而基于奖励的方法在“冲突”（Conflicting）类别上表现更强。子声明分解并未提升性能，这表明拆解声明反而引入了噪声，而非辅助推理。对于阿拉伯语，AraBERT 在大多数指标上均优于多语言基准模型。