Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
Computer Science > Computation and Language arXiv:2610.19003 (cs) [Submitted on 13 Oct 2026] Title: Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali. Authors: Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar.
计算机科学 > 计算与语言 arXiv:2610.19003 (cs) [提交于 2026 年 10 月 13 日] 标题：评估用于多语言抽象摘要的大型语言模型：英语和孟加拉语的综合研究。作者：Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar。
Abstract: Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets.
摘要：跨语言的抽象摘要仍然具有挑战性，特别是在低资源环境下，语言质量和事实一致性可能会出现偏差。我们提出了一个标准化的零样本基准测试，涵盖了八个闭源和开源权重的大型语言模型（LLM），并在大小匹配的英语和孟加拉语 XL-Sum 测试集上进行了评估。
Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization.
我们的评估结合了词汇和语义指标、无参考事实验证以及多维度的“大模型作为裁判”（LLM-as-a-Judge）评估，从而对多语言摘要的表面质量和事实可靠性进行了全面分析。
Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score.
结果显示出显著的跨语言差异：GPT-5.6 Luna 在多项英语词汇指标中领先，但在孟加拉语中并未保持同样的优势，在该语言中 LLaMA 3.3 70B 获得了最高的 ROUGE 和 BLEU 分数。Gemini 2.5 Flash 则获得了观测到的最高孟加拉语事实性得分。
Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.
此外，不同的评估方法产生了不同的模型排名，凸显了仅依赖单一指标进行多语言摘要评估的局限性。这些发现表明，强大的英语摘要性能并不一定能迁移到孟加拉语中，并强调了为低资源语言建立综合评估框架的必要性。