TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding

TalkFa：波斯语对话生成与理解的统一基准

Abstract: Farsi, spoken by more than 120 million people, lacks a comprehensive benchmark for dialogue generation and understanding. We introduce TALKFA, a unified benchmark comprising three complementary datasets: (1) WIKI-FADIAL, 4.2K Wikipedia-grounded dialogues for knowledge-grounded generation; (2) DAILYDIALOG-FA, 6.6K dialogues annotated for dialogue acts and emotions; and (3) PLAYDIAL-FA, 2.1K theatrical dialogues with sentiment labels.

摘要： 波斯语拥有超过 1.2 亿使用者，但目前仍缺乏用于对话生成与理解的综合性基准。我们推出了 TALKFA，这是一个包含三个互补数据集的统一基准：(1) WIKI-FADIAL，包含 4.2K 条基于维基百科知识的对话，用于知识驱动的生成；(2) DAILYDIALOG-FA，包含 6.6K 条标注了对话行为和情感的对话；以及 (3) PLAYDIAL-FA，包含 2.1K 条带有情感标签的戏剧对话。

While LLMs assist data construction, every dialogue undergoes multi-stage review and revision by native Farsi speakers, and only the final human-approved dialogues are released. Experiments with six LLAMA and MISTRAL models show that LoRA substantially improves dialogue generation while requiring only 25-50% of the training data to recover over 90% of the final performance gains.

尽管大语言模型（LLM）辅助了数据构建，但每一条对话都经过了波斯语母语者的多阶段审查和修订，只有最终经人工核准的对话才会被发布。针对六种 LLAMA 和 MISTRAL 模型的实验表明，LoRA 在显著提升对话生成效果的同时，仅需 25-50% 的训练数据即可恢复超过 90% 的最终性能增益。

Across classification tasks, FABERT achieves the best dialogue-act performance, LORA-MISTRAL-7B performs best on emotion recognition, and MISTRAL-24B achieves the highest sentiment score. Human evaluation and independent external validation demonstrate the reliability of the benchmark, while comparisons with GPT-4.1 as an LLM judge reveal that automatic metrics substantially overestimate dialogue quality. Zero-shot evaluation with frontier LLMs further shows that TalkFa remains a challenging benchmark. We will release all datasets, annotation guidelines, code, and checkpoints.

在分类任务中，FABERT 在对话行为识别上表现最佳，LORA-MISTRAL-7B 在情感识别上表现最优，而 MISTRAL-24B 则获得了最高的情感得分。人工评估和独立的外部验证证明了该基准的可靠性，而与作为 LLM 裁判的 GPT-4.1 进行对比则显示，自动评估指标会显著高估对话质量。使用前沿大模型进行的零样本（Zero-shot）评估进一步表明，TalkFa 仍然是一个具有挑战性的基准。我们将发布所有数据集、标注指南、代码和模型检查点。