Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages

Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages

评估针对达罗毗荼语系的专用单语模型与联合多语因果模型

Abstract: Dravidian languages, mainly Tamil, Telugu, Kannada, and Malayalam make up only a small part of the data used to train multilingual language models, so it’s not clear how much per-language ability these models actually keep. 摘要： 达罗毗荼语系（主要包括泰米尔语、泰卢固语、卡纳达语和马拉雅拉姆语）在训练多语种语言模型所使用的数据中仅占很小一部分，因此这些模型究竟保留了多少针对特定语言的能力尚不明确。

I have trained five GPT-2 architecture models from scratch to compare four monolingual models (one each for Tamil, Telugu, Kannada, and Malayalam, each with its own 32K-vocabulary subword tokenizer) against one multilingual model sharing a 64K-vocabulary subword tokenizer across all four languages. 我从零开始训练了五个 GPT-2 架构模型，旨在对比四种单语模型（分别为泰米尔语、泰卢固语、卡纳达语和马拉雅拉姆语各训练一个模型，每个模型拥有独立的 32K 词汇量子词分词器）与一个在四种语言间共享 64K 词汇量子词分词器的多语模型。

All the 5 models are trained on cleaned CC-100, Wikipedia, and Samanantar data. 这五个模型均使用经过清洗的 CC-100、维基百科（Wikipedia）和 Samanantar 数据集进行训练。

I have tested the models on perplexity, bits-per-byte, tokenizer efficiency, and fine-tuning results which are compared against mGPT. 我通过困惑度（perplexity）、每字节比特数（bits-per-byte）、分词器效率以及微调结果对这些模型进行了测试，并将其与 mGPT 进行了对比。

The monolingual models outperform mGPT on sentiment classification and named entity recognition, and their tokenizers proved more efficient than the shared multilingual model across all the languages tested. 实验结果表明，单语模型在情感分类和命名实体识别任务上优于 mGPT，且在所有测试语言中，其分词器的效率均高于共享的多语模型。