What Is Turkish-Language AI? Tokenizers, Training Data, and Language Model Development

什么是土耳其语人工智能？分词器、训练数据与语言模型开发

A Turkish-language interface and an artificial intelligence system developed around the linguistic structure of Turkish are not the same thing. This article examines the difference in light of academic research. 土耳其语界面与围绕土耳其语语言结构开发的人工智能系统并非一回事。本文将结合学术研究探讨二者之间的区别。

Turkish-language AI is not simply software with Turkish menus or a system capable of answering questions in Turkish. More precisely, it refers to AI systems that represent Turkish text efficiently, are trained or adapted using natural Turkish data, account for the language’s morphology and usage contexts, and are evaluated through Turkish-specific benchmarks. 土耳其语人工智能不仅仅是带有土耳其语菜单的软件，也不是仅仅能够用土耳其语回答问题的系统。更准确地说，它指的是那些能够高效表示土耳其语文本、使用自然土耳其语数据进行训练或适配、充分考虑该语言的形态学和使用语境，并通过土耳其语特定基准进行评估的 AI 系统。

This distinction matters because the ability of a multilingual model to generate Turkish text does not necessarily mean that it processes Turkish as efficiently as English or performs reliably on tasks grounded in the cultural, institutional, and linguistic context of Türkiye. 这种区分至关重要，因为一个多语言模型能够生成土耳其语文本，并不一定意味着它处理土耳其语的效率能达到英语水平，也不意味着它在基于土耳其文化、制度和语言背景的任务中表现可靠。

Research indicates that tokenizer selection can affect training costs and downstream task performance, that data quality may be just as important as data volume, and that Turkish requires independent evaluation datasets with linguistic and cultural validity. 研究表明，分词器的选择会影响训练成本和下游任务性能；数据质量可能与数据量同样重要；且土耳其语需要具备语言和文化有效性的独立评估数据集。

The Brief Answer: What Defines Turkish-Language AI?

简要回答：什么是土耳其语人工智能？

Four layers should be considered together: 应综合考虑以下四个层面：

1. The tokenizer and vocabulary
2. 分词器与词汇表
3. Turkish training data
4. 土耳其语训练数据
5. The model’s training or adaptation method
6. 模型的训练或适配方法
7. Turkish-specific evaluation
8. 土耳其语专项评估

Improving any one of these layers can be beneficial. However, improving only one of them is not sufficient to demonstrate that a model genuinely understands Turkish. 改进其中任何一个层面都有益处。然而，仅改进其中一个层面不足以证明模型真正理解土耳其语。

What Is Turkish-Language AI, and What Is It Not?

什么是土耳其语人工智能，什么不是？

In everyday usage, the term “Turkish AI” may refer to three different kinds of systems. The first is a system in which a Turkish interface has been added to a model primarily trained in another language. The second is a multilingual model that is capable of communicating in Turkish. The third is a system that directly targets Turkish when making decisions about tokenization, training data, model adaptation, and evaluation. 在日常使用中，“土耳其语 AI”一词可能指代三种不同类型的系统。第一种是主要在其他语言上训练、但添加了土耳其语界面的模型；第二种是能够进行土耳其语交流的多语言模型；第三种是在分词、训练数据、模型适配和评估决策时直接针对土耳其语进行优化的系统。

From a technical perspective, the third definition is the most meaningful. This distinction also separates the concepts of being “locally developed” and being “proficient in Turkish.” A model may have been developed in Türkiye without its Turkish performance ever being demonstrated through independent evaluations. Conversely, a multinational model may perform well on some Turkish tasks because it was trained on substantial Turkish data and uses a tokenizer that represents Turkish efficiently. The model’s origin should therefore not be treated as a substitute for measurable language proficiency. 从技术角度来看，第三种定义最有意义。这种区分也将“本土开发”与“精通土耳其语”的概念分离开来。一个模型可能是在土耳其开发的，但其土耳其语性能从未通过独立评估得到验证。相反，一个跨国模型可能在某些土耳其语任务上表现良好，因为它使用了大量的土耳其语数据进行训练，并使用了能高效表示土耳其语的分词器。因此，模型的产地不应被视为衡量语言能力的替代指标。

What Is a Tokenizer, and Why Is It Critical for Turkish?

什么是分词器，为什么它对土耳其语至关重要？

Language models do not process text directly as complete words. Text is first divided into numerical units called tokens. Common tokenization methods such as Byte Pair Encoding, WordPiece, and Unigram construct a vocabulary by learning frequently occurring character sequences from training corpora. This process does not necessarily involve learning grammatical rules. It is generally based on statistical frequency. 语言模型不会直接将文本作为完整单词进行处理。文本首先被分割成称为“标记”（tokens）的数字单元。常见的分词方法（如 BPE、WordPiece 和 Unigram）通过从训练语料库中学习高频字符序列来构建词汇表。这个过程并不一定涉及学习语法规则，通常基于统计频率。

Turkish is a highly productive agglutinative language. A single written word such as: evlerinizdekilerden can encode a root together with plural, possessive, locative, nominalizing, and ablative information. A tokenizer does not always divide such structures along meaningful linguistic boundaries. As a result, the same root or suffix may be divided inconsistently across different words, and equivalent information may require longer token sequences in Turkish than in other languages. 土耳其语是一种高产的黏着语。一个书面单词（例如：evlerinizdekilerden）可以包含词根以及复数、所有格、方位格、名词化和离格信息。分词器并不总是能沿着有意义的语言边界来分割此类结构。结果，同一个词根或后缀在不同单词中可能被不一致地分割，且在土耳其语中表达相同信息所需的标记序列可能比其他语言更长。

A study by Ali et al. comparing 24 monolingual and multilingual models found that using English-centric tokenizers in multilingual models could substantially reduce downstream performance and increase training costs because of inefficient vocabulary allocation. However, the study also provides an important warning: simple tokenizer metrics such as fertility, meaning the number of tokens per word, and parity do not reliably predict overall model quality on their own. The conclusion should therefore not be that fewer tokens always produce a better model (Ali et al., 2024). Ali 等人对比 24 个单语和多语言模型的研究发现，在多语言模型中使用以英语为中心的分词器，会因词汇分配效率低下而大幅降低下游性能并增加训练成本。然而，该研究也给出了一个重要警告：简单的分词器指标（如“生育率”即每个单词的标记数量，以及奇偶校验）本身并不能可靠地预测模型的整体质量。因此，结论不应是“标记越少，模型越好”（Ali et al., 2024）。

Why Does Token Efficiency Affect Users?

为什么标记效率会影响用户？

• Cost: Many AI APIs charge according to the number of input and output tokens. A language that requires more tokens to represent the same information may therefore be more expensive to process.
• 成本： 许多 AI API 根据输入和输出的标记数量收费。因此，需要更多标记来表示相同信息的语言处理起来可能更昂贵。
• Context capacity: When the same information occupies more tokens, less space remains within the model’s context window for documents, instructions, or conversation history.
• 上下文容量： 当相同信息占用更多标记时，模型上下文窗口中留给文档、指令或对话历史的空间就会减少。
• Latency: Longer token sequences require more computational steps, particularly during output generation.
• 延迟： 更长的标记序列需要更多的计算步骤，尤其是在生成输出时。
• Linguistic representation: Token divisions that consistently conflict with morpheme boundaries may make it more difficult for a model to learn relationships between word roots and suffixes.
• 语言表示： 与语素边界持续冲突的分词方式，可能会使模型更难学习词根与后缀之间的关系。

A large-scale study published in 2025 reported that performance differences in morphologically complex languages cannot be explained solely through tokenizer alignment. Dataset size, data composition, and evaluation methodology must also be considered together (Arnett et al., 2025). The correct conclusion of the Turkish tokenization debate is therefore not that a morphology-aware tokenizer will solve every problem. Rather, tokenization should be understood as one of the fundamental design decisions in developing a Turkish-language model. 2025 年发表的一项大规模研究报告称，形态复杂的语言在性能上的差异不能仅通过分词器对齐来解释。数据集大小、数据构成和评估方法也必须综合考虑（Arnett et al., 2025）。因此，关于土耳其语分词争论的正确结论并不是“感知形态的分词器能解决所有问题”。相反，分词应被理解为开发土耳其语模型时的基本设计决策之一。

Why Is Turkish Training Data at Least as Important as the Tokenizer?

为什么土耳其语训练数据至少与分词器同样重要？

A language model does not learn a language by memorizing a grammar textbook. It learns from the patterns contained in its training data. When Turkish examples are limited or consist mainly of machine-translated material, the model may learn everyday speech, professional terminology, formal correspondence, regional expressions, and cultural references unevenly. Having a large amount of data is not sufficient on its own. Duplicated content, machine-generated text, corrupted character encoding, personal data, licensing problems, and domain imbalance can all reduce the quality. 语言模型不是通过背诵语法书来学习语言的，而是从训练数据中包含的模式中学习。当土耳其语示例有限或主要由机器翻译材料组成时，模型对日常用语、专业术语、正式信函、区域表达和文化引用的学习可能会不均衡。拥有大量数据本身是不够的。重复内容、机器生成的文本、损坏的字符编码、个人数据、许可问题以及领域不平衡都会降低质量。