Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment

分析多语言可读性评估的传统方法与神经方法

Abstract: Transformer-based models excel at Automatic Readability Assessment (ARA), yet feature-based models remain in active use because their predictions tie back to linguistic properties. This matters because readability labels are subjective and rater-dependent, so high accuracy on noisy ground truth may reflect surface patterns rather than the linguistic structure that defines difficulty.

摘要： 基于 Transformer 的模型在自动可读性评估（ARA）方面表现出色，但基于特征的模型仍在使用中，因为它们的预测结果可以追溯到语言属性。这一点至关重要，因为可读性标签具有主观性和评估者依赖性，因此在嘈杂的基准数据上获得的高准确率可能反映的是表面模式，而非定义难度的语言结构。

We test whether transformers internalize the same features as traditional models across Arabic, English, French, Hindi, and Russian using the ReadMe++ dataset. Shapley Additive Explanations (SHAP) identify the features driving traditional classifiers, which we then use as TCAV concept sets to probe multilingual XLM-R and language-specific encoders.

我们使用 ReadMe++ 数据集，测试了 Transformer 模型在阿拉伯语、英语、法语、印地语和俄语中是否内化了与传统模型相同的特征。我们利用 Shapley 加性解释（SHAP）识别驱动传统分类器的特征，并将其作为 TCAV 概念集，用于探测多语言 XLM-R 模型和特定语言编码器。

Transformers recover surface-length, syntactic, and lexical-diversity signals, and reflect the ordinal CEFR structure of the traditional models. Alignment varies by model family, language, and layer, with language-specific encoders tracking traditional models more clearly than XLM-R. High linear separability does not always imply directional influence, limiting linear probing for count-based readability features.

Transformer 模型能够恢复表面长度、句法和词汇多样性信号，并反映出传统模型中序数 CEFR（欧洲共同语言参考标准）的结构。这种对齐程度因模型系列、语言和层级而异，其中特定语言编码器比 XLM-R 更清晰地追踪了传统模型。高线性可分性并不总是意味着方向性影响，这限制了对基于计数的各种可读性特征进行线性探测的效果。