Interpretable Multimodal Classification with Linear Discriminant Tree Ensembles
Interpretable Multimodal Classification with Linear Discriminant Tree Ensembles
基于线性判别树集成的可解释多模态分类
Abstract: Multimodal affect and behaviour classifiers that fuse heterogeneous text, audio, and visual streams must simultaneously achieve competitive accuracy and produce human-understandable explanations of the cues driving their decisions — a dual objective that current high-capacity models, notably Transformers, only partially address.
摘要： 融合异构文本、音频和视觉流的多模态情感与行为分类器，必须在实现具有竞争力的准确性的同时，对驱动其决策的线索提供人类可理解的解释——这是一个当前高容量模型（尤其是 Transformer）仅能部分解决的双重目标。
While Transformers attain strong predictive performance, their distributed representations and deep nonlinearity make it difficult to assign meaningful importance weights to individual multimodal features, limiting their use in trust-sensitive applications such as clinical affect monitoring and educational assessment.
尽管 Transformer 具有强大的预测性能，但其分布式表示和深层非线性使得为单个多模态特征分配有意义的权重变得困难，从而限制了它们在临床情感监测和教育评估等对信任度敏感的应用中的使用。
We address this gap by developing a framework based on tree-based ensembles that balances accuracy and interpretability. The framework encodes each modality into tokens, extracts and clusters concepts to reduce dimensionality, routes the fused modalities through tree-based ensemble classifiers, and interprets trends using a novel modified feature importance metric.
我们通过开发一个基于树集成（tree-based ensembles）的框架来弥补这一差距，该框架平衡了准确性和可解释性。该框架将每种模态编码为标记（tokens），提取并聚类概念以降低维度，通过树集成分类器路由融合后的模态，并使用一种新颖的改进特征重要性指标来解释趋势。
The modified importance reduces the influence of the negative class in binary classification tasks, thereby improving indicator or marker detection. The proposed tree-based ensembles — Linear Discriminant Tree (LDT), Linear Discriminant Forest (LDF), and Linear Discriminant AdaBoost (LDAB) — achieve F1-mod gains of 4.3% over the Multimodal Transformer and accuracy gains of 3.0% over the primary interpretable multimodal baseline, Interpretable Multimodal Routing (IMR).
这种改进后的重要性指标减少了二分类任务中负类（negative class）的影响，从而提高了指标或标记的检测能力。所提出的树集成模型——线性判别树（LDT）、线性判别森林（LDF）和线性判别 AdaBoost（LDAB）——在 F1-mod 指标上比多模态 Transformer 提升了 4.3%，在准确率上比主要的可解释多模态基准模型“可解释多模态路由”（IMR）提升了 3.0%。
The proposed multimodal feature importance extracts salient inter-modal concepts with substantially higher human-annotator agreement scores than default feature importance (62.2% vs. 43.2% on IEMOCAP; 46.7% vs. 32.1% on CMU-MOSI).
所提出的多模态特征重要性提取出的显著跨模态概念，其人类标注者的一致性得分显著高于默认的特征重要性指标（在 IEMOCAP 数据集上为 62.2% 对 43.2%；在 CMU-MOSI 数据集上为 46.7% 对 32.1%）。