Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

通过蒸馏与生成式多语言转录本的跨模态集成进行音频情感分析

Abstract: Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models to solve the task, but it remains unclear if such models can take all aspects into account.

摘要： 从语音中自动识别情感（正面或负面）是一项具有挑战性的任务，它既需要分析语音语调，也需要解读所表达的词汇。近期的解决方案依赖于音频基础模型来解决这一任务，但目前尚不清楚这些模型是否能兼顾所有方面。

To this end, we propose a multimodal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated via an automatic speech recognition (ASR) tool. Moreover, we create multiple text modalities by automatically translating the transcripts into multiple languages via machine translation tools.

为此，我们提出了一种多模态解决方案，通过跨模态 Transformer 集成音频和文本信息，其中文本转录本是通过自动语音识别（ASR）工具自动生成的。此外，我们通过机器翻译工具将这些转录本自动翻译成多种语言，从而创建了多种文本模态。

Audio and multilingual text features are combined via a cascaded architecture comprising cross-modal transformer blocks that integrate modalities one by one. We further distill knowledge from the multimodal model, called teacher, into a unimodal (audio only) model, called student.

音频和多语言文本特征通过一种级联架构进行组合，该架构包含逐个集成模态的跨模态 Transformer 模块。我们进一步将多模态模型（称为教师模型）的知识蒸馏到单模态（仅音频）模型（称为学生模型）中。

We conduct experiments on a large-scale dataset, demonstrating that the automatically generated textual information can bring significant performance boosts in multimodal sentiment polarity classification. Our ablation study confirms that both automatic transcripts and automatic translations are helpful.

我们在大规模数据集上进行了实验，证明自动生成的文本信息可以显著提升多模态情感极性分类的性能。我们的消融研究证实，自动转录本和自动翻译均对性能提升有所帮助。

Moreover, we show that the audio-only model can be enhanced via distillation, boosting performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at this https URL.

此外，我们展示了仅音频模型可以通过蒸馏得到增强，在推理过程中无需任何额外的计算开销即可提升性能。为了复现上述结果，我们已在链接处公开了代码。