Multimodal Ambivalence and Hesitancy Recognition with Bidirectional Audio--Text Cross-Attention

Computer Science > Computation and Language arXiv:2610.26868 (cs) [Submitted on 16 Jul 2026] Title: Multimodal Ambivalence and Hesitancy Recognition with Bidirectional Audio—Text Cross-Attention Authors: TingTing Gu, JinCheng Luo, WenQiu Guo.

计算机科学 > 计算与语言 arXiv:2610.26868 (cs) [提交于 2026 年 7 月 16 日] 标题：基于双向音视频交叉注意力的多模态矛盾与犹豫识别 作者：TingTing Gu, JinCheng Luo, WenQiu Guo。

Abstract: Ambivalence and hesitancy (A/H) are subtle behavioural states that can be expressed through both what a person says and how it is spoken. This paper presents an audio—text framework for video-level A/H recognition. Full-video transcripts and speech waveforms are encoded using pre-trained multilingual BERT and wav2vec 2.0, respectively, and projected into a shared representation space.

摘要：矛盾与犹豫（A/H）是一种微妙的行为状态，可以通过一个人所说的话及其表达方式表现出来。本文提出了一种用于视频级 A/H 识别的音视频框架。全视频转录文本和语音波形分别使用预训练的多语言 BERT 和 wav2vec 2.0 进行编码，并投影到一个共享的表示空间中。

Two stacked bidirectional cross-attention blocks allow linguistic tokens to attend to acoustic frames and acoustic frames to attend to linguistic context without requiring explicit temporal alignment. The resulting sequences are aggregated through masked mean pooling and combined using concatenation, element-wise products, and absolute differences for binary classification.

两个堆叠的双向交叉注意力模块允许语言标记关注声学帧，并允许声学帧关注语言上下文，而无需显式的时间对齐。所得序列通过掩码平均池化进行聚合，并使用拼接、逐元素乘积和绝对差值进行组合，以进行二分类。

We evaluate the model on the subject-independent split of the Behavioural Ambivalence/Hesitancy (BAH) dataset. The proposed system achieves 72.19% accuracy, 75.00% precision, 81.13% recall, 77.95% A/H F1-score, and 70.16% Macro-F1 on the test set. These results demonstrate the potential of jointly modelling linguistic and acoustic information for automatic A/H recognition.

我们在行为矛盾/犹豫（BAH）数据集的主体独立划分上评估了该模型。所提出的系统在测试集上达到了 72.19% 的准确率、75.00% 的精确率、81.13% 的召回率、77.95% 的 A/H F1 分数以及 70.16% 的宏观 F1 分数。这些结果证明了联合建模语言和声学信息以实现自动 A/H 识别的潜力。