Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition

用于矛盾与犹豫识别的模态差异 Transformer

Abstract: Ambivalence and hesitancy (A/H) are affective states in which individuals express contradictory signals across facial, vocal, and linguistic channels. Automatically recognising A/H in clinical videos requires detecting cross-modal disagreement — the signal that standard fusion methods suppress.

摘要： 矛盾与犹豫（A/H）是一种情感状态，表现为个体在面部、语音和语言通道中表达出相互矛盾的信号。在临床视频中自动识别 A/H 需要检测跨模态的不一致性——而这正是标准融合方法通常会抑制的信号。

Based on the conflict-aware multimodal fusion framework of Bekhouche et al., we present the Modality Discrepancy Transformer (MDT). MDT enriches the original 6-token design to a 9-token representation comprising three modality embeddings, three absolute-difference features, and three Hadamard-product discrepancy features learned through linear projections.

基于 Bekhouche 等人的冲突感知多模态融合框架，我们提出了模态差异 Transformer（MDT）。MDT 将原始的 6-token 设计扩展为 9-token 表示，包含三个模态嵌入、三个绝对差特征，以及通过线性投影学习到的三个 Hadamard 积差异特征。

These nine tokens undergo Transformer self-attention, with FiLM-based text-conditioned modulation and LoRA fine-tuning as core architectural components. A text-guided late fusion branch blends a text-only auxiliary head with the full multimodal output at inference.

这九个 token 经过 Transformer 自注意力机制处理，并以基于 FiLM 的文本条件调制和 LoRA 微调作为核心架构组件。在推理阶段，一个文本引导的后期融合分支将仅包含文本的辅助头与完整的多模态输出进行融合。

On the BAH dataset from the 3rd ABAW Challenge, MDT achieves 0.7408 Macro F1 on the labelled test split and 0.7368 on the private leaderboard, outperforming the strongest published baseline by over 10 points while training in under 20 minutes on a single GPU.

在第三届 ABAW 挑战赛的 BAH 数据集上，MDT 在标记测试集上达到了 0.7408 的 Macro F1 分数，在私有排行榜上达到了 0.7368，比目前最强的已发表基准高出 10 个百分点以上，且在单块 GPU 上的训练时间不到 20 分钟。