MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis
MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis
MIDAS：用于不完整多模态情感分析的互信息解耦与不确定性感知融合框架
Abstract: Most existing multimodal sentiment analysis approaches assume access to complete multimodal inputs. However, real-world applications frequently encounter incomplete or corrupted modalities, posing a critical challenge. Although several methods have been proposed to tackle this issue, they mainly rely on data imputation and heuristic coordination constraints, which fail to effectively extract and leverage task-relevant information from the incomplete multimodal data.
摘要： 大多数现有的多模态情感分析方法都假设可以获取完整的多模态输入。然而，现实世界的应用中经常会遇到模态缺失或损坏的情况，这构成了严峻的挑战。尽管已有多种方法被提出以解决这一问题，但它们主要依赖于数据填充和启发式协调约束，无法有效地从不完整的多模态数据中提取并利用与任务相关的信息。
To address this challenge, we propose a unified framework termed Mutual Information Disentanglement with uncertainty-Aware fuSion (MIDAS), which effectively restructures multimodal representations under incomplete conditions. MIDAS adopts a variational modeling strategy to represent each modality with multivariate Gaussian latent variables and further decomposes them into shared and exclusive factors.
为了应对这一挑战，我们提出了一个统一的框架，即“互信息解耦与不确定性感知融合”（MIDAS），该框架能在不完整条件下有效地重构多模态表示。MIDAS 采用变分建模策略，利用多元高斯潜变量来表示每个模态，并进一步将其分解为共享因子和独有因子。
To obtain reliable representations, we design a minimax objective that minimizes the mutual information between shared and exclusive spaces for stable disentanglement, while maximizing the mutual information among shared spaces across modalities to enhance semantic alignment. In addition, an uncertainty-aware fusion mechanism is introduced, where posterior variance is leveraged as a reliability indicator to adaptively weight latent features during fusion, ensuring robust integration even when modalities are incomplete.
为了获得可靠的表示，我们设计了一个极小极大目标函数：一方面最小化共享空间与独有空间之间的互信息以实现稳定的解耦，另一方面最大化跨模态共享空间之间的互信息以增强语义对齐。此外，我们引入了一种不确定性感知融合机制，利用后验方差作为可靠性指标，在融合过程中自适应地对潜特征进行加权，从而确保即使在模态不完整的情况下也能实现稳健的集成。
Extensive experiments on three widely used datasets show that MIDAS achieves strong and consistent performance gains over competitive baselines across a wide range of incomplete settings, demonstrating its effectiveness and robustness for incomplete data scenarios.
在三个广泛使用的数据集上进行的广泛实验表明，MIDAS 在各种不完整设置下均优于竞争基线，取得了显著且一致的性能提升，证明了其在不完整数据场景下的有效性和鲁棒性。