Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey

通过专家混合模型（MoE）解决多模态学习挑战：综述

Abstract: Mixture-of-Experts (MoE) presents a naturally compatible and scalable framework for multimodal learning, demonstrating strong adaptability across diverse modalities and tasks. Despite its growing success, a comprehensive and systematic review on the MoE methods addressing multimodal challenges remains lacking. Existing surveys tend to evaluate either multimodal learning or MoE independently from method taxonomy, overlooking the unique interplay between them.

摘要： 专家混合模型（MoE）为多模态学习提供了一个天然兼容且可扩展的框架，在处理多种模态和任务时展现出强大的适应性。尽管其日益成功，但目前仍缺乏针对 MoE 方法如何解决多模态挑战的全面系统性综述。现有的综述往往将多模态学习或 MoE 作为独立的方法论进行评估，忽略了两者之间独特的相互作用。

This survey fills that gap by answering a central question: How does MoE effectively resolve multimodal challenges? We approach this from three key perspectives: (1) MoE as an Efficient Multimodal Engine: enabling scalable multimodal modeling by decoupling computational cost from parameter growth and mitigating modality redundancy through selective expert activation; (2) MoE as a Multimodal Representation Learner: integrating complementary multi-opinion expert knowledge to enrich alignment and interaction representations; and (3) MoE as a Multimodal Adapter: providing a modular and flexible mechanism to model imperfect data scenarios such as modality imbalance and missing modality.

本综述通过回答一个核心问题填补了这一空白：MoE 如何有效解决多模态挑战？ 我们从三个关键视角进行探讨：(1) 作为高效多模态引擎的 MoE： 通过将计算成本与参数增长解耦，并利用选择性专家激活来减轻模态冗余，从而实现可扩展的多模态建模；(2) 作为多模态表征学习器的 MoE： 整合互补的多观点专家知识，以丰富对齐和交互表征；以及 (3) 作为多模态适配器的 MoE： 提供一种模块化且灵活的机制，用于建模模态不平衡和模态缺失等不完美数据场景。

Through our extensive literature review, we identify critical research gaps, including interpretable routing, expert communication, modality integration, and lifelong multimodal learning. We position this survey as a foundation for future research toward interpretable and sustainable multimodal Mixture-of-Experts systems.

通过广泛的文献综述，我们确定了关键的研究空白，包括可解释的路由机制、专家间通信、模态整合以及终身多模态学习。我们将本综述定位为未来研究可解释且可持续的多模态专家混合系统的基石。