Generalized Multimodal Foundation Model

Abstract: Making prediction with multimodal data is widely used in diverse scenarios. Existing multimodal fusion models, once deployed, can only handle predefined modalities (e.g., vision, text and audio) and single tasks, making it difficult to quickly adapt to new downstream applications. Therefore, a natural yet rather aggressive question arises, whether there exists a general multimodal fusion model that can be applied to arbitrary modality combinations and arbitrary prediction tasks.

摘要： 利用多模态数据进行预测已广泛应用于各种场景。现有的多模态融合模型一旦部署，通常只能处理预定义的模态（如视觉、文本和音频）及单一任务，这使得它们难以快速适应新的下游应用。因此，一个自然但颇具挑战性的问题随之产生：是否存在一种通用的多模态融合模型，能够应用于任意模态组合和任意预测任务？

We argue that a unified multimodal fusion model should not depend on specific modalities and instead encode transferable patterns of multimodal correlation. To this end, we propose a simple and effective learning paradigm based on training over the generation of large-scale synthetic multimodal datasets with diverse causal structures that formally characterize the generative processes of multimodal data in real world.

我们认为，统一的多模态融合模型不应依赖于特定模态，而应编码多模态相关性中的可迁移模式。为此，我们提出了一种简单有效的学习范式，该范式基于大规模合成多模态数据集的生成训练，这些数据集具有多样的因果结构，能够正式表征现实世界中多模态数据的生成过程。

Building on this framework, we propose the generalized multimodal foundation model, a unified foundation model for generalized multimodal learning. By constructing large-scale synthetic multimodal datasets with diverse correlation patterns, our model encodes transferable multimodal correlations during training and activates appropriate associations through in-context examples during inference.

基于此框架，我们提出了广义多模态基础模型（Generalized Multimodal Foundation Model），这是一个用于广义多模态学习的统一基础模型。通过构建具有多样相关模式的大规模合成多模态数据集，我们的模型在训练过程中编码了可迁移的多模态相关性，并在推理阶段通过上下文示例（in-context examples）激活相应的关联。

Extensive experiments on 18 real-world datasets spanning 12 modalities and 11 prediction tasks demonstrate that our model achieves competitive performance with specialized models without task-specific adaptation.

在涵盖 12 种模态和 11 项预测任务的 18 个真实世界数据集上进行的广泛实验表明，我们的模型无需针对特定任务进行调整，即可达到与专用模型相媲美的性能。

Paper Details:

Authors: Huizi Cui, Zongbo Han, Chenggong Ding, Naichuan Xiao, Jialong Yang, Jingdong Chen, Guangyu Wang, Qinghua Hu, Changqing Zhang
arXiv ID: 2609.22107
Date: 16 Aug 2026

论文详情：

作者： Huizi Cui, Zongbo Han, Chenggong Ding, Naichuan Xiao, Jialong Yang, Jingdong Chen, Guangyu Wang, Qinghua Hu, Changqing Zhang
arXiv ID: 2609.22107
日期： 2026年8月16日