Multi-Modal Generative Fuzzy System: Fuzzy Inference Guided Large Model Interactive Question Answering Framework

多模态生成模糊系统：模糊推理引导的大模型交互式问答框架

Abstract: In Multimodal Question Answering (MQA), models are required to jointly encode and integrate heterogeneous information from multiple modalities, including text, images, and speech, to perform complex semantic reasoning and decision making.

摘要： 在多模态问答（MQA）中，模型需要联合编码并整合来自文本、图像和语音等多种模态的异构信息，以执行复杂的语义推理和决策。

Despite recent advances, existing approaches, including traditional deep learning models and Large Models (LMs) or prompt-based frameworks, continue to face several critical challenges. First, modality bias arises from discrepancies in feature distributions across different modalities, which limits effective cross-modal collaborative understanding. Second, many questions require knowledge drawn from multiple domains, introducing significant uncertainty. Third, current methods often rely on shallow semantic matching, resulting in limited reasoning depth and reduced interpretability.

尽管近期取得了进展，但现有方法（包括传统的深度学习模型、大模型（LMs）或基于提示的框架）仍面临几个关键挑战。首先，模态偏差源于不同模态间特征分布的差异，这限制了有效的跨模态协同理解。其次，许多问题需要来自多个领域的知识，从而引入了显著的不确定性。第三，当前方法往往依赖于浅层的语义匹配，导致推理深度有限且可解释性降低。

To address these issues, inspired by the traditional fuzzy system (FS) framework, we propose a fuzzy-inference-guided multimodal generative architecture termed the Multi-Modal Generative Fuzzy System (MMGFS). The main contributions of MMGFS are two-fold. First, it alleviates modality bias through a multimodal collaborative rumination mechanism. Second, it introduces fuzzy rules and a multi-hop inference mechanism to support cross-domain knowledge fusion and hierarchical reasoning, thereby strengthening uncertainty modelling and deepening semantic understanding.

为了解决这些问题，受传统模糊系统（FS）框架的启发，我们提出了一种名为多模态生成模糊系统（MMGFS）的模糊推理引导多模态生成架构。MMGFS 的主要贡献有两点：首先，它通过一种多模态协同反刍机制缓解了模态偏差；其次，它引入了模糊规则和多跳推理机制，以支持跨领域知识融合和分层推理，从而加强了不确定性建模并深化了语义理解。

We conduct comprehensive evaluations on open-domain question answering datasets, including MultimodalQA and WebQA, as well as domain-specific benchmarks, including BioMol-VQA and EHRxQA. Experimental results demonstrate that MMGFS consistently outperforms existing methods across multiple datasets. It effectively mitigates modality bias and question uncertainty while achieving superior performance in answer accuracy, consistency, and generalization.

我们在开放域问答数据集（包括 MultimodalQA 和 WebQA）以及特定领域基准测试（包括 BioMol-VQA 和 EHRxQA）上进行了全面评估。实验结果表明，MMGFS 在多个数据集上始终优于现有方法。它有效地减轻了模态偏差和问题不确定性，同时在答案准确性、一致性和泛化能力方面表现卓越。