Large Language Models as Unified Multimodal Learners for Clinical Prediction
Large Language Models as Unified Multimodal Learners for Clinical Prediction
大语言模型作为临床预测的统一多模态学习器
Abstract: Electronic health records combine free-text clinical narratives with structured measurements such as vital signs, laboratory values, and comorbidities. Yet most clinical prediction systems still rely on task-specific fusion architectures, pairing dedicated encoders for each modality with learned combination mechanisms that must be re-engineered for every new task and clinical setting.
摘要： 电子健康记录将自由文本临床叙述与结构化测量数据（如生命体征、实验室检查值和合并症）相结合。然而，大多数临床预测系统仍然依赖于特定任务的融合架构，即为每种模态配对专门的编码器，并使用需要针对每个新任务和临床环境重新设计的学习组合机制。
We propose a simpler alternative: convert all patient data, regardless of modality, into a single natural language sequence and fine-tune a pretrained language model end-to-end, with no architectural modification for fusion.
我们提出了一种更简单的替代方案：将所有患者数据（无论模态如何）转换为单一的自然语言序列，并对预训练语言模型进行端到端的微调，无需进行任何融合架构的修改。
We evaluate this approach across three clinically distinct prediction tasks: in-hospital mortality on MIMIC-III, graft failure prediction using longitudinal data from a German transplant center, and emergency triage classification from ambulance records - comparing encoder-based (ModernBERT) and decoder-based (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) fine-tuning against established multimodal baselines and, for graft failure, a gradient boosting model currently used in clinical practice for post-transplant patient management.
我们在三个临床上截然不同的预测任务中评估了该方法：MIMIC-III 上的院内死亡率预测、使用德国移植中心纵向数据的移植物衰竭预测，以及来自救护车记录的急诊分诊分类。我们将基于编码器（ModernBERT）和基于解码器（Llama 3.1、Gemma、DeepSeek-R1-Qwen、Qwen3）的微调结果与现有的多模态基准进行了比较；对于移植物衰竭任务，还与目前临床实践中用于移植后患者管理的梯度提升模型进行了对比。
Across all three tasks, unified textual serialization matches or exceeds task-specific multimodal baselines, and outperforms the clinically deployed gradient boosting system on graft failure prediction. These results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.
在所有三个任务中，统一的文本序列化方法均达到或超过了特定任务的多模态基准，并在移植物衰竭预测方面优于临床部署的梯度提升系统。这些结果表明，一种无需定制融合架构的单一序列化范式足以胜任多模态临床预测——在大幅降低系统复杂性的同时，能够匹配甚至超越专业化设计的效果。