Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment

通过大语言模型增强音频-文本对齐实现零样本呼吸音分类

Abstract: Self-supervised respiratory encoders lack semantic grounding in clinical domain needed for zero-shot inference, limiting their utility without task-specific labeled data. We propose a framework that aligns these encoders with medical terminology in a shared latent space turning them into a zero-shot-capable foundation model.

摘要： 自监督呼吸音编码器缺乏临床领域所需的语义基础，难以进行零样本推理，这限制了它们在没有特定任务标注数据时的实用性。我们提出了一种框架，通过在共享潜在空间中将这些编码器与医学术语对齐，将其转化为具备零样本能力的基座模型。

To address paired data scarcity, we use a medical LLM to synthesize structured reports from metadata, creating dense semantic anchors for contrastive learning. Our training combines a sigmoid-based contrastive loss with encoder’s native SSL objective and similarity-aware negative sampling to sharpen pathological boundaries.

为了解决配对数据稀缺的问题，我们利用医学大语言模型（LLM）从元数据中合成结构化报告，为对比学习创建了密集的语义锚点。我们的训练过程结合了基于 Sigmoid 的对比损失、编码器原生的自监督学习（SSL）目标以及相似度感知负采样，从而更精准地界定病理边界。

Across 9 tasks on 6 datasets, our method achieves a 61.3% mean zero-shot AUC, surpassing CLAP (51.4%) and Qwen2-Audio (54.9%) while reaching the highest linear probing AUC (71.6%) with only 43% of data used by full-scale baselines, showing that structured semantic alignment outperforms large-scale, general-purpose models in clinical diagnostics.

在 6 个数据集的 9 项任务中，我们的方法实现了 61.3% 的平均零样本 AUC，超越了 CLAP (51.4%) 和 Qwen2-Audio (54.9%)；同时，在仅使用全规模基准模型 43% 数据的情况下，达到了最高的线性探测 AUC (71.6%)。这表明在临床诊断中，结构化语义对齐的效果优于大规模通用模型。