Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence

用于脑-语言对应关系的边缘正则化结构化语义对齐

Abstract: With the rapid advancement of large language models, brain-language decoding has achieved remarkable progress. However, it remains unclear whether decoded content genuinely reflects neural representations or is largely reconstructed by the language model itself. This ambiguity limits interpretability and hinders the investigation of intrinsic brain-language correspondence.

摘要： 随着大语言模型的飞速发展，脑-语言解码技术取得了显著进展。然而，目前尚不清楚解码内容是真实反映了神经表征，还是主要由语言模型自身重构而成。这种模糊性限制了模型的可解释性，并阻碍了对内在脑-语言对应关系的深入研究。

To address this challenge, we propose MD-SigLIP. This margin-regularized structured semantic alignment framework directly aligns brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding. This formulation enables explicit modeling of the correspondence between neural representations and language semantics.

为了应对这一挑战，我们提出了 MD-SigLIP。这是一个边缘正则化的结构化语义对齐框架，它在共享语义空间中直接将大脑嵌入（brain embeddings）与文本嵌入进行对齐，从而实现基于检索的解码。这种表述方式能够显式地建模神经表征与语言语义之间的对应关系。

Building upon duplicate-aware sigmoid contrastive learning, we introduce a listwise margin-regularized term that enforces structured ranking constraints between positive semantic clusters and negative samples. By modeling multi-positive semantic structure and margin-based ordering simultaneously, the method captures the manifold organization of language embeddings reflected in neural signals.

在基于重复感知（duplicate-aware）的 Sigmoid 对比学习基础上，我们引入了一个列表式边缘正则化项，用于在正向语义簇和负样本之间强制执行结构化的排序约束。通过同时建模多正向语义结构和基于边缘的排序，该方法能够捕捉到神经信号中所反映的语言嵌入的流形组织结构。

Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.

实验表明，该方法在全词汇表和子集评估设置下均达到了最先进（SOTA）的检索性能。