Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs’ Hallucinations

MoE 中存在知识注入吗？探索 MoE 中的专家感知对比解码以缓解大模型幻觉

Abstract: Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models’ internal knowledge or have poor cross-domain generalization. Contrastive decoding mitigates hallucinations by using layer-wise differences in LLMs. However, prior studies only explore transformer-based models (e.g., GPT), ignoring other effective frameworks like mixture-of-experts (MoE) models.

摘要： 现有的缓解大语言模型（LLM）幻觉的方法，包括提示工程和模型优化，要么难以改变模型的内部知识，要么跨领域泛化能力较差。对比解码通过利用 LLM 的层间差异来缓解幻觉。然而，先前的研究仅探索了基于 Transformer 的模型（如 GPT），忽略了其他有效的框架，例如混合专家（MoE）模型。

Since MoE alters the traditional transformer architecture, we conduct empirical studies to investigate whether similar layer-wise differences exist in MoEs. Our results show that they do not exist in MoE with shared experts; nevertheless, across different MoEs, higher layers exhibit distinct expert activation patterns between factual and non-factual outputs.

由于 MoE 改变了传统的 Transformer 架构，我们进行了实证研究，以调查 MoE 中是否存在类似的层间差异。我们的结果表明，在共享专家的 MoE 中不存在这种差异；然而，在不同的 MoE 中，高层在事实性输出和非事实性输出之间表现出截然不同的专家激活模式。

Building on these, we propose EAACD, an expert-aware adaptive contrast decoding that uses expert differences in MoE’s higher layers to mitigate hallucinations on QA tasks. EAACD splits high-layer experts into a higher-reliability group and several lower-reliability groups based on their confidence and consistency. It contrasts the higher-reliability group’s prediction with each lower-reliability group’s prediction to calibrate the model’s original predictions.

基于这些发现，我们提出了 EAACD（专家感知自适应对比解码），这是一种利用 MoE 高层专家差异来缓解问答任务中幻觉的方法。EAACD 根据高层专家的置信度和一致性，将其分为高可靠性组和若干低可靠性组。它通过对比高可靠性组与各低可靠性组的预测结果，来校准模型的原始预测。

To strengthen this contrast, EAACD amplifies hallucinations from lower-reliability experts via attention and masking to provide stronger negative references. EAACD outperforms all baselines on four datasets.

为了增强这种对比效果，EAACD 通过注意力机制和掩码技术放大了来自低可靠性专家的幻觉，从而提供更强的负面参考。在四个数据集上的实验表明，EAACD 的表现优于所有基准模型。