Purify Before You Align: Improving Representation Alignment for Diffusion Models

Title: Purify Before You Align: Improving Representation Alignment for Diffusion Models

标题：在对齐前净化：改进扩散模型的表征对齐

Abstract: Representation alignment improves and accelerates diffusion training by encouraging intermediate features to match a pretrained visual encoder. However, standard alignment matches pretrained representations without distinguishing what information may be more useful for generation. We introduce local appearance purification (LAP), which improves generation quality and makes encoders such as MAE more effective alignment targets by removing locally predictable content and aligning to the residual.

摘要：表征对齐通过鼓励中间特征与预训练的视觉编码器相匹配，从而改进并加速了扩散模型的训练。然而，标准的对齐方式在匹配预训练表征时，并未区分哪些信息对生成任务更有用。我们引入了局部外观净化（LAP）方法，通过去除局部可预测的内容并对齐残差，提高了生成质量，并使 MAE 等编码器成为更有效的对齐目标。

LAP modifies only the alignment target, and requires no changes to diffusion training or inference. Through controlled interventions within a fixed encoder, we show that effective alignment depends on what information the target emphasizes, not simply how much image information it contains. Our theory gives a variational interpretation of standard alignment and, for fixed local predictors, a sum of token-wise information bounds conditioned on local appearance.

LAP 仅修改对齐目标，无需对扩散模型的训练或推理过程进行任何更改。通过在固定编码器内进行受控干预，我们证明了有效的对齐取决于目标强调了什么信息，而不仅仅是它包含了多少图像信息。我们的理论为标准对齐提供了变分解释，并针对固定的局部预测器，给出了以局部外观为条件的标记级信息界限之和。

To understand why purification benefits some encoders more than others, we derive an exact constant-readout test that measures how well the target can be matched without using the diffusion model’s hidden state. LAP improves generation quality across five pretrained representations (MAE, AIMv2, CLIP, DINOv2, DINOv3), with the largest gain for MAE: on ImageNet-256 with SiT-B/2, it reduces FID from $53.1$ to $42.3$. Its gains persist with stronger alignment recipes and tokenizers. Computed before diffusion training, the constant-readout score predicts the benefit of purification (Spearman $\rho=0.94$ across 13 encoders).

为了理解为什么净化对某些编码器的益处比其他编码器更大，我们推导了一种精确的常数读出测试，用于衡量在不使用扩散模型隐藏状态的情况下，目标能够被匹配的程度。LAP 在五种预训练表征（MAE、AIMv2、CLIP、DINOv2、DINOv3）上均提升了生成质量，其中对 MAE 的提升最为显著：在 ImageNet-256 数据集上使用 SiT-B/2 模型时，它将 FID 从 53.1 降低至 42.3。这种增益在更强的对齐方案和分词器下依然存在。在扩散模型训练前计算出的常数读出分数，能够预测净化的收益（在 13 个编码器上的 Spearman $\rho=0.94$）。