Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.10550 (cs) [Submitted on 24 Sep 2026] Title: Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models Authors: Tianjing Li, Wei Zhu.

计算机科学 > 计算与语言 arXiv:2610.10550 (cs) [提交于 2026 年 9 月 24 日] 标题：Diffu-LoRA：一种用于个性化扩散模型的新型低秩自适应方法作者：Tianjing Li, Wei Zhu。

Abstract: Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers.

摘要：通过少量参考图像对文本到图像的扩散模型进行个性化设置，需要在保持主体身份的同时遵循描述新语境的提示词。全模型微调需要大量参数，而低秩自适应（LoRA）虽然减少了可训练参数的数量，但并未解决如何跨层分配自适应容量的问题。

We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component.

我们引入了 Diffu-LoRA，这是一种通过门控低秩自适应来学习这种分配的参数高效方法。Diffu-LoRA 将可训练的低秩组件插入到 Transformer 块的线性层中，并为每个组件分配一个可学习的门控。

Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen.

双层优化在不同的数据拆分上更新自适应权重和门控参数，同时渐进式剪枝会移除门控值最低的组件，以满足预设的秩预算。该过程在保持预训练主干冻结的同时，实现了跨层的非均匀自适应容量分配。

Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement.

在 DreamBooth 和其他收集的数据集上对 Stable Diffusion 进行的实验表明，与评估的微调基线相比，该方法提高了整体主体保真度和提示词对齐度。消融研究考察了双层优化、渐进式剪枝和适配器放置的贡献。

These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.

这些结果支持将学习到的秩分配作为一种实现参数高效扩散模型个性化的实用方法。