PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks

PixelUp：用于细粒度视觉任务的零样本语义特征上采样

Abstract: Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required.

摘要： 自监督视觉基础模型（VFMs）凭借其强大且可迁移的视觉表征，已成为下游任务不可或缺的骨干网络。然而，当需要精确的细粒度预测时，其补丁标记（patch-token）级别的特征对于语义分割和深度估计等密集预测任务来说往往过于粗糙。

Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for specific encoders and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions.

为了恢复像素级的细节，研究人员开发了多种特征上采样方法，但这些方法仍面临局限性。可学习的上采样器通常针对特定的编码器设计，必须针对不同的编码器进行重新训练。而使用浅层像素编码器的图像引导方法，往往会引入纹理伪影，且缺乏准确下游预测所需的语义引导。

We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features.

我们引入了 PixelUp，这是一种零样本、与 VFM 无关的上采样器。它通过由多尺度语义特征引导的“从粗到细”窗口交叉注意力架构链，实现了语义感知。

We demonstrate that PixelUp outperforms both VFM-specific and VFM-agnostic upsamplers, achieving state-of-the-art performance on dense prediction tasks with an average improvement of +1.2 mIoU on semantic segmentation and +0.25 $\delta_1$ on NYUv2 depth estimation across VFMs. PixelUp further improves training-free open-vocabulary and unsupervised semantic segmentation by an average of +1.3 mIoU and +0.5 mIoU, respectively.

实验证明，PixelUp 的表现优于 VFM 专用和 VFM 通用的上采样器，在密集预测任务上达到了最先进的性能。在跨 VFM 的测试中，其语义分割任务平均提升了 +1.2 mIoU，NYUv2 深度估计任务平均提升了 +0.25 $\delta_1$。此外，PixelUp 还将无需训练的开放词汇语义分割和无监督语义分割分别平均提升了 +1.3 mIoU 和 +0.5 mIoU。

Code available at: https://github.com/ (Please refer to the original paper for the specific link).

代码地址：https://github.com/（具体链接请参考原论文）。