Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

更优、更强、更快、更广：基于 MLLM 分割的结构化全掩码预测

Abstract: MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction.

摘要： 基于多模态大语言模型（MLLM）的分割技术面临着一个核心的“分割三难困境”：即如何在保持高分割性能、保留对话能力的同时，实现快速推理。嵌入预测（Embedding-prediction）方法可能会通过像素级目标干扰语言建模，而基于下一个 token 生成的方法对于密集掩码而言效率低下。我们提出了“全掩码预测”（All-Mask Prediction），将自回归对话与非自回归掩码预测解耦。

Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference.

其二值化实例 STAMP（同步文本全掩码预测）通过输出词表内的 <SEG> 触发符，将图像对齐的掩码 token 与相应的 patch 特征融合，并利用混合注意力机制在一次传递中将所有 token 分类为前景或背景。因此，它在保持多模态能力和高效推理的同时，结合了强大的指代分割和推理分割能力。

However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass.

然而，二值掩码无法在不进行重复目标特定预测的情况下保留多个语义或实例身份。因此，我们提出了“结构化全掩码预测”并开发了 STAMPlus。它生成包含明确 ID 和可选边界框的目标列表，将这些 ID 绑定到共享的多类掩码空间，并在一次非自回归传递中联合预测所有目标。

A single unified checkpoint retains STAMP’s referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s.

单一的统一检查点在保留 STAMP 指代和推理能力的同时，扩展到了开放词汇语义分割、实例感知分割以及遥感小目标分割，其中高分辨率掩码 token 缩放保留了更精细的空间证据。在这些设置中，STAMPlus 实现了最先进的分割性能，保留了通用的多模态指令遵循能力，并将 12 类目标的推理延迟从 STAMP 重复推理的 13.50 秒降低至 5.16 秒。

Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.

进一步分析表明，准确的目标提示改善了分割效果，而学习到的空间定位能力则有助于“二次观察”（look-twice）推理。总而言之，STAMPlus 解决了超越单目标预测的“三难困境”。