Enabling Fully Integer-Only Inference for Lightweight Detection Transformers

实现轻量级检测 Transformer 的全整数推理

Abstract: Vision Transformer detectors now approach the accuracy of CNNs but remain difficult to deploy on NPUs and microcontrollers because key components, including deformable attention, feature fusion, and nonlinear activation functions, are not natively compatible with integer arithmetic. Existing quantized detectors either retain operators such as Softmax, GELU, and LayerNorm or focus on heavyweight backbones, leaving lightweight detection transformers without an end-to-end integer implementation.

摘要： 视觉 Transformer 检测器目前的准确率已接近卷积神经网络（CNN），但由于可变形注意力机制（deformable attention）、特征融合和非线性激活函数等关键组件无法原生兼容整数运算，它们在 NPU 和微控制器上的部署仍然困难重重。现有的量化检测器要么保留了 Softmax、GELU 和 LayerNorm 等算子，要么侧重于重量级骨干网络，导致轻量级检测 Transformer 缺乏端到端的整数实现方案。

We address this gap with I-LW-DETR, the first fully integer-only lightweight DETR, in which every operation in the forward pass, including transformer nonlinearities, is executed in integer arithmetic. I-LW-DETR is built upon three key components: a scale-preserving split convolution that assigns independent activation scale to each branch of the multi-scale projector; SD-ShiftGELU, a sign-dependent GELU approximation that preserves element-wise behavior while avoiding the accuracy degradation; and a constrained Shiftmax that maintains stable Softmax normalization.

我们通过 I-LW-DETR 填补了这一空白。这是首个完全仅使用整数的轻量级 DETR，其前向传播中的每一个操作（包括 Transformer 的非线性运算）均以整数算术执行。I-LW-DETR 基于三个关键组件构建：一种为多尺度投影仪的每个分支分配独立激活尺度的“保尺度拆分卷积”（scale-preserving split convolution）；一种名为 SD-ShiftGELU 的符号依赖型 GELU 近似方法，它在保持逐元素行为的同时避免了精度下降；以及一种能够维持稳定 Softmax 归一化的受限 Shiftmax。

Experimental results demonstrate that the proposed quantization pipeline consistently produces efficient fully integer-only models across different model scales. Across all model scales, the proposed pipeline incurs only a moderate accuracy degradation while reducing the model size by approximately $3.6\times$ and the computational cost by more than one order of magnitude.

实验结果表明，所提出的量化流程在不同模型规模下均能稳定生成高效的全整数模型。在所有模型规模中，该流程仅导致了适度的精度下降，同时将模型大小缩小了约 3.6 倍，并将计算成本降低了一个数量级以上。