SIMD-accelerated computer vision on a $2 microcontroller. I wrote an accelerated FAST feature detector for the ESP32-S3 that leverages its 128-bit SIMD instructions. It achieves roughly double the performance of the reference implementation, and takes around ~6ms to process a QVGA (320x240) frame. You can find the source code here.

在 2 美元的微控制器上实现 SIMD 加速的计算机视觉。我为 ESP32-S3 编写了一个加速的 FAST 特征检测器，利用了其 128 位 SIMD 指令。它的性能大约是参考实现的两倍，处理一帧 QVGA (320x240) 图像大约需要 6 毫秒。你可以在这里找到源代码。

For its price, the ESP32-S3 is a powerhouse of a microcontroller. Within its unassuming plastic package lies a dual-core CPU running at a maximum of 240MHz with a slew of peripherals, including WiFi and Bluetooth Low Energy radios. While digging through its technical reference manual I discovered that the chip supports a limited set of SIMD instructions. For silicon that’s cheaper than the average coffee, that’s pretty cool. Let’s see what exciting things we can do with it!

就其价格而言，ESP32-S3 是一款功能强大的微控制器。在其朴实无华的塑料封装内，搭载了一颗最高运行频率为 240MHz 的双核 CPU，并配备了大量外设，包括 WiFi 和低功耗蓝牙无线电。在查阅其技术参考手册时，我发现该芯片支持一组有限的 SIMD 指令。对于比一杯咖啡还便宜的芯片来说，这非常酷。让我们看看能用它做些什么令人兴奋的事情吧！

In the process of writing a SIMD-accelerated FAST corner detector, I learned the basics of assembly on the ESP32-S3, worked around its eccentric limitations, and ended up writing my own basic register allocator, basm. After some more web browsing, I realized that these instructions weren’t exactly secret, but rather relatively unknown. I wanted to experiment with them myself and write more advanced code than what I could find online. As I’ve been really interested in computer vision lately, I decided on writing a SIMD-accelerated implementation of the FAST feature detector for the ESP32-S3.

在编写 SIMD 加速的 FAST 角点检测器的过程中，我学习了 ESP32-S3 汇编的基础知识，克服了它的一些特殊限制，并最终编写了自己的基础寄存器分配器 basm。在进一步浏览网页后，我意识到这些指令并非什么秘密，只是相对鲜为人知。我想亲自尝试一下，并编写比网上能找到的更高级的代码。由于我最近对计算机视觉非常感兴趣，我决定为 ESP32-S3 编写一个 SIMD 加速的 FAST 特征检测器实现。

I started by looking for example code that uses SIMD instructions. At the time of writing, I could only find three sources: Larry Bank’s blog, where he explores the feature and provides a minimal example; Espressif’s esp-dsp library, used for DSP-esque features (convolutions, FFT, etc.); Espressif’s esp-dl library, used for on-chip AI acceleration. After perusing through them, I was relieved to find that assembly on the ESP32-S3 isn’t that bad.

我首先寻找使用 SIMD 指令的示例代码。在撰写本文时，我只找到了三个来源：Larry Bank 的博客（他探索了该功能并提供了最小示例）、用于 DSP 类功能（卷积、FFT 等）的 Espressif esp-dsp 库，以及用于片上 AI 加速的 Espressif esp-dl 库。在仔细研读之后，我感到欣慰的是，ESP32-S3 上的汇编并没有那么糟糕。

I settled on this general structure: .section .rodata # read-only data here .section .text .align 4 .global add .type add,@function # a simple function that adds two numbers

int32_t add(int32_t a, int32_t b)

add: entry sp, 48 # sp (stack pointer) is an alias for a1 #define r_a a2 #define r_b a3

code here

add a2, r_a, r_b # a2 = a + b retw.n # return, with a2 containing the return value.

我确定了以下通用结构： .section .rodata # 此处为只读数据 .section .text .align 4 .global add .type add,@function # 一个简单的加法函数

int32_t add(int32_t a, int32_t b)

add: entry sp, 48 # sp (栈指针) 是 a1 的别名 #define r_a a2 #define r_b a3

此处为代码

add a2, r_a, r_b # a2 = a + b retw.n # 返回，a2 中包含返回值。

From there, I started writing a SIMD-accelerated version of the FAST corner pre-test. Essentially, it samples one pixel in each cardinal direction and checks if there are at least three “extreme” pixels. In this case, an extreme pixel is one whose absolute difference from the center pixel is above a certain threshold. I decided on a function that would calculate the extrema count of sixteen pixels at a time, since each vector register can store sixteen eight-bit values. This would be achieved by sampling four “chunks” of pixels in each cardinal direction and comparing them to their respective center pixels.

此后，我开始编写 SIMD 加速版的 FAST 角点预测试。本质上，它在每个基准方向上采样一个像素，并检查是否有至少三个“极端”像素。在这种情况下，极端像素是指其与中心像素的绝对差值超过特定阈值的像素。我决定编写一个一次计算 16 个像素极端值数量的函数，因为每个向量寄存器可以存储 16 个 8 位值。这可以通过在每个基准方向上采样四个像素“块”并将它们与各自的中心像素进行比较来实现。

However, this is where I ran into my first limitation- the ISA doesn’t allow for direct misaligned reads. Under the assumption that the center pixel would be aligned, the east and west blocks would be misaligned: Thankfully, the chip has an instruction that allows for extracting a slice of two concatenated registers into another one. Using this functionality, I was able to extract the east and west blocks by reading and extracting slices from the two blocks adjacent to the center block.

然而，我在这里遇到了第一个限制——指令集架构（ISA）不允许直接进行非对齐读取。假设中心像素是对齐的，那么东侧和西侧的块将是非对齐的：幸运的是，该芯片有一条指令，允许将两个连接寄存器的切片提取到另一个寄存器中。利用此功能，我通过读取并从中心块相邻的两个块中提取切片，成功提取了东侧和西侧的块。

Okay, so now I have the pixel data. Time to do some comparisons, right? Wrong. Turns out the ESP32-S3’s only implements comparisons for signed eight-bit numbers. This made processing a little more difficult, because all my pixel data was stored as unsigned eight-bit numbers. If you just reinterpreted the unsigned data as signed data, you get some weird results:

np.int8(np.uint8(range(0, 256, 8))) array([ 0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, -128, -120, -112, -104, -96, -88, -80, -72, -64, -56, -48, -40, -32, -24, -16, -8], dtype=int8) The range [128 255] gets nonlinearly mapped to [-128 -1], breaking all compare operations.

好了，现在我有像素数据了。该进行比较了，对吧？错了。事实证明，ESP32-S3 仅实现了针对有符号 8 位数字的比较。这使得处理变得更加困难，因为我所有的像素数据都存储为无符号 8 位数字。如果你直接将无符号数据重新解释为有符号数据，会得到一些奇怪的结果：

np.int8(np.uint8(range(0, 256, 8))) array([ 0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, -128, -120, -112, -104, -96, -88, -80, -72, -64, -56, -48, -40, -32, -24, -16, -8], dtype=int8) 范围 [128 255] 被非线性映射到 [-128 -1]，破坏了所有的比较操作。

To make this work, I needed to shift the range of the values from [0, 255] to [-128 127]. This should’ve been a simple subtraction, but the instruction set got in the way again: on the ESP32-S3, all the SIMD arithmetic operations are saturating, which meant that all pixel values above 127 would end up being clamped at -128 instead of underflowing to their appropriate values. To overcome this problem, I had to somehow subtract 128 from each number in parallel without using the subtraction instruction.

为了解决这个问题，我需要将值的范围从 [0, 255] 移动到 [-128 127]。这本应是一个简单的减法，但指令集再次阻碍了我：在 ESP32-S3 上，所有的 SIMD 算术运算都是饱和的，这意味着所有大于 127 的像素值最终会被钳位在 -128，而不是下溢到它们应有的值。为了克服这个问题，我必须在不使用减法指令的情况下，并行地从每个数字中减去 128。

This seemed like a daunting task to me at first, but I eventually arrived at a solution. Let’s assume that x is the minuend. Here is its digits’ place values: -128 64 32 16 8 4 2 1. Notice anything interesting? Subtraction is the same as addition by -128, so to compute x - 128, we can just “increment” the number by -128 by adding one to x’s most significant digit, whose place value is -128. We can compute this by simply flipping the highest bit of x. Since XORing any bit with one is the same as taking the bitwise NOT, we can selectively flip the highest bit of x by XORing it with the bit mask 0b10000000, which equal to 0x80. Therefore, x - 128 == x ^ 0x80.

起初这对我来说似乎是一项艰巨的任务，但我最终找到了解决方案。假设 x 是被减数。这是其数字的位值：-128 64 32 16 8 4 2 1。注意到什么有趣的事情了吗？减法等同于加上 -128，所以要计算 x - 128，我们只需通过给 x 的最高有效位（其位值为 -128）加 1 来将数字“增加” -128。我们可以通过简单地翻转 x 的最高位来计算这一点。由于将任何位与 1 进行异或运算等同于进行按位取反，我们可以通过将 x 与位掩码 0b10000000（等于 0x80）进行异或运算来选择性地翻转 x 的最高位。因此，x - 128 == x ^ 0x80。

Now, whenever unsigned eight-bit values are loaded in, they are XOR’d with 0x80 to convert them an appropriate (linear) range where the arithmetic and comparisons make sense again. With those two tricks, I was able to circumvent the limitations of the ESP32-S3 and write the SIMD-accelerated FAST corner pre-test. I also wrote a SIMD-accelerated scoring function using a similar set of operations but I won’t be writing about that here.

现在，每当加载无符号 8 位值时，它们都会与 0x80 进行异或运算，将其转换为一个合适的（线性的）范围，使算术和比较运算再次变得有意义。通过这两个技巧，我成功绕过了 ESP32-S3 的限制，编写了 SIMD 加速的 FAST 角点预测试。我还使用类似的操作集编写了一个 SIMD 加速的评分函数，但我不会在这里介绍它。

In the end, I was able to improve the throughput of the FAST feature detector by about 220%, from 5.1MP/s to 11.2MP/s in my testing. This is well within the acceptable range of performance for realtime computer vision tasks, enabling the ESP32-S3 to easily process a 30fps VGA stream. Not bad for $2!

最终，在我的测试中，我将 FAST 特征检测器的吞吐量提高了约 220%，从 5.1MP/s 提升到了 11.2MP/s。这完全在实时计算机视觉任务的可接受性能范围内，使 ESP32-S3 能够轻松处理 30fps 的 VGA 流。对于 2 美元的成本来说，这相当不错！