The NX bit is not just about security

NX 位不仅仅关乎安全

September 4, 2026 | Guest post | Lobsters 2026年9月4日 | 客座文章 | Lobsters

While I’m taking a short break from low-level programming, here’s a story by a friend of mine, Sonya, about debugging a seemingly impossible bug in ARM code. 在我暂时放下底层编程工作之际，分享一篇我朋友 Sonya 的故事，讲述了她在调试 ARM 代码时遇到的一个看似不可能解决的 Bug。

This bug hunting saga started several months ago. While developing a bare-metal hypervisor on ARM64 for postmarketOS, I hit a strange bug: if I enabled the CTR_EL0 intercept (which was the whole purpose of the HV, so could not be skipped), the phone would randomly lock itself up. After a few seconds, the watchdog kicked in and reset the system. At first I thought that the boot got slowed down so much that the system simply did not have enough time to boot, but disabling the watchdog did not help either (fortunately, the phone in question had removable battery, and I didn’t have to wait several hours for it to discharge). So, I started digging deeper. 这场 Bug 狩猎之旅始于几个月前。在为 postmarketOS 开发 ARM64 裸机 Hypervisor 时，我遇到了一个奇怪的 Bug：如果我启用了 CTR_EL0 拦截（这是该 Hypervisor 的核心目的，因此无法跳过），手机就会随机死机。几秒钟后，看门狗（watchdog）介入并重置了系统。起初我以为是启动过程变慢，导致系统没有足够的时间完成引导，但禁用看门狗也无济于事（幸运的是，那部手机有可拆卸电池，我不需要等几个小时让它耗尽电量）。于是，我开始深入挖掘。

Hypothesis #1: My emulation of MRS is wrong

假设 #1：我对 MRS 的模拟有误

On Aarch64, so-called Special Function Registers (which CTR_EL0 is one of) are accessed using MRS and MSR machine instructions: 在 Aarch64 架构上，所谓的特殊功能寄存器（Special Function Registers，CTR_EL0 即其中之一）是通过 MRS 和 MSR 机器指令进行访问的：

mrs x3, ctr_el0 // read access
msr ctr_el0, x3 // write access

These instructions move data between the specified SFR and the specified general-purpose register (in this case, X3), with all other registers remaining unchanged. So the failure must have meant that I was either corrupting some of the registers I had to preserve, or not writing the real output register correctly. Thus the first two things I verified were the exception handler trampoline: 这些指令在指定的 SFR 和指定的通用寄存器（本例中为 X3）之间移动数据，而所有其他寄存器保持不变。因此，故障一定意味着我要么破坏了某些必须保留的寄存器，要么没有正确写入实际的输出寄存器。因此，我首先验证的两件事是异常处理程序的蹦床（trampoline）：

trap_from_el1:
    sub sp, sp, #256
    stp x0, x1, [sp]
    stp x2, x3, [sp, #16]
    stp x4, x5, [sp, #32]
    // ...
    stp x28, x29, [sp, #224]
    stp x30, xzr, [sp, #240]
    mov x0, sp
    bl handle_trap_from_el1
    mrs x1, elr_el2
    add x0, x0, x1
    msr elr_el2, x0
    ldp x0, x1, [sp]
    ldp x2, x3, [sp, #16]
    ldp x4, x5, [sp, #32]
    // ...
    ldp x28, x29, [sp, #224]
    ldp x30, xzr, [sp, #240]
    add sp, sp, #256
    eret

And the stack allocation: 以及栈分配：

.section .data.stack
.p2align 12
.long 0
.p2align 12
stack:

Both were correct, with no obvious signs of issues. I singlestepped the whole exception handler in QEMU and verified that it was acting exactly as expected. 两者都是正确的，没有明显的异常迹象。我在 QEMU 中单步执行了整个异常处理程序，并验证了它的行为完全符合预期。

To make sure things work smoothly on real hardware, I added debug prints before and after the exception handler, and it turned out that, on real hardware, the exception handler was not modifying any registers, even the intended output register. 为了确保在真实硬件上运行顺畅，我在异常处理程序前后添加了调试打印信息。结果发现，在真实硬件上，异常处理程序并没有修改任何寄存器，甚至连预期的输出寄存器也没有修改。

The culprit turned out to be in this innocent invocation: 罪魁祸首竟然是这段看似无辜的调用：

msr_accessor_sort(
    msr_accessors,
    ((uintptr_t)msr_accessors_end - (uintptr_t)msr_accessors) / sizeof(struct msr_accessor)
);

As you may know, ARM is not Icache/Dcache-coherent, which means that modifications to the data do not automatically propagate to the instruction fetches: either the modified data may not have been committed to RAM yet, or the instruction cache might be holding stale cached data from before the write. And since the buffer contained executable machine instructions, sorting the array and later trying to execute it was not going to work. So I moved the sorting to the build phase and… 众所周知，ARM 的指令缓存（Icache）和数据缓存（Dcache）并不一致，这意味着对数据的修改不会自动同步到指令获取中：修改后的数据可能尚未提交到内存，或者指令缓存中可能仍保留着写入前的陈旧数据。由于缓冲区中包含可执行的机器指令，对数组进行排序后再尝试执行是行不通的。于是我将排序移到了构建阶段，然后……

The debug prints showed that the handlers worked as intended. But the system still didn’t boot. 调试打印显示处理程序按预期工作了。但系统仍然无法启动。

Hypothesis #2: out-of-spec hardware

假设 #2：硬件不符合规范

As you all know, x86(-64) hardware is only produced by two vendors – Intel and AMD, so we can expect very consistent behavior across systems. 众所周知，x86(-64) 硬件仅由 Intel 和 AMD 两家厂商生产，因此我们可以预期不同系统间的行为非常一致。

On the ARM side the situation is different: while ARM does provide a reference implementation of the architecture, vendors are free to customize it at will, or even roll their own implementations. This means that ARM CPUs tend to have many subtle (and not so subtle) bugs, some of which were probably considered features by their developers. So the next obvious guess was that the CPU was somehow out-of-spec, and was not behaving as it was supposed to. 在 ARM 方面情况则不同：虽然 ARM 确实提供了架构的参考实现，但厂商可以随意定制，甚至自行开发实现。这意味着 ARM CPU 往往存在许多细微（甚至不那么细微）的 Bug，其中一些可能被开发者视为特性。因此，下一个显而易见的猜测是 CPU 在某些方面不符合规范，行为不符合预期。

With that in mind, I added extra handlers to make sure that every exception is printed: 考虑到这一点，我添加了额外的处理程序，以确保每个异常都能被打印出来：

vbar_el2:
    bl unknown_trap
    .p2align 7
    bl unknown_trap
    .p2align 7
    // ... (省略部分代码)
    b trap_from_el1
    .p2align 7
    // ...

The unknown_trap routine would then print out the X30 register (lr for those of you more familiar with Aarch32), ELR_EL2, and other SFRs to determine the cause of the exception. However, none of this was actually firing. unknown_trap 例程会打印出 X30 寄存器（对于熟悉 Aarch32 的人来说是 lr）、ELR_EL2 以及其他 SFR，以确定异常原因。然而，这些代码实际上从未触发。

The next best guess was that the kernel was panicking due to some wrong handling, so the next thing I did was using /proc/last_kmsg to read the crashed kernel’s logs. Unfortunately for me, the last messages I got in the log were: 下一个合理的猜测是内核因为某些错误处理而崩溃（panic），所以我接下来使用 /proc/last_kmsg 读取了崩溃内核的日志。不幸的是，日志中最后的消息是：

[ 3.113676] (0)[153:init]fs_mgr: Running /system/bin/e2fsck on /dev/block/platform/mtk-msdc.0/11230000.msdc0/by-name/userdata
[ 3.125072] (0)[158:e2fsck]random: e2fsck urandom read with 12 bits of entropy available

This meant that the kernel did not crash cleanly. I suspected that the /dev/urandom device was responsible for the crash, and patched it out of the kernel to test that hypothesis – which made the boot go a bit further, but not much. Still a dead end. 这意味着内核没有正常崩溃。我怀疑 /dev/urandom 设备导致了崩溃，于是从内核中将其补丁掉以测试该假设——这让启动过程稍微推进了一点，但没多大进展。依然是死胡同。

At that point I had no idea where exactly things were going wrong, but I knew it had something to do with the intercepts, since disabling the intercepts entirely fixed the issue. So, I filtered the kernel for suspicious instructions using objdump: 那时我完全不知道问题出在哪里，但我知道这一定与拦截有关，因为完全禁用拦截可以解决问题。于是，我使用 objdump 在内核中过滤了可疑指令：

$ aarch64-unknown-linux-gnu-objdump -D -b binary -m aarch64 kernel.orig | grep ctr_el0

This yielded 22 matches, which I manually patched in the binary to return the correct value. After booting the patched kernel, it was still unable to make it to Android, but adb shell worked, which meant that the patch was (at least somewhat) successful. After that I was able to reduce the patch to only a few “hot” instructions, but was still clueless about the actual reason for the hangups. At that point I suspected that my hypervisor and the kernel were som… 这产生了 22 个匹配项，我手动在二进制文件中修补了它们以返回正确的值。在启动修补后的内核后，它仍然无法进入 Android，但 adb shell 可以工作，这意味着补丁（至少在某种程度上）是成功的。此后，我能够将补丁缩减到仅剩几个“热点”指令，但对于死机的真正原因仍然毫无头绪。那时我怀疑我的 Hypervisor 和内核……