Hunting Down a Go Runtime Bug on 32-bit Embedded Systems

追踪 32 位嵌入式系统上的 Go 运行时错误

Our daily work usually revolves around Linux and security topics, deep down in the software stack. Still, more often than you might think, we end up debugging applications which live much higher up. Sometimes such a problem has its roots in the Linux kernel, sometimes elsewhere. In this blog post we show how we found and fixed a bug inside the Go runtime. Recently a customer reported that an application written in Go crashes from time to time on one of their embedded Linux systems.

我们的日常工作通常围绕 Linux 和安全主题，深入软件栈的底层。尽管如此，我们处理的调试任务往往比你想象的要高层得多。有时这类问题的根源在于 Linux 内核，有时则在其他地方。在这篇博文中，我们将展示如何发现并修复 Go 运行时中的一个错误。最近，一位客户报告称，他们的一款 Go 语言应用程序在嵌入式 Linux 系统上偶尔会崩溃。

Introduction

引言

The crash was always the same fatal error with the following signature: runtime: netpoll: eventfd ready for 5 fatal error: runtime: netpoll: eventfd ready for something unexpected …stack trace… At first we assumed that the application itself was buggy and needed fixing. But after inspecting the error more closely, it looked much more like an internal assumption in Go’s netpoll mechanism no longer holds.

崩溃总是表现为相同的致命错误，特征如下： runtime: netpoll: eventfd ready for 5 fatal error: runtime: netpoll: eventfd ready for something unexpected …堆栈跟踪… 起初，我们认为应用程序本身有 Bug 需要修复。但在仔细检查错误后，情况看起来更像是 Go 的 netpoll 机制中的一个内部假设不再成立了。

The error message comes from netpoll() in src/runtime/netpoll_epoll.go:

if ev.Events != linux.EPOLLIN {
    println("runtime: netpoll: eventfd ready for", ev.Events)
    throw("runtime: netpoll: eventfd ready for something unexpected")
}

In this code path, the netpoll code expects EPOLLIN to be the only firing event, but it got something else. In our case it got 5, which is EPOLLIN|EPOLLOUT. Why would epoll suddenly report more than EPOLLIN if the code asked only for EPOLLIN?

该错误消息来自 src/runtime/netpoll_epoll.go 中的 netpoll() 函数：

if ev.Events != linux.EPOLLIN {
    println("runtime: netpoll: eventfd ready for", ev.Events)
    throw("runtime: netpoll: eventfd ready for something unexpected")
}

在此代码路径中，netpoll 代码预期 EPOLLIN 是唯一的触发事件，但它却收到了其他内容。在我们的案例中，它收到了 5，即 EPOLLIN|EPOLLOUT。如果代码只请求了 EPOLLIN，为什么 epoll 会突然报告除 EPOLLIN 之外的其他事件呢？

Before digging deeper, we threw the error message into a search engine, hoping that somebody else had faced the same issue before. This led us straight to a report in the Go project’s issue tracker: runtime: netpoll: eventfd ready for something unexpected. The issue describes exactly the fatal error we saw. Also on a 32-bit ARM embedded Linux system! The reporters also noted that the crash happens in applications which run for a long time. That matched our customer’s description, too. Bingo!

在深入研究之前，我们将错误消息输入搜索引擎，希望其他人也遇到过同样的问题。这直接引导我们找到了 Go 项目问题追踪器中的一个报告：runtime: netpoll: eventfd ready for something unexpected。该问题描述的正是我们看到的致命错误，而且同样是在 32 位 ARM 嵌入式 Linux 系统上！报告者还指出，崩溃发生在长时间运行的应用程序中。这与我们客户的描述完全吻合。找到了！

The issue had been open and unresolved since March 2025. The Go maintainers had also rejected one attempt to fix the problem. From the comments on the issue we learned that the fatal error only ever showed up on 32-bit ARM and i386 Linux systems, with all kinds of kernel versions. Some kernels were rather old, others recent. Not a single reporter saw it on an x86_64 or arm64 system.

该问题自 2025 年 3 月起就一直处于开启且未解决状态。Go 的维护者甚至拒绝过一次修复该问题的尝试。从问题的评论中我们了解到，该致命错误仅出现在 32 位 ARM 和 i386 Linux 系统上，且涉及各种内核版本。有些内核相当老旧，有些则是最新的。没有一个报告者在 x86_64 或 arm64 系统上遇到过此问题。

Accepting the Challenge

接受挑战

The problem had multiple reporters but no fix, so we decided to dig into the issue ourselves. At the very least we could give the Go folks better input. Initially we suspected that epoll behaves differently on 32-bit ARM or i386. We ditched this idea quickly since epoll is generic core code in the kernel. Why would it return a spurious event set only on 32-bit ARM or i386?

这个问题有多个报告者但没有修复方案，因此我们决定亲自深入研究。至少，我们可以为 Go 团队提供更好的反馈。最初，我们怀疑 epoll 在 32 位 ARM 或 i386 上的行为有所不同。但我们很快放弃了这个想法，因为 epoll 是内核中的通用核心代码。为什么它只会在 32 位 ARM 或 i386 上返回虚假的事件集呢？

Still, the fact that the error showed up only on 32-bit systems gnawed at us. As a next step we reviewed the epoll usage in Go’s netpoll code. With the help of an LLM we went through src/runtime/netpoll_epoll.go, focusing on 32-bit pitfalls such as integer conversions.

尽管如此，错误仅出现在 32 位系统上的事实一直困扰着我们。作为下一步，我们审查了 Go 的 netpoll 代码中对 epoll 的使用。在 LLM 的帮助下，我们通读了 src/runtime/netpoll_epoll.go，重点关注 32 位系统下的陷阱，例如整数转换。

The review revealed that Go’s netpoll code uses the data field of struct epoll_event. Linux epoll can store an 8-byte cookie in the kernel and returns it as part of the firing event to user space. Applications use this cookie to attach metadata to an event, for example to tell different event sources apart.

审查发现，Go 的 netpoll 代码使用了 struct epoll_event 的 data 字段。Linux epoll 可以在内核中存储一个 8 字节的 cookie，并将其作为触发事件的一部分返回给用户空间。应用程序使用此 cookie 将元数据附加到事件上，例如用于区分不同的事件源。

struct epoll_event {
    __poll_t events; /* ev.Events in Go netpoll */
    __u64 data;      /* ev.Data in Go netpoll */
};

Deep inside the Go runtime, the main event handler needs to know whether an event belongs to an event fd or a socket fd. It decides by comparing ev.Data to the address of its internal event fd object:

if *(**uintptr)(unsafe.Pointer(&ev.Data)) == &netpollEventFd {
    ...
}

Reading further through the code showed that ev.Data holds either a raw pointer to netpollEventFd or a tagged pointer to a per-socket object, pollDesc. The pointer tag is a counter, fdseq, which distinguishes recycled pollDesc objects. So far so good. Mixing raw and tagged pointers in the same field looked fishy to us. But how this relates to the crash was not clear yet.

在 Go 运行时的深处，主事件处理程序需要知道事件是属于 event fd 还是 socket fd。它通过将 ev.Data 与其内部 event fd 对象的地址进行比较来做出决定：

if *(**uintptr)(unsafe.Pointer(&ev.Data)) == &netpollEventFd {
    ...
}

进一步阅读代码显示，ev.Data 要么保存指向 netpollEventFd 的原始指针，要么保存指向每个 socket 对象 pollDesc 的带标签指针（tagged pointer）。指针标签是一个计数器 fdseq，用于区分循环利用的 pollDesc 对象。到目前为止一切正常。在同一个字段中混合使用原始指针和带标签指针看起来很可疑，但它与崩溃之间的联系尚不明确。

Inspecting how Go lays out tagged pointers in memory finally revealed the core of the issue.

检查 Go 如何在内存中布局带标签指针，最终揭示了问题的核心。

The Aha Moment

顿悟时刻

On 32-bit platforms, Go’s tagged pointer logic packs the full 32-bit address and up to 32 tag bits into an 8-byte word. The tag goes into the lower 4 bytes and the address into the upper 4 bytes.

在 32 位平台上，Go 的带标签指针逻辑将完整的 32 位地址和最多 32 位的标签位打包成一个 8 字节的字。标签位于低 4 字节，地址位于高 4 字节。

Storing a tagged pointer with address 0x00123456 and tag 0x12 fills ev.Data like this:

	ev.Data[0:4]	ev.Data[4:8]
	fdseq (e.g. 0x00000012)	*pollDesc (e.g. 0x00123456)

Storing the raw pointer, on the other hand, leaves the following contents in ev.Data:

	ev.Data[0:4]	ev.Data[4:8]
	&netpollEventFd (e.g. 0x00123456)	0 (untouched)

存储一个地址为 0x00123456、标签为 0x12 的带标签指针时，ev.Data 的填充方式如下：

	ev.Data[0:4]	ev.Data[4:8]
	fdseq (例如 0x00000012)	*pollDesc (例如 0x00123456)

另一方面，存储原始指针时，ev.Data 中的内容如下：

	ev.Data[0:4]	ev.Data[4:8]
	&netpollEventFd (例如 0x00123456)	0 (未触动)

The lower 4 bytes hold the address of the object. The upper 4 bytes stay untouched since an address is only 4 bytes long on a 32-bit system. This finally shows the root of the problem. The comparison of ev.Data with &netpollEventFd evaluates only the lower 4 bytes of ev.Data. The code casts ev.Data to uintptr, which is 4 bytes on 32-bit platforms. So the address of the netpollEventFd object aliases with fdseq.

低 4 字节保存对象的地址。高 4 字节保持不变，因为在 32 位系统上地址只有 4 字节长。这最终揭示了问题的根源。将 ev.Data 与 &netpollEventFd 进行比较时，仅评估了 ev.Data 的低 4 字节。代码将 ev.Data 转换为 uintptr，在 32 位平台上它是 4 字节。因此，netpollEventFd 对象的地址与 fdseq 发生了别名冲突。

As soon as fdseq grows large enough to match &netpollEventFd, the netpoll logic mistakes a socket fd for an event fd. Internal assumptions fall apart, among them the assumption that the ready event is just EPOLLIN.

一旦 fdseq 增长到足以匹配 &netpollEventFd，netpoll 逻辑就会将 socket fd 误认为是 event fd。内部假设随之崩溃，其中包括“就绪事件仅为 EPOLLIN”这一假设。

Note that this aliasing can only happen on 32-bit little endian systems. On a 64-bit system, the comparison always covers the whole 8 bytes. On a 32-bit big endian system, the comparison would read the upper 4 bytes, which contain the address.

请注意，这种别名冲突仅发生在 32 位小端序系统上。在 64 位系统上，比较总是覆盖整个 8 字节。在 32 位大端序系统上，比较会读取包含地址的高 4 字节。

fdseq needs to grow into the millions before it matches &netpollEventFd. That’s why the problem shows up only in long-running programs which create lots of pollDesc objects over time.

fdseq 需要增长到数百万才能匹配 &netpollEventFd。这就是为什么该问题仅出现在长时间运行、且随着时间推移创建了大量 pollDesc 对象的程序中。