The occasional ECONNRESET (part 1/2)

偶尔出现的 ECONNRESET（第一部分/共两部分）

2026-05-05 2026-05-05

Two services running on the same machine. One of them opens a listening TCP socket bound to localhost, the other one connects to that. They exchange data. Every now and then, the service that initiated the connection gets an ECONNRESET while reading data from the socket — but no other errors show up in the logs, no crashes, nothing. What’s going on? 两项服务运行在同一台机器上。其中一个服务在 localhost 上打开一个监听 TCP 套接字，另一个服务则连接到它。它们进行数据交换。偶尔，发起连接的服务在从套接字读取数据时会收到 ECONNRESET 错误——但日志中没有显示其他错误，没有崩溃，什么都没有。到底发生了什么？

A reproducer in the “lab”

“实验室”中的复现程序

Let’s start with the “server”, i.e. the service that opens the listening socket. The following program does just that: Create a new TCP socket, wait for connections, fork into a new process for each request. There’s not much of a “request”, though: The server simply dumps 600,000 x bytes to the client upon connection. The number 600,000 bears some meaning: It needs to be large enough to trigger the behavior that I want to show. 600 bytes, for example, probably won’t work. 让我们从“服务器”开始，即打开监听套接字的服务。以下程序正是这样做的：创建一个新的 TCP 套接字，等待连接，并为每个请求 fork 出一个新进程。不过，这里并没有太多的“请求”：服务器在连接建立后，只需向客户端倾倒 600,000 个字节的 ‘x’。数字 600,000 有其特殊含义：它必须足够大，才能触发我想要展示的行为。例如，600 字节可能无法触发该行为。

Now on to the client: It connects to port 8125 on localhost where our server is waiting, and then it calls recv() until EOF or error. We’ll get to the --spam flag in a second. 现在来看客户端：它连接到 localhost 的 8125 端口（我们的服务器正在那里等待），然后调用 recv() 直到遇到 EOF 或错误。我们稍后会讨论 --spam 标志。

Let’s run the two programs: 让我们运行这两个程序：

[terminal1]$ ./server
[terminal2]$ ./client
Read 600000 bytes, final return value was 0, errno was 0

Nothing spectacular. But let’s use the --spam flag: 没什么特别的。但让我们使用 --spam 标志试试：

$ ./client --spam
Read 600000 bytes, final return value was -1, errno was 104 Connection reset by peer
...

--spam causes the client to first send some data to the server before it tries to receive data. And apparently this causes the connection to break at some point: The client’s recv() sees a return value of -1 and errno gets set to 104 = Connection reset by peer. --spam 会导致客户端在尝试接收数据之前先向服务器发送一些数据。显然，这导致连接在某个时刻中断了：客户端的 recv() 返回值为 -1，且 errno 被设置为 104，即“Connection reset by peer”（对端重置连接）。

What tcpdump sees

tcpdump 观察到的情况

First, what’s “on the wire”? Okay. So there actually is a TCP RST. Could have also been a programming error or misinterpretation on my part. 首先，“线路”上发生了什么？好的，确实存在 TCP RST（重置包）。这也有可能是我个人的编程错误或误解。

What strace ./server sees

strace ./server 观察到的情况

The RST originates from the server, so let’s attach strace and see what we get: RST 源自服务器，所以让我们挂载 strace 看看会得到什么：

No crash. It forked and used sendto() to dump all the data to the client. Then it quit. Also note that sendto() returned the full 600,000, so from the perspective of this program, “all data got sent” (there’s a footnote, obviously, as the manpage explains: “Successful completion of a call to sendto() does not guarantee delivery of the message. A return value of -1 indicates only locally-detected errors.”). In fact, there is no difference here whether you use --spam on the client or not. 没有崩溃。它 fork 了进程并使用 sendto() 将所有数据倾倒给客户端，然后退出。还要注意 sendto() 返回了完整的 600,000，因此从该程序的角度来看，“所有数据都已发送”（显然有一个脚注，正如手册页所解释的：“成功完成 sendto() 调用并不保证消息已送达。返回值为 -1 仅表示本地检测到的错误。”）。事实上，无论客户端是否使用 --spam，这里都没有区别。

What strace ./client —spam sees

strace ./client —spam 观察到的情况

Nothing out of the ordinary here, either. We ran recvfrom() until one of them returned with -1 / ECONNRESET. 这里也没有什么异常。我们运行 recvfrom() 直到其中一个返回 -1 / ECONNRESET。

A first hypothesis

初步假设

Let’s test when we get the ECONNRESET. Because, if you scroll back up, some of the invocations read the full 600,000 bytes and some returned other values. So there’s probably some timing issue here. Let’s do what is most obvious in hindsight: Delay the close() in the server. Because if anything legitimately creates a RST, it’ll probably be that. 让我们测试一下何时会收到 ECONNRESET。因为如果你向上滚动，会发现有些调用读取了完整的 600,000 字节，而有些则返回了其他值。所以这里可能存在某种时序问题。让我们做一件事后看来最明显的事情：延迟服务器中的 close()。因为如果有什么合法的原因导致了 RST，那很可能就是它。

In serve_client(), simply add a sleep(1) before the call to close(): 在 serve_client() 中，只需在调用 close() 之前添加一个 sleep(1)：

sleep(1);
if (close(s) == -1) {
    perror("close");
    return 1;
}

And there you go, we can clearly see a one second delay now. A tcpdump shows it best: So, without having dug deeper, here is some speculation: The server sees the incoming data on its socket, but it doesn’t read them. When we call close() in the server, the socket is “dirty” (because of pending data), so a RST gets fired to (hopefully) tell the client that not all data has been read. This makes sense to me at the moment, although I haven’t found any definitive source. 就这样，我们现在可以清楚地看到一秒钟的延迟。tcpdump 最能说明问题：所以，在没有深入挖掘的情况下，这里有一些推测：服务器在套接字上看到了传入的数据，但它没有读取它们。当我们在服务器中调用 close() 时，套接字是“脏”的（因为有待处理的数据），因此会触发一个 RST，（希望）告诉客户端并非所有数据都已被读取。目前这对我来说是有道理的，尽管我还没有找到任何确凿的来源。