When “Active” Doesn’t Mean Healthy: Debugging PostgreSQL ENOSPC Across a Multi-VM Linux Stack
When “Active” Doesn’t Mean Healthy: Debugging PostgreSQL ENOSPC Across a Multi-VM Linux Stack
当“活跃”不代表健康：跨多虚拟机 Linux 堆栈调试 PostgreSQL ENOSPC 错误
PostgreSQL was active. Nginx was active. PHP-FPM was active. The application still returned: HTTP 503. The actual root cause was a completely full filesystem backing a PostgreSQL tablespace. I built this incident deliberately in my Ubuntu/KVM support lab to test a common troubleshooting mistake: Treating process state as proof of service health. The incident became a useful example of why application health, dependency health, and process health need to be investigated separately. PostgreSQL 处于活跃状态，Nginx 处于活跃状态，PHP-FPM 也处于活跃状态，但应用程序仍然返回 HTTP 503 错误。真正的根本原因是承载 PostgreSQL 表空间的磁盘空间已完全耗尽。我在 Ubuntu/KVM 支持实验室中特意构建了这一故障场景，旨在测试一个常见的排障误区：将进程状态视为服务健康的证明。这一事件成为了一个很好的案例，说明了为什么必须将应用程序健康、依赖项健康和进程健康分开进行调查。
Lab architecture
实验室架构
I used three Ubuntu VMs connected through a private libvirt network: 我使用了三台通过私有 libvirt 网络连接的 Ubuntu 虚拟机：
- Client | v vm-web-01 (192.168.100.10) Nginx + PHP-FPM
- TCP/5432 | v vm-db-01 (192.168.100.20) PostgreSQL 14
- node_exporter | v vm-monitor-01 (192.168.100.30) Prometheus + Alertmanager
The database VM also had a dedicated disk mounted at: /mnt/inc012-db. That filesystem was used for the PostgreSQL tablespace, which meant I could create storage pressure without filling the VM’s root filesystem.
数据库虚拟机还挂载了一块专用磁盘，挂载点为 /mnt/inc012-db。该文件系统用于存放 PostgreSQL 表空间，这意味着我可以在不填满虚拟机根文件系统的情况下制造存储压力。
Establishing the baseline
建立基准
Before introducing the failure, I checked the application, database, filesystem, and monitoring state. 在引入故障之前，我检查了应用程序、数据库、文件系统和监控状态。
-
The PostgreSQL tablespace filesystem was: 8.03% used
-
The application returned: HTTP 200
-
Database writes succeeded.
-
Prometheus showed:
DBDiskNearlyFull state=inactive health=ok -
PostgreSQL 表空间文件系统使用率为：8.03%
-
应用程序返回：HTTP 200
-
数据库写入成功。
-
Prometheus 显示：
DBDiskNearlyFull state=inactive health=ok
This gave me a known-good state to compare against during the incident. 这为我在故障期间提供了一个可供对比的已知良好状态。
Monitoring detected the problem first
监控率先发现问题
I then consumed storage on the dedicated database filesystem in a controlled way. When usage reached: 88.49%, Prometheus changed the storage alert to: DBDiskNearlyFull firing. But the application was still returning: HTTP 200.
随后，我以受控方式占用了专用数据库文件系统的存储空间。当使用率达到 88.49% 时，Prometheus 将存储警报更改为：DBDiskNearlyFull firing。但应用程序仍然返回：HTTP 200。
This was one of the most important observations in the test. The monitoring system had identified a capacity problem before the customer-facing service failed. That is exactly what useful monitoring should do: create an intervention window before an operational condition becomes an outage. 这是测试中最关键的观察结果之一。监控系统在面向客户的服务失败之前就识别出了容量问题。这正是有效的监控应该做到的：在运行状况演变为服务中断之前，创造一个干预窗口。
Then the filesystem reached 100%
随后文件系统达到 100%
I continued the controlled storage consumption until the dedicated filesystem reached: 100%. The database VM’s root filesystem remained healthy. PostgreSQL also continued reporting: active. So at a quick glance, the database server could appear healthy. It wasn’t.
我继续进行受控的存储占用，直到专用文件系统达到 100%。数据库虚拟机的根文件系统保持健康。PostgreSQL 也继续报告为 active。因此，乍一看，数据库服务器似乎是健康的。但事实并非如此。
Testing the operation that actually mattered
测试真正关键的操作
A connectivity test alone wasn’t enough. PostgreSQL could still accept connections, but when I performed a write that required additional filesystem blocks, it failed with: ERROR: could not extend file No space left on device. The PostgreSQL logs showed the same ENOSPC condition.
仅进行连通性测试是不够的。PostgreSQL 仍然可以接受连接，但当我执行需要额外文件系统块的写入操作时，它失败并报错：ERROR: could not extend file No space left on device。PostgreSQL 日志也显示了相同的 ENOSPC（设备上没有空间）状况。
That exposed the distinction I was looking for: PostgreSQL process health != PostgreSQL write health != Application health. A running database process does not necessarily mean the database can complete the operations the application depends on. 这揭示了我所寻找的区别：PostgreSQL 进程健康 != PostgreSQL 写入健康 != 应用程序健康。正在运行的数据库进程并不一定意味着数据库能够完成应用程序所依赖的操作。
The user-facing failure appeared at the web tier
面向用户的故障出现在 Web 层
The actual failure originated in database storage. But the visible symptom appeared somewhere else. The web endpoint changed from: HTTP 200 to: HTTP 503 {"status":"degraded","database":"write_failed"}.
实际故障源于数据库存储，但可见的症状却出现在其他地方。Web 端点从 HTTP 200 变为 HTTP 503 {"status":"degraded","database":"write_failed"}。
At the same time: 与此同时：
-
Nginx active
-
PHP-FPM active
-
PostgreSQL active
-
Nginx 活跃
-
PHP-FPM 活跃
-
PostgreSQL 活跃
All three processes were alive while the application was degraded. If my investigation had stopped at systemctl is-active, I could easily have concluded that the web and database layers were fine. They weren’t.
在应用程序降级时，所有三个进程都处于存活状态。如果我的调查停留在 systemctl is-active，我很容易得出 Web 层和数据库层运行正常的结论。但事实并非如此。
My investigation path
我的调查路径
I worked down the dependency chain instead: 我转而沿着依赖链向下排查：
HTTP 503 ↓ Check Nginx and PHP-FPM ↓ Processes active ↓ Check PostgreSQL reachability ↓ Database reachable ↓ Test an actual database write ↓ Write fails with ENOSPC ↓ Inspect PostgreSQL error ↓ Check tablespace filesystem ↓ Filesystem 100% full ↓ Correlate with Prometheus ↓ Storage alert already firing. HTTP 503 ↓ 检查 Nginx 和 PHP-FPM ↓ 进程活跃 ↓ 检查 PostgreSQL 可达性 ↓ 数据库可达 ↓ 测试实际的数据库写入 ↓ 写入因 ENOSPC 失败 ↓ 检查 PostgreSQL 错误 ↓ 检查表空间文件系统 ↓ 文件系统 100% 满 ↓ 与 Prometheus 关联 ↓ 存储警报已触发。
The key shift was moving from: “Is the process running?” to: “Can this dependency perform the operation the application requires?” That narrowed the failure to storage rather than networking, web-server availability, process crashes, or general VM capacity. 关键的转变在于从“进程是否在运行？”转向“该依赖项能否执行应用程序所需的操作？”。这缩小了故障范围，将其锁定在存储问题上，而不是网络、Web 服务器可用性、进程崩溃或常规虚拟机容量问题。
Recovery
恢复
Because the disk pressure had been generated intentionally for the lab, I removed the temporary filler data from the isolated database filesystem. I did not restart PostgreSQL. After the storage was released, filesystem usage returned to: 8.27%. PostgreSQL writes succeeded again. The application returned: HTTP 200. Prometheus returned the storage alert to inactive, and Alertmanager had no remaining active DBDiskNearlyFull alert.
由于磁盘压力是为实验室特意制造的，我从隔离的数据库文件系统中删除了临时填充数据。我没有重启 PostgreSQL。存储释放后，文件系统使用率恢复到 8.27%。PostgreSQL 写入再次成功。应用程序返回 HTTP 200。Prometheus 将存储警报恢复为非活动状态，Alertmanager 中也没有剩余的 DBDiskNearlyFull 活动警报。
Before, failure and recovery
故障前、故障中与恢复后对比
| Check | Baseline | Failure | Recovery |
|---|---|---|---|
| DB filesystem | 8.03% | 100% | 8.27% |
| PostgreSQL process | Active | Active | Active |
| PostgreSQL writes | Successful | ENOSPC | Successful |
| Application | HTTP 200 | HTTP 503 | HTTP 200 |
| Prometheus alert | Inactive | Firing | Inactive |
| Alertmanager | Clear | Active | Clear |
| 检查项 | 基准 | 故障 | 恢复 |
|---|---|---|---|
| 数据库文件系统 | 8.03% | 100% | 8.27% |
| PostgreSQL 进程 | 活跃 | 活跃 | 活跃 |
| PostgreSQL 写入 | 成功 | ENOSPC | 成功 |
| 应用程序 | HTTP 200 | HTTP 503 | HTTP 200 |
| Prometheus 警报 | 非活动 | 触发 | 非活动 |
| Alertmanager | 清除 | 活动 | 清除 |
The most interesting line is still: PostgreSQL process Active → Active → Active. The process never stopped. The service still failed from the application’s perspective. 最有趣的一行仍然是：PostgreSQL 进程 活跃 → 活跃 → 活跃。进程从未停止，但从应用程序的角度来看，服务仍然失败了。
What I took away from the incident
我从这次事件中学到的
- Process status is only one health signal:
systemctl is-activetells me whether systemd considers a process active. It does not prove that the service can perform useful work. 进程状态只是健康信号之一：systemctl is-active只能告诉我 systemd 是否认为进程处于活跃状态，它不能证明该服务能够执行有效的工作。 - Check the dependency the application actually uses: For a PostgreSQL-backed application, TCP connectivity or
pg_isreadyis useful, but it may not be enough. The application needed database writes. So write capability was the meaningful test during this incident. 检查应用程序实际使用的依赖项：对于基于 PostgreSQL 的应用程序，TCP 连通性或pg_isready很有用，但可能还不够。应用程序需要数据库写入，因此写入能力才是本次事件中有意义的测试。 - Monitoring should create time to respond: The alert fired at 88.49% while the application was still healthy. That gap between warning and outage is operationally valuable. Capacity alerts should give engineers enough time to investigate and remediate before the filesystem reaches exhaustion. 监控应创造响应时间：警报在 88.49% 时触发，而此时应用程序仍然健康。警告与中断之间的这段时间在运维上非常有价值。容量警报应给予工程师足够的时间，在文件系统耗尽之前进行调查和修复。
- Validate recovery through the entire stack: Freeing disk space was not enough for me to call the incident resolved. I checked: Filesystem capacity → PostgreSQL write → application HTTP response → Prometheus state → Alertmanager state. Only after all of those recovered did I consider the incident validated. 通过整个堆栈验证恢复情况：释放磁盘空间不足以让我宣布事件已解决。我检查了：文件系统容量 → PostgreSQL 写入 → 应用程序 HTTP 响应 → Prometheus 状态 → Alertmanager 状态。只有在所有这些都恢复后，我才认为事件已得到验证。
- Restarting isn’t automatically the fix: (Post…) 重启并不总是万能的修复方案：（Post…）