I Built a Self-Healing Web App (And It Took a Tab Character to Break My Will to Live) Or: a love letter to restart: always

The idea What if your web app could detect that it’s dead and just… come back? No PagerDuty. No 3am phone call. No “sorry, deploying a fix.” Just a container that notices it’s down and says “nah, I’m fine now” within seconds. That’s what I built. And I’m going to walk you through the whole thing including the parts where I’m an idiot.

这个想法是：如果你的 Web 应用能检测到自己挂了，然后……自动恢复呢？不需要 PagerDuty，不需要凌晨三点的电话，也不需要说“抱歉，正在部署修复”。只需要一个容器，它能察觉到自己宕机了，并在几秒钟内说“不，我现在好了”。这就是我构建的东西。我将带你了解整个过程，包括我表现得像个白痴的部分。

self-healing-app/ ├── app/ # Flask app (the thing that dies) │ ├── app.py │ └── Dockerfile ├── nginx/ # Reverse proxy │ └── nginx.conf ├── prometheus/ # Monitoring + alerting │ ├── prometheus.yml │ └── alert.rules.yml └── docker-compose.yml Four containers on a bridge network: Container Job web-app Flask app with /health and /metrics endpoints nginx-proxy Routes traffic to the app prometheus Scrapes metrics every 5s, fires alerts grafana Makes it look like you know what you're doing Five seconds of up == 0 and the alert fires. No flapping on a single missed scrape.

self-healing-app/ ├── app/ # Flask 应用（会挂掉的那个） │ ├── app.py │ └── Dockerfile ├── nginx/ # 反向代理 │ └── nginx.conf ├── prometheus/ # 监控 + 告警 │ ├── prometheus.yml │ └── alert.rules.yml └── docker-compose.yml 桥接网络上的四个容器：容器任务 web-app 带有 /health 和 /metrics 端点的 Flask 应用 nginx-proxy 将流量路由到应用 prometheus 每 5 秒抓取一次指标，触发告警 grafana 让看起来你很专业 连续 5 秒 up == 0 就会触发告警。不会因为单次抓取失败而抖动。

The debugging journey (a.k.a. the real tutorial) Problem 1: docker-compose not found. Fix: It’s docker compose (no hyphen) now. The old v1 binary is deprecated. Install docker-compose-plugin. **Problem 2: **Port 9090 already in use. Fix: sudo lsof -i :9090 → it was a leftover process. Killed it. Problem 3: Port 80 already in use. Fix: System nginx was running. sudo systemctl stop nginx && sudo systemctl disable nginx. Problem 4: Prometheus target stuck at health: down with error received unsupported Content-Type “text/html”. Fix: Flask returns text/html for string responses. Changed to: return Response(“up 1\n”, content_type=“text/plain; version=0.0.4; charset=utf-8”) Problem 5: A tab character in alert.rules.yml. Fix: sed -i ‘s/\t/ /g’ alert.rules.yml. I will never use tabs in YAML again. Problem 6: I pasted a YAML healthcheck block into my Python file. Fix: I don’t talk about this one.

调试之旅（也就是真正的教程）：问题 1：找不到 docker-compose。修复：现在是 docker compose（没有连字符）。旧的 v1 二进制文件已被弃用。安装 docker-compose-plugin。问题 2：端口 9090 已被占用。修复：sudo lsof -i :9090 → 是一个残留进程。杀掉它。问题 3：端口 80 已被占用。修复：系统 nginx 正在运行。sudo systemctl stop nginx && sudo systemctl disable nginx。问题 4：Prometheus 目标卡在 health: down，错误提示收到不支持的 Content-Type “text/html”。修复：Flask 对字符串响应返回 text/html。改为：return Response(“up 1\n”, content_type=“text/plain; version=0.0.4; charset=utf-8”)。问题 5：alert.rules.yml 中有一个制表符。修复：sed -i ‘s/\t/ /g’ alert.rules.yml。我再也不会在 YAML 中使用制表符了。问题 6：我把一个 YAML 健康检查块粘贴到了 Python 文件里。修复：我不谈论这个。

Testing the self-healing Kill the app docker kill web-app Watch Prometheus (http://localhost:9090) - Target goes red - up metric drops to 0 - Alert fires after 5s - Docker restarts the container - Target goes green - Alert resolves Total downtime: ~5-10 seconds. No human required.

测试自愈：杀掉应用 docker kill web-app。观察 Prometheus (http://localhost:9090) - 目标变红 - up 指标降为 0 - 5 秒后触发告警 - Docker 重启容器 - 目标变绿 - 告警解除。总停机时间：约 5-10 秒。无需人工干预。

Running it docker compose up —build -d App: http://localhost Prometheus: http://localhost:9090 Grafana: http://localhost:3000 (admin/admin) Add Prometheus as a data source in Grafana (URL: http://prometheus:9090), import dashboard ID 1860, and you’ve got a monitoring wall that would make your DevOps friend jealous.

运行它：docker compose up —build -d。应用：http://localhost，Prometheus：http://localhost:9090，Grafana：http://localhost:3000 (admin/admin)。在 Grafana 中添加 Prometheus 作为数据源（URL: http://prometheus:9090），导入仪表板 ID 1860，你就拥有了一个会让你的 DevOps 朋友嫉妒的监控墙。

What’s next Migrate to Azure Container Apps (managed, serverless-ish, auto-scaling) Add Alertmanager for actual notifications (Slack, email, PagerDuty) Automate failure injection (no more manual docker kill) Add node_exporter for host-level metrics Maybe a Terraform module for the whole thing

下一步：迁移到 Azure Container Apps（托管、类无服务器、自动扩缩容），添加 Alertmanager 以实现实际通知（Slack、电子邮件、PagerDuty），自动化故障注入（不再需要手动 docker kill），添加 node_exporter 以获取主机级指标，也许为整个项目编写一个 Terraform 模块。

Final thoughts The “self-healing” part is the easiest bit. The hard part is making sure your monitoring is actually monitoring — that the scrape works, the content type is right, the network is correct, the YAML doesn’t have a rogue tab. The system doesn’t heal itself. You heal it first. Then it heals itself. Then you sleep.

最后的话：“自愈”部分是最简单的。困难的部分是确保你的监控确实在工作——抓取正常、内容类型正确、网络通畅、YAML 中没有流氓制表符。系统不会自动愈合。你先治愈它。然后它才会自愈。然后你就可以睡觉了。