TLS Fails Only Inside the Container: Fixing "certificate signed by unknown authority"

本文为原文前 6,000 字符的节选翻译,完整内容请查看原文。

If the same HTTPS request works on your laptop and fails inside the container, the problem is almost never the network — it’s that the container has a different (or empty) trust store than your machine. Your laptop trusts the corporate root CA your proxy re-signs traffic with, or ships a full CA bundle; a slim base image may ship none.

如果同样的 HTTPS 请求在你的笔记本电脑上可以正常工作,但在容器内却失败了,问题几乎从来不是网络原因,而是容器的信任存储(trust store)与你的机器不同(或为空)。你的笔记本电脑信任代理服务器重签流量所使用的企业根证书颁发机构(CA),或者内置了完整的 CA 证书包;而精简的基础镜像可能什么都没有。

The fix is to install the right root certificate into the image’s trust store and, separately, to make sure your language runtime actually reads that trust store — several don’t. What does the error look like in each stack? The underlying cause is identical; the message is not. These are the four you’ll meet most often:

解决方法是将正确的根证书安装到镜像的信任存储中,并确保你的语言运行时确实会读取该信任存储——有些运行时并不会这样做。在不同的技术栈中,错误表现是什么样的?根本原因是一样的,但提示信息不同。以下是你最常遇到的四种情况:

Go

Get “https://api.example.com/v1/ping”: tls: failed to verify certificate: x509: certificate signed by unknown authority

Go

Get “https://api.example.com/v1/ping”: tls: failed to verify certificate: x509: certificate signed by unknown authority

Python (requests)

requests.exceptions.SSLError: HTTPSConnectionPool(host=‘api.example.com’, port=443): Max retries exceeded … [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)

Python (requests)

requests.exceptions.SSLError: HTTPSConnectionPool(host=‘api.example.com’, port=443): Max retries exceeded … [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)

Node.js

Error: unable to verify the first certificate code: ‘UNABLE_TO_VERIFY_LEAF_SIGNATURE’

Node.js

Error: unable to verify the first certificate code: ‘UNABLE_TO_VERIFY_LEAF_SIGNATURE’

curl

curl: (60) SSL certificate problem: unable to get local issuer certificate

curl

curl: (60) SSL certificate problem: unable to get local issuer certificate

What tripped me up the first time: the Node and curl wording (“unable to verify the first certificate”) points at the server, while the Go wording points at the trust store. They are the same two failure modes wearing different hats, and you cannot tell which one you have from the application log alone. Takeaway: the error text tells you verification failed, not whose fault it was — stop guessing and go look at the chain. Is it a missing root, or a chain the server never sent? There are two distinct bugs here, and patching the wrong one wastes an afternoon.

第一次让我困惑的是:Node 和 curl 的措辞(“无法验证第一个证书”)指向服务器,而 Go 的措辞则指向信任存储。它们是披着不同外衣的同两种故障模式,仅凭应用程序日志无法判断你遇到的是哪一种。结论:错误文本只告诉你验证失败,并没有告诉你是谁的错——别再瞎猜了,去检查证书链。是缺少根证书,还是服务器根本没发送证书链?这里有两种截然不同的错误,修补错误的对象只会浪费一下午的时间。

Missing/incomplete trust store. The server sends a perfectly good chain, but the container has no root certificate that signs it. Typical in alpine, scratch, and hand-rolled minimal images, and universal behind a TLS-inspecting corporate proxy whose root CA only exists on managed laptops.

信任存储缺失或不完整。服务器发送了完美的证书链,但容器中没有签署该证书的根证书。这种情况在 alpine、scratch 和手动构建的最小化镜像中很常见,在使用 TLS 检测的企业代理(其根 CA 仅存在于受管笔记本电脑上)的环境中更是普遍。

Broken chain from the server. The server sends its leaf certificate but omits the intermediate. Browsers and macOS/Windows paper over this by fetching the missing intermediate from the AIA extension in the certificate; OpenSSL, Go, and Python do not. So it works everywhere except your service.

服务器证书链损坏。服务器发送了叶子证书,但遗漏了中间证书。浏览器和 macOS/Windows 会通过从证书的 AIA 扩展中获取缺失的中间证书来掩盖这个问题;但 OpenSSL、Go 和 Python 不会。因此,除了你的服务外,它在其他地方都能正常工作。

Run this from inside the failing container: openssl s_client -connect api.example.com:443
-servername api.example.com -showcerts </dev/null 2>/dev/null
| grep -E ‘Verify return code|^subject=|^issuer=’

在故障容器内运行以下命令: openssl s_client -connect api.example.com:443
-servername api.example.com -showcerts </dev/null 2>/dev/null
| grep -E ‘Verify return code|^subject=|^issuer=’

Read the verify code: 20 (unable to get local issuer certificate) means the chain walked up to an issuer your container doesn’t trust — case 1. 21 (unable to verify the first certificate) usually means the server handed you a leaf with nothing above it — case 2.

读取验证代码:20(无法获取本地颁发者证书)意味着证书链追溯到了你的容器不信任的颁发者——属于情况 1。21(无法验证第一个证书)通常意味着服务器只给了你一个叶子证书,上面没有任何内容——属于情况 2。

Counting what the server actually sent settles it: openssl s_client -connect api.example.com:443 -servername api.example.com
-showcerts </dev/null 2>/dev/null | grep -c ‘BEGIN CERTIFICATE’

计算服务器实际发送了多少证书可以确定问题: openssl s_client -connect api.example.com:443 -servername api.example.com
-showcerts </dev/null 2>/dev/null | grep -c ‘BEGIN CERTIFICATE’

One certificate for a publicly-signed endpoint is a server-side misconfiguration — fix it there (most reverse proxies want the fullchain file, not the leaf), because every other client will eventually hit it too. Also check subject= on the top certificate: if it says something like your employer’s name, you’re behind an inspecting proxy and need its root installed. Takeaway: one BEGIN CERTIFICATE in the handshake means fix the server; a code 20 with a complete chain means fix the image.

对于公共签名的端点,如果只发送了一个证书,那就是服务器端的配置错误——请在服务器端修复它(大多数反向代理需要完整证书链文件,而不是叶子证书),因为其他所有客户端最终也会遇到同样的问题。另外,检查顶部证书的 subject=:如果它显示的是你雇主的名字,说明你处于检测代理之后,需要安装其根证书。结论:握手中只有一个 BEGIN CERTIFICATE 意味着要修复服务器;如果代码为 20 且证书链完整,则意味着要修复镜像。

How do I install a custom root CA in a container image? Put the PEM-encoded root in the distro’s anchor directory and run the distro’s update command — at build time, as root, in the final stage.

我该如何在容器镜像中安装自定义根 CA?将 PEM 编码的根证书放入发行版的锚点目录中,并运行发行版的更新命令——在构建时,以 root 身份,在最终阶段执行。

Debian/Ubuntu

FROM python:3.12-slim COPY corp-root.crt /usr/local/share/ca-certificates/corp-root.crt RUN apt-get update && apt-get install -y —no-install-recommends ca-certificates
&& update-ca-certificates
&& rm -rf /var/lib/apt/lists/*

Debian/Ubuntu

FROM python:3.12-slim COPY corp-root.crt /usr/local/share/ca-certificates/corp-root.crt RUN apt-get update && apt-get install -y —no-install-recommends ca-certificates
&& update-ca-certificates
&& rm -rf /var/lib/apt/lists/*

Alpine

FROM alpine:3.20 COPY corp-root.crt /usr/local/share/ca-certificates/corp-root.crt RUN apk add —no-cache ca-certificates && update-ca-certificates

Alpine

FROM alpine:3.20 COPY corp-root.crt /usr/local/share/ca-certificates/corp-root.crt RUN apk add —no-cache ca-certificates && update-ca-certificates

On RHEL-family images (including Red Hat UBI) the paths differ: drop the file in /etc/pki/ca-trust/source/anchors/ and run update-ca-trust. Two details that cause silent failures: the file extension must be .crt on Debian/Alpine or update-ca-certificates skips it, and the content must be PEM (-----BEGIN CERTIFICATE-----), not DER — convert with openssl x509 -inform der -in corp.der -out corp.crt.

在 RHEL 系列镜像(包括 Red Hat UBI)中,路径有所不同:将文件放入 /etc/pki/ca-trust/source/anchors/ 并运行 update-ca-trust。有两个细节会导致静默失败:在 Debian/Alpine 上文件扩展名必须是 .crt,否则 update-ca-certificates 会跳过它;内容必须是 PEM 格式(-----BEGIN CERTIFICATE-----),而不是 DER——可以使用 openssl x509 -inform der -in corp.der -out corp.crt 进行转换。

Verify inside the built image rather than trusting the build log: docker run —rm your-image sh -c
‘curl -sS -o /dev/null -w ”%{http_code}\n” https://api.example.com/healthz’

在构建好的镜像内进行验证,而不是盲目相信构建日志: docker run —rm your-image sh -c
‘curl -sS -o /dev/null -w ”%{http_code}\n” https://api.example.com/healthz’

Takeaway: a root CA added at runtime via a mounted volume without running the update command does nothing — the bundle is generated, not scanned on demand.

结论:通过挂载卷在运行时添加的根 CA,如果不运行更新命令是不会生效的——证书包是在构建时生成的,而不是按需扫描的。

Which runtimes ignore the system trust store? This is the part that makes the bug feel haunted: you install the root, curl inside the container works, and the app still fails. That’s because not every runtime reads /etc/ssl/certs.

哪些运行时会忽略系统信任存储?这正是让这个 Bug 看起来像“闹鬼”的原因:你安装了根证书,容器内的 curl 可以正常工作,但应用程序仍然失败。这是因为并非每个运行时都会读取 /etc/ssl/certs。

Node is the classic trap: it ships its own CA list, so a system-installed corporate root is invisible to it. Set NODE_EXTRA_CA_CERTS to a file (it may contain several concatenated PEM blocks, but it must be one file), or start with —use-openssl-ca to use the OpenSSL store instead. Python’s requests is the same story through certifi; if you’d rather have Python defer to the OS store entirely, the truststore package does that and is the cleaner answer for corporate-proxy environments.

Node 是一个典型的陷阱:它自带 CA 列表,因此系统安装的企业根证书对它来说是不可见的。请将 NODE_EXTRA_CA_CERTS 设置为一个文件(它可以包含多个拼接的 PEM 块,但必须是一个文件),或者使用 —use-openssl-ca 启动以改用 OpenSSL 存储。Python 的 requests 库通过 certifi 也有同样的问题;如果你希望 Python 完全依赖操作系统存储,truststore 包可以实现这一点,这对于企业代理环境来说是一个更简洁的解决方案。