Three Silent Failures Between You and Sidecar Injection

Three Silent Failures Between You and Sidecar Injection

阻碍 Sidecar 注入的三种静默故障

Running kubectl label deployment my-app pandocore.io/inject=enabled modifies the Deployment’s top-level metadata.labels, not spec.template.metadata.labels. Those are two different things, and the difference is the bug. 运行 kubectl label deployment my-app pandocore.io/inject=enabled 命令只会修改 Deployment 顶层的 metadata.labels,而不是 spec.template.metadata.labels。这是两个完全不同的概念,而这种差异正是问题的根源。

Sidecar injection works through a mutating admission webhook that fires when a pod is created. The webhook matches on the pod’s labels. Pods inherit their labels from exactly one place: the Deployment’s pod template. Labels on the Deployment object itself never propagate down to pods. They exist so you can select the Deployment with things like kubectl get deploy -l team=payments. Sidecar 注入是通过一个变更准入控制器(mutating admission webhook)实现的,该控制器在 Pod 创建时触发。Webhook 会匹配 Pod 的标签。Pod 的标签仅来源于一个地方:Deployment 的 Pod 模板。Deployment 对象本身的标签永远不会下发到 Pod。这些标签的存在只是为了让你能通过类似 kubectl get deploy -l team=payments 的命令来筛选 Deployment。

So when you label the Deployment, three things go wrong at once, all silently. First, kubectl prints deployment.apps/my-app labeled. Success. Second, because the pod template didn’t change, the Deployment controller sees nothing to do, so no rollout happens and no new pods are created. Third, even if pods were recreated, they wouldn’t carry the label, so the webhook would never match them. 因此,当你给 Deployment 打标签时,会同时发生三个静默故障。首先,kubectl 会输出 deployment.apps/my-app labeled,显示操作成功。其次,由于 Pod 模板没有改变,Deployment 控制器认为无需执行任何操作,因此不会触发滚动更新,也不会创建新 Pod。第三,即使 Pod 被重新创建,它们也不会携带该标签,导致 Webhook 永远无法匹配到它们。

There is no error at any layer. You check the Deployment, the label is right there, and every pod is still running 1/1 with no sidecar in sight. We hit this while reviewing our own onboarding docs. The instruction looked correct, the command succeeded, and injection never happened. It’s an easy mistake to ship because nothing anywhere tells you it didn’t work. 在任何层面都不会报错。你检查 Deployment,标签确实在那里,但每个 Pod 依然显示 1/1 运行状态,且看不到任何 Sidecar。我们在审查自己的入职文档时就遇到了这个问题。指令看起来是正确的,命令也执行成功了,但注入从未发生。这是一个很容易犯的错误,因为没有任何地方会提示你操作失败了。

The fix is to put the label where pods are actually born: the pod template. 解决方法是将标签放在 Pod 真正诞生的地方:Pod 模板中。

kubectl patch deployment my-app --type merge \
-p '{"spec":{"template":{"metadata":{"labels":{"pandocore.io/inject":"enabled"}}}}}'

Because this changes the pod template, it triggers a rollout on its own. The controller creates a new ReplicaSet, and every new pod comes up with the label and passes through the webhook. 因为这改变了 Pod 模板,它会自动触发滚动更新。控制器会创建一个新的 ReplicaSet,每个新创建的 Pod 都会带有该标签,并顺利通过 Webhook 的处理。

If you set the label some other way and aren’t sure the template has changed, kubectl rollout restart deployment my-app forces fresh pods. Verify with kubectl get pods -l pandocore.io/inject=enabled. The containers column should now read 2/2. 如果你通过其他方式设置了标签,且不确定模板是否已更改,可以使用 kubectl rollout restart deployment my-app 强制更新 Pod。使用 kubectl get pods -l pandocore.io/inject=enabled 进行验证。此时容器列应该显示为 2/2。

Label the template, roll the pods, check the count. The Deployment’s own labels were never going to do it. 给模板打标签,滚动更新 Pod,检查容器数量。Deployment 本身的标签是永远无法实现注入的。