A practical guide · back to the rollout safety checker
A RollingUpdate deploy reports success. kubectl rollout status says the deployment is available. And a handful of users got a 502 or a connection reset while it was happening — not enough to page anyone, just enough to show up as a small, unexplained error spike that recurs on exactly the days you deploy.
This is not a bug in Kubernetes, and it is not usually a bug in the application either. It is the default behaviour of a manifest that has not been told otherwise. There are two separate races, one on each side of a pod's life, and they need different fixes.
Because without a readinessProbe, Ready is defined as "the container process started". Kubernetes has no other way to know whether the application inside can serve a request, so it assumes it can. The Service adds the pod to its EndpointSlice immediately and the load balancer starts routing to it — while the app is still opening database connections, loading configuration, or warming a cache. Every request that lands in that window fails, on every rollout, and the rollout still reports success.
The fix is a readiness probe that actually fails while the app cannot serve:
readinessProbe:
httpGet:
path: /readyz
port: 8080
periodSeconds: 5
failureThreshold: 2The endpoint matters as much as the probe. A /readyz that returns 200 from a static handler proves only that the HTTP server is listening — which is almost the same thing as no probe at all. It should check the dependencies a real request needs, and report not-ready while any of them is unavailable.
Because pod deletion starts two independent processes at the same instant, and neither waits for the other. The kubelet sends SIGTERM to the container. Separately, the endpoints controller removes the pod from EndpointSlice, and that removal then has to propagate to every kube-proxy on every node and to every ingress controller. Propagation takes time. An application that shuts down promptly and correctly on SIGTERM can therefore finish before the proxies have stopped routing to it, and every connection they open in that window is refused.
This one surprises people because the application is behaving well. Handling SIGTERM quickly makes it worse, not better: the faster the container exits, the wider the window in which proxies are still sending it work.
The fix is a preStop hook whose entire job is to wait, so the removal has time to land before shutdown begins:
lifecycle:
preStop:
sleep:
seconds: 10 # Kubernetes 1.29+
# older clusters:
# exec:
# command: ["/bin/sh", "-c", "sleep 10"]
terminationGracePeriodSeconds: 45Long enough for EndpointSlice removal to reach every proxy, which depends on cluster size and ingress controller. Five to fifteen seconds is a common starting point, and the way to tune it is to measure: send steady traffic, delete a pod, and see whether any request fails. The important part is that terminationGracePeriodSeconds must exceed the preStop wait plus the longest request the application needs to finish — the grace period covers both, and SIGKILL at the end of it does not care what is still in flight.
That last point is the trap worth stating plainly. The grace period clock starts when the pod is deleted, not when preStop returns. A 30-second preStop under the default 30-second grace period means the container is killed at the exact moment it was finally asked to shut down, so it never drains at all — a configuration that looks more careful than having no preStop hook, and behaves worse.
Kubernetes resolves maxUnavailable by rounding down and maxSurge by rounding up. The 25%/25% default at 3 replicas therefore means maxUnavailable: 0, maxSurge: 1 — a replacement starts and becomes ready before anything is removed, which is the behaviour you want. It is worth knowing the default is usually fine, because the instinct on reading an article like this one is to change every setting, and most of the damage here comes from a value someone set deliberately:
strategy:
rollingUpdate:
maxUnavailable: 0 # never drop below the current capacity
maxSurge: 1 # start one extra, wait for Ready, then retire oneNot directly — a Deployment's own rollout is governed by maxUnavailable and maxSurge, not by the budget. A PodDisruptionBudget governs other voluntary disruptions, and the one that matters most is a node drain during a cluster upgrade, which is otherwise free to evict every replica of a workload at once. It is also possible to set a budget that can never be satisfied (minAvailable equal to the replica count), and that failure is unusually quiet: kubectl drain does not error, it blocks and retries indefinitely, so the upgrade appears to hang.
Check your own manifest. The rollout safety checker reads a Deployment and reports every one of these, with the YAML to fix each. It runs in your browser or as a zero-dependency CLI, never connects to a cluster, and uploads nothing.
maxSurge: 1 when capacity is tight, so the service never dips below its current size.