How to Fix: Kubernetes Readiness Probe Failed

The Scenario

You’re rolling out a new version of your payment-api service via a deployment, and the rollout hangs. kubectl get pods shows the new pods stuck in a Running state but not Ready, and the old pods are still serving traffic. The deployment is stuck, and your CI pipeline times out—right before a major sales event.

Symptoms

kubectl get pods shows 0/1 ready containers for new pods.
kubectl describe pod <pod-name> shows Warning Unhealthy events with Readiness probe failed: HTTP probe failed with statuscode: 503.
The deployment rollout is stuck, and kubectl rollout status hangs indefinitely.

Root Cause

The readiness probe is failing because the application’s /healthz endpoint is returning an HTTP 503 (Service Unavailable) instead of a 200 OK. This is often caused by the application starting up but not yet being fully initialized to accept traffic—e.g., it’s still warming a cache, connecting to a database, or loading configuration. The initialDelaySeconds in the probe spec is too short, or the application doesn’t signal readiness correctly during its startup window.

Resolution (Step-by-Step)

Check the probe configuration in your deployment YAML:
```
kubectl get deployment payment-api -o yaml | grep -A 15 readinessProbe
```
Look for httpGet, initialDelaySeconds, periodSeconds, and failureThreshold.

Increase initialDelaySeconds: If the app takes ~10 seconds to start, but initialDelaySeconds is set to 5, the probe fires too early. Bump it to 15-20 seconds:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 20
  periodSeconds: 10
  failureThreshold: 3

Verify the application’s health endpoint: Exec into a failing pod and test manually:
```
kubectl exec -it <pod-name> -- curl -v http://localhost:8080/healthz
```
If you get a 503, the app logic is the problem—check its logs for initialization failures.
Check application logs for startup errors:
```
kubectl logs <pod-name> --tail=50
```
Look for connection errors (e.g., “cannot connect to database”) or config loading failures.
If the app is healthy but probe still fails, switch to a tcpSocket probe: Some apps may not have an HTTP endpoint ready. Use a TCP probe instead:
```
readinessProbe:
  tcpSocket:
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
```

Why This Sometimes Doesn’t Work

If you increase initialDelaySeconds but the app still fails, the issue might be a circular dependency: your app’s /healthz endpoint checks the database, but the database is also behind the same service and hasn’t started yet. In that case, the probe will never succeed until the database is ready. The fix is to decouple the health check—use a lightweight /livez endpoint for liveness and a separate /readyz endpoint for readiness that only checks app-internal state, not downstream dependencies.

Verification

After applying the fix, watch the pods become ready:

kubectl get pods -l app=payment-api -w

Expected output:

payment-api-7d9f8c6b4c-abc12   0/1   Running   0   10s
payment-api-7d9f8c6b4c-abc12   1/1   Running   0   25s

Also verify the rollout completes:

kubectl rollout status deployment/payment-api

Expected: deployment "payment-api" successfully rolled out

Common Follow-up Questions

Q: Why is my readiness probe failing even though the app is running? The app may be running but not listening on the expected port, or the health endpoint might be returning a non-2xx status code. Exec into the pod and curl the endpoint directly to confirm.

Q: What’s the difference between liveness and readiness probes? Liveness probes restart the container if they fail; readiness probes stop sending traffic to the pod. A failed readiness probe alone won’t restart the pod—it just removes it from the service’s endpoints.

Q: Can I use a command probe instead of HTTP? Yes, but HTTP and TCP probes are more idiomatic in Kubernetes. Command probes execute a shell command inside the container, which can be slower and more resource-intensive.

Prevent This in the Future

This class of issue—silent startup failures that block rollouts—is best caught before it hits production by monitoring pod readiness transitions and deployment rollout status in real time. Teams that already use Better Stack can set up alerts for when pods fail to become ready after a deploy, catching these problems within seconds.
Set up uptime monitoring and alerts with Better Stack