Fixing ImagePullBackOff: every cause, fastest check first

A kubernetes ImagePullBackOff has exactly one honest meaning: the kubelet asked a registry for an image and the registry said no. Everything else - the backoff, the restarts, the red dashboard tile - is decoration. The registry's actual refusal is sitting in the pod events, and that's always the first move:

kubectl describe pod <pod> | grep -A8 Events

Read the Failed event message word for word. It contains the registry's own error string, and that string maps almost one-to-one onto a cause. Here's the full diagnostic tree, ordered by how often each cause is the answer and how cheap it is to check.

ErrImagePull vs ImagePullBackOff

Quick vocabulary. ErrImagePull is a single failed pull attempt. ImagePullBackOff is Kubernetes giving up on retrying quickly - it will keep trying, with exponentially longer waits, capped at five minutes. They are the same problem at different ages. If you fix the underlying cause, you don't have to wait out the backoff:

kubectl delete pod <pod>   # if owned by a Deployment; the replacement pulls immediately

For a standalone pod, patching the spec (or just waiting) works too. Now, the causes.

1. The image name is wrong

Boring, and the most common. A typo in the repository name, a copy-paste that lost a character, the wrong registry prefix (myregistry.io/app vs myregistry.io/team/app).

Event says: repository does not exist or may require 'docker login', or pull access denied ... repository does not exist.

Path: read the image string back to yourself, slowly.

kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}'

Then confirm it exists from any machine with registry access:

docker pull <exact-image-string>
# or, without a docker daemon:
crane manifest <exact-image-string>

Note the ambiguity: registries deliberately return the same error for "doesn't exist" and "you're not allowed to see it", so this event string also appears for cause #3. Check the typo first - it's free.

2. The tag doesn't exist

The repository is fine; the tag isn't. CI failed before pushing, someone deployed v1.4.2 when the pipeline produced v1.4.2-rc1, or the manifest says latest and the repository has never had a latest.

Event says: manifest for <image>:<tag> not found or manifest unknown.

Path: list what the registry actually has:

crane ls myregistry.io/team/app | tail -20

If the tag should exist, go look at the CI run that was supposed to push it - the fix is upstream, not in the cluster. And if you're deploying by mutable tags like latest, this is your periodic reminder that digests (image@sha256:...) don't have this failure mode.

3. Registry auth is missing or wrong

Private registry, and the kubelet has no credentials - or expired ones. This is the most common cause that isn't a typo.

Event says: unauthorized: authentication required, pull access denied, or a 401/403.

Path: check that an imagePullSecret is attached and actually decodes to the right registry:

kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secret <name> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Three things to verify in that JSON: the registry hostname matches the image's hostname exactly (registry.example.com ≠ registry.example.com:5000), the credentials aren't expired (cloud registry tokens often live 12 hours - ECR is the classic offender), and the secret lives in the same namespace as the pod. Secrets don't cross namespaces; every namespace needs its own copy or a service account with the secret attached:

kubectl get sa default -o jsonpath='{.imagePullSecrets}'

If the pod spec has no imagePullSecrets at all and the image is private, that's the whole bug.

4. You hit a rate limit

Docker Hub limits anonymous pulls per source IP. A cluster of nodes NAT'd behind one egress IP burns through that fast, and suddenly images that pulled fine all week start failing.

Event says: 429 Too Many Requests or toomanyrequests: You have reached your pull rate limit.

Path: the message is unambiguous, so there's nothing to diagnose - only to decide. Short-term: authenticate the pulls (authenticated accounts get higher limits) by adding an imagePullSecret with a Docker Hub token. Long-term: run a pull-through cache or mirror the images you depend on into your own registry, so a third party's rate limiter is no longer in your deploy path.

Also check imagePullPolicy. Always on a high-churn workload multiplies pulls for no benefit if you deploy by digest or immutable tag; IfNotPresent lets the node cache work.

5. The node can't reach the registry at all

DNS failure, egress firewall, proxy misconfiguration, a self-hosted registry that's down, or a TLS certificate the node doesn't trust.

Event says: dial tcp: lookup registry.example.com: no such host, i/o timeout, connection refused, or x509: certificate signed by unknown authority.

Path: the pull happens on the node, by the container runtime - not in a pod. So test from the node's perspective:

kubectl debug node/<node> -it --image=busybox
# then, inside:
nslookup registry.example.com
wget -qO- https://registry.example.com/v2/

no such host → node DNS config (check /etc/resolv.conf on the node, not CoreDNS - the kubelet doesn't use cluster DNS). i/o timeout → firewall or proxy; check whether the runtime is configured with an HTTP proxy that excludes your registry. x509 → the registry's CA isn't in the node's trust store; fix the node image, don't mark the registry insecure.

If only some nodes fail, compare a healthy node's config against a failing one - this is where you find the one node pool that missed the CA-certificate update.

One check before you leave

Whatever the cause was, verify the actual state after the fix - not the absence of the old error:

kubectl get pod <pod> -w

Watch it go Pending → ContainerCreating → Running. If it advances past the pull and then starts crashing, that's a different failure with a different guide - see debugging CrashLoopBackOff, and for the map of which status means which problem, the pod status taxonomy. Pull failures are also a staple of live troubleshooting interviews precisely because the event text hands you the answer - if that's on your horizon, here's how to prepare for a Kubernetes troubleshooting interview.

FAQ

How do I force Kubernetes to retry an image pull immediately? Delete the pod (if a controller owns it) or recreate it. The backoff timer belongs to the pod; a fresh pod pulls immediately. There is no kubectl retry-pull - deleting the pod is the idiomatic version of it.

Why does the image pull fine with docker pull on my laptop but fail in the cluster? Different network, different credentials. Your laptop has your docker login session and your DNS; the node has neither. Test from the node (or a debug pod on it) with the exact image string from the pod spec, and check imagePullSecrets in the pod's namespace.

What's the difference between ErrImagePull and ImagePullBackOff? Same failure, different stage. ErrImagePull is the immediate error from a pull attempt; ImagePullBackOff means repeated failures have pushed Kubernetes into exponential backoff between retries (capped at five minutes). Diagnose them identically: read the events.

Can a pull succeed and the pod still fail? Yes - a successful pull of the wrong image. Mutable tags like latest can move under you, and an arm64 image on an amd64 node pulls fine and then dies with exec format error. When behavior doesn't match expectations, compare digests, not tags.