How to debug a Kubernetes pod in CrashloopBackOff

Stranded at sea with a failing container?

As you use Kubernetes you will eventually find yourself in the situation where your pod is in CrashLoopBackoff and you scratching your head on what is going on.

All you want to do is get a shell to the failed container, so you can troubleshoot why it is failing, but there is no container to connect to!

No sweat! The first thing to do is change your pod command spec and set it into an infinite loop.

As specified in the Kubernetes Run a command in a Shell documentation, you can override the command that docker uses to run your container.

Edit the workload that defines your pod, for example a deployment or replica set:

$ kubectl edit deploy mydeployment -n mynamespace

Find where your pod containers are defined, for example if you have a startup script named bootstrap.sh:

spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        command: bootstrap.sh

If your spec does not define the command, you can also add one, it will override the default defined in the image.

Now replace that with a command that sets the container in an infinite loop:

    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo hello; sleep 10;done"]

Note that args is an array, so when you save and reload, it wil be expanded to something like:

spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        command:
        - /bin/sh
        args:
        - -c
        - while true; do echo hello; sleep 60;done

You can also add environment variables:

spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        command:
        - /bin/sh
        args:
        - -c
        - while true; do echo hello; sleep 60;done
        env:
        - name: MYAPP_DEBUG
          value: "true"

Now save the resource definition and your pod should go to running state.

Once it’s running, you can now Get a shell to the running container:

$ kubectl exec -it mydeployment-65gvm -n mynamespace -- sh

and now you have a shell in the running container:

/ # echo $MYAPP_DEBUG
true
/ #

all done!

Until the next one.

Ciao!