How to Debug a Kubernetes Pod that Crash at Startup (works on Microsoft Azure Kubernetes)?
Posted on: September 7, 2022
I had a pod that worked well on my machine but was not once deployed on Azure Kubernetes. It was constantly rebooting with the status
CrashLoopBackOff. It was impossible to use
kubectl exec -it on the ill pod: it was always closing and restarting.
However, there is a trick that is not perfect but might help you. The idea is to create another pod and have a
command telling the pod to sleep.
1# Get all the pods: find the one that is crashing2kubectl get pods -n realtimepixel-prod34# Take the name of the crashing pod and find the image5kubectl describe pod backend-deployment-576495d6f8-nk7rw -n realtimepixel-prod67# Create a new pod from that image and make it sleep8kubectl run debug-demo -n realtimepixel-prod --image=realtimepixel.azurecr.io/realtimepixel_backend:8ee1dadc954c6217270aaea47fbb8118dd3f1170 --restart=Never --command sleep infinity910# Debug11kubectl exec -it debug-demo -n realtimepixel-prod -- sh1213# Once done, delete the debug pod14kubectl delete pod debug-demo -n realtimepixel-prod
However, the pod does not have all the information from the Helm. In my case, it meant that environment variables were missing. But, you can pass them manually.
1kubectl run debug-demo -n realtimepixel-prod \2--image=realtimepixel.azurecr.io/realtimepixel_backend:8ee1dadc954c6217270aaea47fbb8118dd3f1170 \3--restart=Never \4--command sleep infinity \5--env="IP_BACKEND=backend-service" \6--env="OUTER_PORT_BACKEND=80" \7--env="IP_REDIS=redis-service" \8--env="OUTER_PORT_REDIS=6379" \9--env="IP_FRONTEND=frontend-service" \10--env="OUTER_PORT_FRONTEND=80"
I hope it helps you debug your failing pod that keeps crashing at startup. I tested with Microsoft Azure Pod, and it worked flawlessly. I believe it would work on any provider, but always better to clarify under which environment I have tested the debug trick.