Advanced debugging in Kubernetes
--
Introduction
Pods are the fundamental building block of Kubernetes applications. They are the smallest, most basic deployable resource and can represent as little as a single instance of a running process. Pods are made of one or more containers sharing specific Linux namespaces ( netns, utsns and ipcns). That is why containers in a pod can share the network interface, IP address, network ports and hostname and communicate over localhost or the 127.0.0.1 IP address. On the other hand, containers inside the pod do not share the filesystem ( mntns), nor can they see each other processes ( pidns) by default. Although this is as designed, it might complicate things when you need to troubleshoot when things go wrong. You can visualize this behavior by:
# Quickly launch a pod
kubectl run --image nginx demowww
# identify the node where the pod is running
kubectl get po -o wide
# On the scheduled node, let's find the process id of an nginx process
ps aux | grep -i nginx
root 6229 0.0 0.0 9092 6548 ? Ss 16:07 0:00 nginx: master process nginx -g daemon off;
# Based on the process id, find all assigned namespaces
sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | grep 6229
6229 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: master process nginx -g daemon off;
...
# Based on the ex. netns, we can find all processes of the pod and find the (non)-shared namespaces.
sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | grep 4026532927
4759 4026532927 4026532396 4026532397 4026532395 4026532398 /pause
6229 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: master process nginx -g daemon off;
6778 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: worker process
7016 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: worker process
7017 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: worker process
7018 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: worker process
...In this example, we can clearly see that all the nginx processes belong to the same container. The /pause process belongs to a seperate container, but shares the the netns, utsns and ipcns with the nginx container and form the pod.
Let's cleanup and move on to more advanced debugging strategies.
kubectl delete po demowww
