Learning and debugging namespaces in #docker and #kubernetes by example.
I’ve always been intrigued by how kubernetes pods relate to containers.
As you probably know, containers running on the same host, share the Linux kernel. That’s why a container image does not contain a kernel, only software and tools that make up a distro like for example a package manager.
So, if containers share the same kernel, how are they ’isolated’? Technically, containers are processes that can use limited resources (controlled by cgroups) and isolated by Linux namespaces.
Namespaces are a feature of the kernel that partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources. Examples of such resources are process IDs, hostnames, user IDs, file names, and some names associated with network access, and interprocesses communication. (cfr. https://en.wikipedia.org/wiki/Linux_namespaces)
So, let’s give it a try.
NEW: Free hands-on labs are now available https://cloudyuga.guru/hands_on_lab/namespaces.
Create a container and find the pid
docker run -d --name demo-nginx nginxWe can find the process ID using different ways
# PID=$(docker inspect -f '{{.State.Pid}}' demo-nginx); echo $PID
195284or
# ps aux | grep -i nginxFind the Linux namespaces
There are several ways to find the namespace id’s, but the most effective way I found is:
# sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep $PID
195284 4026532300 4026532296 4026532297 4026532295 4026532298 nginx: master process nginx -g daemon off;The first number is the pid, folowed by the namespace id. So we could filter easily on the netns namespace.
# sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep 4026532300
195284 4026532300 4026532296 4026532297 4026532295 4026532298 nginx: master process nginx -g daemon off;
195335 4026532300 4026532296 4026532297 4026532295 4026532298 nginx: worker processSo, we found other processes spawned inside the same container.
You can easily add containers to an existing namespace, ex. netns
# docker run -it --net=container:demo-nginx xxradar/hackon
root@584db8ec8188:/curl 127.0.0.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...From another terminal, we can verify the namespace sharing for netns but not for other namespaces.
# sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | grep 4026532300
195284 4026532300 4026532296 4026532297 4026532295 4026532298 nginx: master process nginx -g daemon off;
195335 4026532300 4026532296 4026532297 4026532295 4026532298 nginx: worker process
206012 4026532300 4026532356 4026532357 4026532355 4026532358 /bin/bashWhat about kubernetes?
Let’s create a pod.
$ kubectl run --image nginx www-demo
pod/www-demo created$ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
www-demo 1/1 Running 0 44s 10.0.1.207 ip-10-1-2-40 <none> <none>
On the correct node, we can find the nginx process
$ ps aux | grep -i nginx
root 6229 0.0 0.0 9092 6548 ? Ss 16:07 0:00 nginx: master process nginx -g daemon off;$ sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep 6229
6229 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: master process nginx -g daemon off;$ sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep 4026532927
4759 4026532927 4026532396 4026532397 4026532395 4026532398 /pause
6229 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: master process nginx -g daemon off;
6778 4026532927 4026532396 4026532397 4026532456 4026532457 nginx: worker process
We now found all the processes that share the netns, but what we see is even more interesting. We found the /pause container. It’s a container which holds the network netns, utsnsand ipcns namespace for the pod. Kubernetes creates the ‘pause’ containers to acquire for example the respective pod’s IP address and share it with all other containers that join that pod.
Could we modify the content of the www-demo ?
$ sudo nsenter --target 6229 --mount --uts --ipc --net --pid
root@www-demo-8c4d7cc75-vfh9d:/# echo /
"Hacking" >/usr/share/nginx/html/hacking.html$ exit$ kubectl run -it --rm --image xxradar/hackon debug
root@debug:/# curl 10.0.1.207/hacking.html
Hacking
...
Conclusion
I hope this small tutorial helps in better understanding how containers and pods relate to Linux namespaces and sheds a light on how isolation works in containerized environments.
