Published: Wed 01 March 2017
in Linux
Examining a process in Linux.
I've been thinking about writing a blog post about Linux tools and commands related to processes. Let's take a look at some of them.
The process that we'll be looking at is a webserver that I wrote some time ago to practice my C and write some code that does network related work. This webserver runs a threadpool where N threads are waiting for server requests that they're going to execute.
So let's start the server:
$ ./server
Running on port: 8000
Great. So which process is this server running as? We can use the pidof command to find that out. Its output looks like this:
$ pidof server
8876
If we had other processes which were running an executable with that name, we'd see more process ids, but since we only have one, we see one process id.
What next? Let's see how the process is layed out in memory. To do this, we can use the pmap command. Its output looks like this:
$ sudo pmap -p 8876
8876: ./server
0000000000400000 16K r-x-- /home/petko/work/github/webserver/server
0000000000603000 4K r---- /home/petko/work/github/webserver/server
0000000000604000 4K rw--- /home/petko/work/github/webserver/server
000000000110f000 132K rw--- [ anon ]
00007fd5ca731000 4K ----- [ anon ]
00007fd5ca732000 8192K rw--- [ anon ]
00007fd5caf32000 4K ----- [ anon ]
00007fd5caf33000 8192K rw--- [ anon ]
00007fd5cb733000 4K ----- [ anon ]
00007fd5cb734000 8192K rw--- [ anon ]
00007fd5cbf34000 4K ----- [ anon ]
00007fd5cbf35000 8192K rw--- [ anon ]
00007fd5cc735000 1792K r-x-- /lib/x86_64-linux-gnu/libc-2.23.so
00007fd5cc8f5000 2044K ----- /lib/x86_64-linux-gnu/libc-2.23.so
00007fd5ccaf4000 16K r---- /lib/x86_64-linux-gnu/libc-2.23.so
00007fd5ccaf8000 8K rw--- /lib/x86_64-linux-gnu/libc-2.23.so
00007fd5ccafa000 16K rw--- [ anon ]
00007fd5ccafe000 96K r-x-- /lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccb16000 2044K ----- /lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccd15000 4K r---- /lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccd16000 4K rw--- /lib/x86_64-linux-gnu/libpthread-2.23.so
00007fd5ccd17000 16K rw--- [ anon ]
00007fd5ccd1b000 152K r-x-- /lib/x86_64-linux-gnu/ld-2.23.so
00007fd5ccf22000 12K rw--- [ anon ]
00007fd5ccf3e000 8K rw--- [ anon ]
00007fd5ccf40000 4K r---- /lib/x86_64-linux-gnu/ld-2.23.so
00007fd5ccf41000 4K rw--- /lib/x86_64-linux-gnu/ld-2.23.so
00007fd5ccf42000 4K rw--- [ anon ]
00007ffca0861000 132K rw--- [ stack ]
00007ffca09eb000 8K r---- [ anon ]
00007ffca09ed000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 39316K
What you see here are virtual memory addresses. For example, let's take a look at this line:
00007fd5cc735000 1792K r-x-- /lib/x86_64-linux-gnu/libc-2.23.so
This is the code for libc, which is the C standard library. This code is shared between processes that need it. We can see the x flag, which means that this is executable memory. The size if roughly the same as the size of this so file. This library is memory mapped into a region starting at address 00007fd5cc735000, but in physical memory it's only stored in one place. To learn more about memory in Linux, here's a great post going into detail about it.
Another interesting command is lsof. lsof stands for "list of open files". Let's see its output:
$ sudo lsof -p 8876
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
server 8876 petko cwd DIR 8,1 4096 262299 /home/petko/work/github/webserver
server 8876 petko rtd DIR 8,1 4096 2 /
server 8876 petko txt REG 8,1 25536 306491 /home/petko/work/github/webserver/server
server 8876 petko mem REG 8,1 1864888 1184834 /lib/x86_64-linux-gnu/libc-2.23.so
server 8876 petko mem REG 8,1 138744 1184980 /lib/x86_64-linux-gnu/libpthread-2.23.so
server 8876 petko mem REG 8,1 162632 1184806 /lib/x86_64-linux-gnu/ld-2.23.so
server 8876 petko 0u CHR 136,9 0t0 12 /dev/pts/9
server 8876 petko 1u CHR 136,9 0t0 12 /dev/pts/9
server 8876 petko 2u CHR 136,9 0t0 12 /dev/pts/9
server 8876 petko 3u IPv4 81993 0t0 TCP *:8000 (LISTEN)
As you can see, we have file descriptors 0,1 and 2, which are stdin, stdout and stderr. They are linked to the terminal in which the process is running in. You can write to that terminal btw. Just type echo "hello world" > /dev/pts/9 and you'll see that text in the terminal where your webserver is running. File descriptor number 3 is our socket which accepts connections.
Another interesting way to inspect processes is the ps command. Its basic output looks like this:
$ ps --pid 8876
PID TTY TIME CMD
8876 pts/9 00:00:00 server
This is simple. We can also show the threads inside a process, like this:
$ ps m --pid 8876 -o pid,tid,cmd
PID TID CMD
8876 - ./server
- 8876 -
- 8877 -
- 8878 -
- 8879 -
- 8880 -
We have five threads here. One is our main thread and the other four are the threadpool threads. The m option tells ps to show the threads of a process. The -o option specifies fields to output. We can even get fancy and output the addresses of the threads' stack pointers and instruction pointers, like this:
$ ps m --pid 8876 -o pid,tid,cmd,esp,eip
PID TID CMD ESP EIP
8876 - ./server - -
- 8876 - a0880b70 ccb0e7ad
- 8877 - cc733ec0 ccb0b3a0
- 8878 - cbf32ec0 ccb0b3a0
- 8879 - cb731ec0 ccb0b3a0
- 8880 - caf30ec0 ccb0b3a0
So all the threads are at the same instruction, but they have different stack pointers, which makes sense. If I execute something on one of the threads, both the ESP and EIP can possibly change.
A lot of data about processes lives in the proc filesytem, located in /proc. For each running process, there's a subdirectory of /proc named after the process id. For example, for our process 8876, there's a status file which lists various information about the process. Let's look at it:
$ cat /proc/8876/status
Name: server
State: S (sleeping)
Tgid: 8876
Ngid: 0
Pid: 8876
PPid: 2604
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 256
Groups: 4 24 27 30 46 113 128 999 1000
NStgid: 8876
NSpid: 8876
NSpgid: 8876
NSsid: 2604
VmPeak: 39316 kB
VmSize: 39316 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 800 kB
VmRSS: 800 kB
VmData: 32988 kB
VmStk: 136 kB
VmExe: 16 kB
VmLib: 2040 kB
VmPTE: 48 kB
VmPMD: 12 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
Threads: 5
SigQ: 0/7848
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp: 0
Cpus_allowed: 1
Cpus_allowed_list: 0
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 3
nonvoluntary_ctxt_switches: 2
There's a lot of data in here, but remember how we used ps to count the number of threads in this process. That's also available here on the line saying Threads: 5.
Our last command is pidstat. pidstat shows statistics about a running process, which can be updated at a regular time interval. A possible invocation can be:
$ pidstat -p 8876 1
Linux 4.4.0-64-generic (virtbox) 03/01/2017 _x86_64_ (1 CPU)
12:22:00 PM UID PID %usr %system %guest %CPU CPU Command
12:22:01 PM 1000 8876 0.00 0.00 0.00 0.00 0 server
12:22:02 PM 1000 8876 0.00 0.00 0.00 0.00 0 server
Our server is not doing anything right now, so you see a lot of zeroes.
There are many other interesting commands that you can look to figure out what processes are doing. strace shows system calls run by a process. ltrace shows dynamic library calls. tcpdump can be used to show traffic going in and out of a process.
So, that's all for today. Happy running of processes.