The Linux init system.

I decided that I'll dig down into init systems in Linux and learn more about them. I'm running Ubuntu 16.04, so this might look different on other distributions.

The init system in Linux is mainly responsible for starting essential service processes, mounting file systems and possibly other tasks. The main init systems are systemd, System V init and Upstart. Ubuntu uses systemd.

The init system starts after the Kernel starts its first user space process - init. Indeed, let's see what's running with PID 1:

$ ps 1
    1 ?        Ss     0:04 /sbin/init splash

It's /sbin/init. Let's see what this file is:

$ ls -l /sbin/init
lrwxrwxrwx 1 root root 20 Sep 28 18:40 /sbin/init -> /lib/systemd/systemd

From this output, we can figure out that Ubuntu is using systemd. systemd is a fairly new project (initial release was 6 years ago), but it looks like its widely adopted now. systemd would take care of running various services like your ssh server, your web server and various other ones which are more "under the hood" oriented.

systemd organizes itself with unit files which contain the description of various units and their dependencies. The units are organized in configuration files, which live in various directories. The main directories are:

  • System unit directory: /usr/lib/systemd/. Your distribution maintains this, so don't edit it.
  • System configuration directory: /etc/systemd. Make your local changes here.

Well, these are not all directories that contain unit files. Here's the full set of paths that systemd uses:

$ systemctl -p UnitPath show
UnitPath=/etc/systemd/system /run/systemd/system /run/systemd/generator /usr/local/lib/systemd/system /lib/systemd/system /usr/lib/systemd/system /run/systemd/generator.late

I won't go into details about what the unit files contain, but instead look at two services that I was curious about - ssh and apache. Who runs them? When are they run? How can I verify that they are running?

Let's start with ssh. The main command to interface with systemd is systemctl. We can use it to list all services that are running, by calling systemctl list-units. Let's look for ssh in here:

$ systemctl list-units | grep ssh
ssh.service                                                                              loaded active running   OpenBSD Secure Shell server

Indeed, we have ssh running. Now, let's look at its status and its config file.

$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-02-09 19:27:57 PST; 4 days ago
 Main PID: 786 (sshd)
    Tasks: 1
   Memory: 6.4M
      CPU: 199ms
   CGroup: /system.slice/ssh.service
           └─786 /usr/sbin/sshd -D

Feb 12 18:23:50 virtbox sshd[5791]: Accepted password for petko from port 57805 ssh2
Feb 12 18:23:50 virtbox sshd[5791]: pam_unix(sshd:session): session opened for user petko by (uid=0)

So here it is. The ssh service is running as process 786. We can see that this process is listening on port 22:

$ sudo netstat -tulpn | grep 786
tcp        0      0    *               LISTEN      786/sshd        
tcp6       0      0 :::22                   :::*                    LISTEN      786/sshd 

Indeed, it is. systemctl has another useful command that allows us to print the configuration file for a unit. It works like this:

$ systemctl cat ssh
# /lib/systemd/system/ssh.service
Description=OpenBSD Secure Shell server auditd.service

ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID


So here you can see where is the configuration file located.

Alright, enough ssh. Let's move on to apache. First, a little history though. Before systemd, apparently the main init system in Linux was System V. System V is different than systemd, because it executes services in sequential order, while systemd can be parallel. System V also can't start services on "as-needed" basis. So I guess that's why systemd was implemented. systemd has its config files in /etc/init.d. That's where Apache installs its command files as well - it doesn't create systemd unit files. However, systemd knows how to execute the System V init files. I won't go into details of how System V init works, but basically it executes commands on different runlevels and at each runlevel they are executed in sequential order.

Let's see how apache looks like in systemd:

$ systemctl list-units | grep apache
apache2.service                                                                          loaded active running   LSB: Apache2 web server

It's running. Now let's get its status:

$ systemctl status apache2
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
   Active: active (running) since Mon 2017-02-13 12:49:55 PST; 8h ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 55
   Memory: 6.5M
      CPU: 17.042s
   CGroup: /system.slice/apache2.service
           ├─9097 /usr/sbin/apache2 -k start
           ├─9100 /usr/sbin/apache2 -k start
           └─9101 /usr/sbin/apache2 -k start

Feb 13 12:49:54 virtbox systemd[1]: Starting LSB: Apache2 web server...
Feb 13 12:49:54 virtbox apache2[9071]:  * Starting Apache httpd web server apache2
Feb 13 12:49:54 virtbox apache2[9071]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using Set the 'ServerName' directive globall
Feb 13 12:49:55 virtbox apache2[9071]:  *
Feb 13 12:49:55 virtbox systemd[1]: Started LSB: Apache2 web server.

Look at something interesting here. The file responsible for starting apache is listed as /etc/init.d/apache2. That's the file indeed. It's adapted into systemd by using the systemd-sysv-generator. So that is systemd running. We can run a cat to see that:

$ systemctl cat apache2
# /run/systemd/generator.late/apache2.service
# Automatically generated by systemd-sysv-generator

Description=LSB: Apache2 web server

ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
ExecReload=/etc/init.d/apache2 reload

# /lib/systemd/system/apache2.service.d/apache2-systemd.conf

This is the file that systemd created in order to integrate the System V init command into its system.

What are some other interesting systemctl commands? Let's list them:

# Start / stop /restart a service.
$ sudo systemctl restart apache2

# List all services:
$ systemctl list-units --type=service

# List dependencies:
$ systemctl list-dependencies sshd.service

# See low level properties of a unit:
$ systemctl show sshd.service

And one last cool command:

$ systemd-analyze
Startup finished in 3.989s (kernel) + 7.673s (userspace) = 11.663s

This command prints the time it took to startup our system.

So that's all for today. There's definitely more to explore in systemd land - the syntax of unit files, how systemd executes them and so on. I'll leave that for some other time.