systemd

I use systemd for as a system / service manager for my Amazon Linux operating system on the EC2 instance server that runs this site. I don't know much about systemd, so I am creating this note to help me manage the server more effectively.

Date Created:

Last Edited:

References

Bash code snippets that start with $ denote that the command is being run in a Linux-based OS.

Definitions

parallelization
- Parallelization is a technique that divides a large computational task into smaller sub-tasks that can be executed simultaneously on multiple processors or cores. The goal of parallelization is to reduce the overall computation time.
daemon
- In multitasking computing operating systems, a daemon is a computer program that runs as a background process, rather than being under direct control of an interactive user. Traditionally, the process names of a daemon end with the letter d, for clarification that the process is in fact a daemon, and for differentiation between a daemon and a normal computer program.

- In a Unix environment, the parent process of a daemon is often, but not always, the init process. A daemon is usually created either by forking a child process and then immediately exiting, thus causing init to adopt the child process, or by the init process directly launching the daemon.
- Systems often start daemons at boot time that will respond to network requests, hardware activity, or other programs by performing some task. Daemons such as cron may also perform defined tasks at scheduled times.
- Examples:
- - syslogd
  - - syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message i labeled with a facility code, indicating the type of system generating the message, and is assigned a severity level. Computer system designed may use syslog for system management and security auditing as well as general informational, analysis and debugging messages. When operating over a network, syslog uses a client-server architecture where a syslog server listens for and logs messages coming from clients.
    - - Message Logging - keeping a log of events that occur in a computer system, such as problems, errors, or just information on current operations. The events may occur is the OS or other software. A message of log entry is recorded for each such event. Logging is important in multi-user software - computer software that allows access by multiple users of a computer. - to have a central overview of the operation of the system.
  - sshd - a daemon that serves incoming SSH connections
Linux Control Groups
- cgroups (abbreviated from control groups) is a Linux Kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, etc.) of a collection of processes.
sysv
- Unix System V is one of the first commercial versions of the Unix operating system.
LSB script
- LSB stands for Linux Standard Base. LSB was started by the Linux foundation to reduce the difference between several Linux distributions, and thereby reducing the cost involved in porting between different distributions. Init scripts are one among them to be standardized.
- Init scripts are used to start|stop a software|service.
- LSB compliant scripts need to:
- - Provide at-least 'start, stop, restart, force-reload, and status'
  - Return Proper exit code
  - Document run-time dependencies
sysinint
- sysinit is a mechanism for scheduling the execution of initialization and teardown routines. this is similar to init and fini routines with the addition of explicit ordering metadata. It allows runtime ordering of subsystem initialization in the kernel as well as kernel modules (KLDs).
date locale
- A locale is a collection of settings that define a user's environment, including language, region, and script. For a program, a locale's date and time formatting conventions are country dependent specifications that allow the program to use the appropriate conventions for different regions.
init
- INIT is a software program used to initialize processes during a computer's boot up process. It is typically used in Unix-based operating systems, and is responsible for starting up system services and managing related tasks. In Unix based operating systems, INIT is typically the first process to run during the boot-up sequence. It is responsible for starting up other system processes and services and is designed to ensure that all necessary system components are initialized and ready to use.
- Systemd is a newer system initialization process that is designed to replace the traditional init process in many UNIX based systems. Unlike init, which uses a series of scripts to start-up services, systemd uses a binary format and a set of configuration files to manage system processes and services.
forking
- A fork is an operation whereby a process creates a copy of itself. It is an interface that is required for compliance for Single UNIX Specification standards. It is usually implemented a s C standard library wrapper to the fork, clone, or other system calls of the kernel.
- In multitasking operating systems, processes (running programs) need a way to create new processes, e.g. to run other programs. Fork and its variants are typically the only way of doing so in Unix-like systems.
- Fort a process to start the execution of a different program, it first forks to create a copy of itself. Then, the copy called the child process calls the exec system call to overlay itself with the other program. it ceases execution of the former program in favor of the other.
- When a process calls fork, it is deemed the parent process and the newly create process is its child. After the fork, both processes not only run the same program, but they resume as though both had the system call. They can then inspect the call's return value to determine their status, child or parent, and act accordingly.
cron
- The cron command-line utility is a job scheduler on Unix-like operating systems. users who set up and maintain software environments use cron to schedule jobs, also known as cron jobs, to run periodically at fixed times, dates, or intervals. It typically automated system maintenance or administration - though its general purpose nature makes it useful for things like downloading files from the internet ad downloading email at regular intervals.
kernel
- The kernel is a computer program at the core of a computer's operating system and generally has complete control over everything in the system. The kernel is also responsible for preventing and migrating conflicts between different processes. It is the portion of the operating system code that is always resident in memory and facilitates the interactions between hardware and software components.
- A full kernel controls all hardware resources (e.g. I/O, memory, cryptography) via device drivers, arbitrates conflicts between processes concerning such resources, and optimizes the utilization of common resources e.g. CPU and cache usage, file systems, and network sockets.

- On most systems, the kernel is one of the first programs loaded on startup (after the bootloader). It handles the rest of the startup as well as memory, peripherals, and input/output requests from software, translating them into data-processing instructions for the central processing unit.

What Is systemd

systemd is a suite of basic building blocks for a Linux system. It provides a system and service managers that runs as PID 1 and starts the rest of the system.

systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting of daemons, keeps tract of processes using Linux control groups, maintains mount and automount points, and implements an elaborate transactional dependency-based service control logic. systemd supports SysV and LSB scripts and works as a replacement for sysinit.

Other parts including logging daemon, utilities to control basic system configuration like the hostname, date, locale, maintain a list of logged in users and running containers and virtual machines, system accounts, runtime directories and settings, and daemons to manage simple network configuration, network time synchronization, log forwarding, and name resolution.

Systemd Essentials

Basic Unit Management

The basic object that systemd manages and acts upon is a "unit". Units can be of many types, but the most common type is a "service" (indicated by a unit file ending in .service. To manage services on a systemd enabled server, our main tool is the systemctl command.

$ sudo systemctl start nginx.service # start the service
$ sudo systemctl stop nginx.service # stop the service
$ sudo systemctl restart nginx.service # restart the service
$ sudo systemctl reload nginx.service # reload the service

Enabling or Disabling Units

By default, most systemd unit files are not started automatically at boot. To configure this functionality, you need to "enable" to unit. This hooks it up to a certain boot "target", causing ut to be triggered when trhat target is started.

$ sudo systemctl enable nginx.service # Enable a service
$ sudo systemctl disable ngnix.service # Disable a service

Getting an Overview of the System State

# List all unit files that systemd has listed as active
$ systemctl list-units
# List all of the units that systemd has loaded or attempted to load into memory
$ systemctl list-units --all
# List all of the units installed on the system, including those that systemd has not tries to loas into memory
$ systemctl list-unit-files

Viewing Basic Log Information

A systemd component called journald collects and manages journal entries from all parts of the system. This is basically log information from applications and the kernel. Some distributions configure journald to save previous boot records, but other you need to this yourself by either:

Editing the /etc/systemd/journald.conf file and set the Storage= option to "persistent"
Or create the persistent directory by typing sudo mkdir -p /var/log/journal.

# To see all log entries, starting at oldest
$ journalctl
# See only jounral entries from the current boot
$ journalctl -b
# See only kernel messages, such as those that are typicallu represented by dmesg, use the -k flag
$ jounralctl -k
# See only journal entries from the current boot'
$ journalctl -b -k

Querying Unit States and Logs

You can also get information about a specific service:

# Shows atcive status of the unit, info about the process, and the latest journal entries
$ systemctl status nginx.service
# Get the journal entries for a certain service
$ journalctl -u nginx.service
# Get the journal entries for a certian service for the current boot
$ journalctl -b -u nginx.service

Inspecting Units and Unit Files

A unit file contains parameters that systemd uses to manage and run a unit.

# To see thf ull contents of a unit file, use 
$ systemctl cat nginx.service
# To see the dependency tree of a unit (which units systemd will attempt to activate when starting the unit), type
$ systemctl list-dependencies nginx.service
# To expand all  depdent units recursively, pass the --all flag
$ systemctl list-dependencies --all nginx.service
# Show the low-level details of the unit's settings on the system
$ systemctl show nginx.service

Modifying Unit Files

# You can use the systemctl command itself to make edits to the unit file so that you don't have to navigate to disk location:
$ sudo systemctl edit nginx.service
# modify the entire content of the unit file insitead of creating a snippet
$ sudo systemctl edit --full nginx.service
# After editig a unit file, you should reload the systemd process to pick up your changes
$ sudo systemctl daemon-reload

Using Targets

Another function of an init system is to transition the server itself between different states. Traditional init systems typically refer to these as "runlevels", allowing the system to only be in one runlevel at any one time.

In systemd "targets" are used instead. Targets are basically synchronization points that the server can use to bring the server into a specific state. Service and other unit files can be tied to a target and multiple targets can be active at the same time.

# See all of the targets available on the system
$ systemctl list-unit-files --type=target
# View the default target that systemd tries to reach at boot (which starts all of the init files that make up the dependency tree of that target)
$ systemctl get-default
# You can change the default target that will be used by using the set-default-option
$ sudo systemctl set-default multi-user.target
# See what units are ties to a target
$ systemctl list-dependencies multi-user.target
# Modify the system state to transition between targets with the isolate option. This will stop any units that are not ties to the specific target
$ sudo systemctl isolate multi-user.target

Stopping or Rebooting the Server

# turn off your server
$ sudo systemctl poweroff
# Reboot the system instead
$ sudo systemctl reboot
# Boot into resuce mode
$ sudo systemctl rescue

Understanding Systemd Units and Unit Files

Units are the objects that systemd knows how to manage. These are basically a standardized representation of system resources that can be managed by the suite of daemons and manipulated by the provided utilities.

What do Systemd Units Give You?

features that units are able to easily implement:

socket-based activation: Sockets associated with a service are best broken out of the daemon itself in order to be handled separately. This provides a number of advantages. This also allows the system to create all sockets early in the boot process, making it possible to boot the associated services in parallel.
bus-based activation: Units can also be activated on the bus interface provided by D-Bus. A unit can be started when an associated bus is published.
path-based activation: A unit can be started based on activity on or the availability of certain filesystem paths. This utilized inotify
implicit dependency mapping: Most of the dependency tree for units can be built by systemd itself. you can still add dependency and ordering information, but most of the heavy lifting is taken care of for you.
instances and templates: Template unit files can be used to create multiple instances of the same general unit. This allows for slight variations or sibling units that all provide the same general function.
easy security hardening: Units can implement some fairly good security features by adding simple directives. For example, you can specify no or read-only access to part of the filesystem, limit kernel capabilities, and assign private /tmp and network access.
drop-ins and snippets: Units can easily be extended by providing snippets that will override parts of the system unit file. This makes it easy to switch between vanilla and customized unit implementations.

Where are Systemd Unit Files Found

The files that define how systemd will handle a unit can be found in many different locations
/lib/systemd/system
- The system's copy of unit files are generally kept in the /lib/systemd/system directory. When software installs unit files on the system, this is where they are placed by default.
- Unit files stored in /lib/systemd/system directory are able to be started and stopped on demand during a session. You should not edit files in this directory, instead you should override the file.
/etc/systemd/system
- If you want to modify the way a unit functions, the best location to do so is within the /etc/systemd/system directory. Unit files found in this directory location take precedence over any of the other locations in the file system.
- If you wish to overwrite only specific directives from the system's unit file, you can actually provide unit file snippets within a subdirectory. These will append or modify the directives of the system's copy, allowing you to specify only the options you want to change.
- The correct way to do this is to make a directory named after the unit file with .d appended on the end - e.g., myapp.service.d directory, and inside the directory, create a file ending in .conf that can be used to override or extend the attributes of the system's unit file.
/run/systemd/system
- The location for run-time unit definitions. Unit files found in this directory have a priority landing between those in /etc/systemd/system and /lib/systemd/system. They are given less wight than the former definition, but less weight that the latter.
- The systemd process itself uses this location for dynamically created unit files that are created at runtime. This directory can be used to change the system's unit behavior for the duration of the session.

Types of Units

systemd categorizes units according to the type of resource they describe. Best way to determine the type of unit is the type of the suffix, which is appended to resource name:
- .service: A service unit describes how to manage a service or application on the server, e.g.: how to start/stop server, under which circumstances it should be automatically started, and the dependency and ordering information for related software.
- .socket: A socket unit file describes a network or IPC socket, or a FIFO buffer that systemd uses for socket-based activation.
- .device: A unit that describes a device that has been designated as needing systemd uses for socket-based activation. These always have an associated .service file that will be started when an activity is seen on the socket the unit defines.
- .mount: The unit defined a mountpoint on the system to be managed by systemd. These are named after the mount path, with slashed changed to dashes. Entries within /etc/fstab can have units created automatically.
- .automount: AN .automount unit configures a mountpoint that will be automatically mounted. These must be named after the mount point they refer to and must have a matching .mount unit to describe the specifics of the mount.
- .swap: This unit describes swap space on the system. The name of these units must reflect the device or file path of the space.
- .target: A target unit is used to provide synchronization points for other units when booting up or changing states. They also can be used to bring the system to a new state. Other units specify their relation to targets to become ties to the target's operations.
- .path: This unit defined a path that can be used for path based activation. By default, a .service unit of the same base name will be started when the path reaches the specified state. This uses inotify to monitor the path for changes.
- - The inotify API provides a mechanism for monitoring filesystem events.
- .timer: A .timer unit defined a timer that will be managed by systemd, similar to a cron job for delayed or scheduled activation. A matching unit will be started when the timer is reached.
- .snapshot: A .snapshot unit is created automatically by the systemctl snapshot command. It allows you to reconstruct the current state of the system after making changes. Snapshots do not survive across sessions and are used to roll back temporary states.
- .slice: A slice unit is associated with Linux Control Group nodes, allowing resources to be restricted or assigned to any process associated with the slice. The name reflects hierarchal position within the cgroup tree. Units are placed in certain slices by default depending on their type.
- .scope: Scope units are created automatically by the systemd from information from its bus interfaces. These are used to manage sets of system processes that are created externally.

Anatomy of a Unit File

The internal structure of unit files are organized with sections. Sections are denoted by a pair of square brackets "[" and "]" with the section name enclosed within. Each section name extends until all of the subsequent section or until the end of the file.

Section names are well-defined and case-sensitive
If you need to add non-standard section names to be parsed by other applications, other than systemd, add a X- prefix to the section name
Within sections, unit behavior and metadata is defined through the use of simple directives using a key-value format with assignment indicated by equal sign:

[Section]
Directive1=value
Directive2=value
...

In an override file, directives can be reset by assigning them to an empty sing

Directive1=

[Unit] Section Directives

The first unit found in most unit files is the [Unit] section. This is generally used for metadata for the unit and for configuring the relationship of the unit to other units.
Section ordering doesn't matter to systemd
Common directives in the [Unit] section:
- Description=: This directive can be used to describe the name and basic functionality of the unit . It is good to have something short, specific, and informative.
- Documentation=: This directive provides the location for a list f URLs for documentation. These can be either internally available man pages or web accessible URLs.
- Requires=: This directive lists any units upon which this nit essentially depends. The units must successfully activate. The units are started in parallel with the current unit.
- Wants=: Similar to Requires, but less strict. Systemd will not fail if the units are not found or fail to start.
- BindsTo=: Directive similar to Requires but also causes the current unit to stop when the associated unit terminates.
- Before=: The units listed in this directive will not be started until the current unit is marked as started if they are activated at the same time. This does not imply a dependency relationship and must be used in conjunction with one of the above directives if this is desired.
- After=: The units listed in this directive will be started before starting the current unit. This does not imply a dependency relationship and must be used in conjunction with one of the above directives if this is desired.
- Conflicts=: This can be used to list units that cannot be run at the same time as the current unit. Starting a unit with this relationship will cause the other units to be stopped.
- Condition...=: There are a number of directives that start with the Condition, which allow the administrator to test certain conditions prior to starting the unit. This can be used to provide a generic unit file that will only be run when on appropriate systems. If the condition is not met, the unit is gracefully skipped.
- Assert...=: Similar to the directives that start with Condition, these directives check for different aspects of the running environment to decide whether the unit should activate.

[Install] Section Directives

The last section is often the [Install] section. This section is optional and it is used to define the behavior of a unit if it is enables or disabled. Enabling a unit marks it to be automatically started at boot. Only units that can be enabled have this section.
Directives:
- WantedBy=: This directive is the most common way to specify how a unit should be enabled. The directive allows you to specify a dependency relationship similar to Wants= directive in the [Unit] section. When a unit with this directive is enabled, a directory will be created within /wtc/systemd/system named after the specified unit with .wants appended to the end. Within this, a symbolic link to the current unit will be created within /etc/systemd/system and a symbolic link to trhe current unit will be placed within.

- RequiredBy=: This directive is very similar to the WantedBy= directive, but instead specifies a required dependency that will cause the activation to fail if not met. When enabled, a unit with this directive will create a directory ending with .requires

- Alias=: This directive allows the unit to be enabled under another name as well. Among other uses, this allows multiple providers of a function to be available, so that related units can look for any provider of the common aliased name.

- Also=: This directive allows units to be enabled and disabled as a set. Supporting units that should always be available when this unit is active can be listed here. They will be managed as a group for installation tasks.

- DefaultInsiance=: For template units which can produce unit instances with unpredictable names, this can be used as a fallback value for the name if an appropriate name is not provided.

Unit-Specific Section Directives

Sandwiched between the two previous sections, you will likely find unit type-specific sections. Most unit types offer directives that only apply to their specific type. These are available within sections named after their type.
The device, target, snapshot, and scope unit types have no unit-specific directives, and thus have no associated sections for their type.

The [Service] Section

Provides configuration that is only applicable to services.
The Type= directive should be specified. It can be one of the following:
- simple: The main process is specified in the start line. This is the default if the Type= and Busname= directives are not set, but the ExecStart= is set. Any communication should be handled outside the unit through a second unit of the appropriate type (like through a .socket unit if this unit must communicate through sockets)
- forking: This service type is used when the service forks a child process, exiting the parent process almost immediately. This tells systemd that the process is still running event though the parent exited.
- oneshot: Used when the service forks a child process, exiting the parent process almost immediately. This tells systemd that the process is still running even though the parent exited.
- dbus: This type indicates that the process will be short-lived and that systemd should wait for the process to exit before continuing on with other units. This is the default Type= and ExecStart= are not set. It is used for one-off tasks.
- notify: This indicates that unit will take a name on the D-Bus bus. When this happens, systemd will continue to process the next unit.
- idle: This indicates that the service will not be run until all jobs are dispatched.
Additional Directives:
- RemainAfterExit=: This directive is commonly used with the oneshot type. It indicates that the service should eb considered active even after the process exits.
- PIDFile=: If the service type is marked as "forking", this directive is used to set the path of the file that should contain the process ID number of the main child that should be monitored.
- BusName=: This directive should be set to the D-Bus bus name that the service will attempt to acquire when using the "dbus" service type.
- NotifyAccess=: This specifies access to the socket that should be used to listen for notifications when the "notify" service type is selected.
Directives to Manage Services:
- ExecStart=: This specifies the fill path and the arguments of the command to be executed to start the process. This may be specified once. If the path to the command is specified by a dash "-" character, non-zero exit statuses will be accepted without marking the unit activation as failed.
- ExecStartPre=: This can be used to provide additional commands that should be executed before the main process is started. This can be used multiple times. If the path to the command is specified by a dash "-" character, non-zero exit statuses will be accepted without marking the unit activation as failed.
- ExecStartPost=: Same as ExecStartPre= except the command is run after the main process is started.
- ExecReload=: This optional directive indicates the command necessary to reload the configuration of the service if available.
- ExecStop=: This indicates the command needed to stop the service. If this is not given, the process will be killed immediately when the service is stopped.

- ExecStopPost=: This can be used to specify commands to execute following the stop command.
- RestartSec=: If automatically restarting the service is enabled, this specifies the amount of time to wait before attempting to restart the service.
- Restart=: This indicates the circumstances under which systemd will attempt to automatically restart the service.
- TimeoutSec=: This configures the amount of time that systemd will wat when stopping or stopping the service before marking it as failed or forcefully killing it. You can separate timeouts with TimeoutStartSec= and TimeoutStopSec= as well.

The [Socket] Section

Socket units are very common in systemd configurations because many services implement socket-based activation to provide better parallelization and flexibility. Each socket unit must have a matching service unit that will be activated when the socket receives activity.

Common Socket Directives

ListenStream=: Defined the address for a stream socket which supports sequential, reliable communication. Services that use TCP should use this socket type.
ListenDatagram=: This defined an address for a datagram socket which supports fast, unreliable communication packets. Services that use UDP should set this socket type.
ListenSequentialPacket: This defined an address for sequential, reliable communication with max length datagrams that preserves message boundaries. This is found most often for Unix sockets.
ListenFIFO: Along with other listening types, you can also specify a FIFO buffer instead of a socket.

Other Socket Directives

Accept=: This determines whether an additional instance of the service will be started for each connection. Id set to false one instance will handle all connections.

SocketUser=: With a Unix socket, specifies the owner of the socket. This will be the root user if left unset.

SocketGroup=: With a Unix socket, specifies the group owner of the socket. This will be the root group if neither this or the above are set. If only the SocketUser= is set, systemd will try to find a matching group.

SocketMode=: For Unix sockets or FIFO buffers, this sets the permissions on the created entity.

Service=: If the service name does not match the .socket name, the service can be specified with this directive.

The [Mount] Section

Mount units allow for mount point management within systemd. Mount points are named after the directory that they control, with a translation algorithm applied. Mount points are names after the directory they control, with a translation algorithm applied.

The leading slash is removed, all other slashes are translated into dashes "-", and all dashes and unprintable characters are replaced with C-style escape codes. The result of this translation is used as the mount unit name. Mount units will have an implicit dependency on other mounts above it in the hierarchy.

Mount units are often translated directly from the /etc/fstab files during the boot process.

The following directives are useful:

What=: The absolute path to the resource that needs to be mounted.
Where=: The absolute path of the mount point where the resource should be mounted. This should be the same as the unit file name, except using conventional filesystem notation.
Type=: The filesystem type of the mount.
Options=: Any mount options that need to be applied. This is a comma-separated list.
SloppyOptions=: A Boolean that determines whether the mount will fail if there is an unrecognized mount option.
DirectoryMode=: If parent directories need to be created for the mount point, this determines the permission mode of these directories.
TimeoutSec=: Configures the amount of time the system will wait until the mount operation is marked as failed.

The [Automount] Section

The unit allows an associated .mount unit to be automatically mounted at boot. As with the .mount unit, these units must be named after the translated mount point's path.

The [Automount] section is pretty simple, with only the following two options allowed:

Where=: The absolute path of the automount point on the filesystem. This will match the filename except that it uses conventional path notation instead of the translation.
DirectoryMode=: If the automount point or any parent directories need to be created, this will determine the permissions settings of those path components.

The [Swap] Section

Swap units are used to configure swap space on the system. The units must be named after the swap file or the swap device, using the same filesystem translation that was discussed above.

Like the mount options, the swap units can be automatically created from /etc/fstab entries, or can be configured through a dedicated unit file

The [Swap] section of a unit file can contain the following directives for configuration:

What=: The Absolute path to the location of the swap space, whether this is a file or a device
Priority=: This takes an integer that indicates the priority of the swap being configured.
Options=: Any options that are typically set in the /etc/fstab file can be set with this directive instead. A comma-separated list is used.
TimeoutSec=: The amount of time that the systemd waits for the swap to be activated before marking the operation as a failure.

The [Path] Section

A path unit defined a filesystem path that systemd can monitor for changes. Another unit must exist that will be activated when certain activity is detected at the path location. Path activity is determined through inotify events.

The [Path] section of a unit file can contain the following directives:

PathExists=: This directive is used to check whether the path in question exists. If it does, the associated unit is activated.
PathExistsGlob=: This is the same as the above, but supports file glob expressions for determining path existence.
PathChanged=: This watches for changes like the above directive, but it activates on file writes as well as when the watched file is closed.
PathModified=: This watches for changes like the above directive, but it activates on file writes as well as when the file is closed.
DirectoryNotEmpty=: This directive allows systemd to activate the associated unit when the directory is no longer empty.
Unit=: This specifies the unit to activate when the path conditions specified above are met. If this is omitted, systemd will look for a .service file that shares the same base unit name as this unit.
MakeDirectory=: This determines if systemd will create the directory structure of the path in question prior to watching.
DirectoryMode=: If the above is enabled, this will set the permission mode of any path components that must be created.

The [Timer] Section

Timer units are used to schedule tasks to operate at a specific time or after a certain delay. This unit type replaces or supplements some of the functionality of the cron and at daemons. An associated unit must be provided which will be activated when the timer is reached.

The [Timer] section of a unit file can contain some of the following directives:

OnActiveSec=: This directive allows the associated unit to be activated relative to the .timer unit's activation.
OnBootSec=: This directive is used to specify the amount of time after the system is booted when the associated unit should be activated.
OnStartupSec=: This directive is similar to the above timer, but in relation to when the systemd process itself was started.
OnUnitActiveSec=: This sets a timer according to when the associated unit was last activated.
OnUnitInactiveSec=: This sets the timer in relation to when the associated unit was last marked as inactive.
OnCalender=: This allows you to activate the associated by specifying an absolute instead of relative to an event.
AccuracySec=: This unit is used to set the level of accuracy with which the timer should be adhered to. By default, the associated unit will be activated within one minute of the timer being reached.
Unit=: This directive is used to specify the unit should be activated when the timer elapses. If unset, systemd will look for a .service unit with a name that matches this unit.
Persistent=: If this is set, systemd will trigger the associated unit when the timer becomes active if it would have been triggered during the period in which the timer was inactive.
WakeSystem=: Setting this directive allows you to wake a system from suspend if the timer is reached when in that state.

The [Slice] Section

The [Slice] section of a unit file actually does not have any .slice unit specific configuration. Instead, it can contain some resource management directives that are actually available to a number of the units listed above.
The directives for the [Slice] section can be found on the systemd.resource-control man page. The directives are valid in the following unit-specific directives:
- [Slice]
- [Scope]
- [Service]
- [Socket]
- [Mount]
- [Swap]

Creating Instance Units from Template Unit Files

Template unit files are in most ways, no different than regular unit files. However, these provide flexibility in configuring units by allowing certain parts of the file to utilize dynamic information that will be available at runtime.

Template and Instance Unit Names

Template unit files can be identified because they contain an @ symbol after the base unit name and before the unit type suffix:

example@.service

When an instance is created from a template, an instance identifier is places between the @ symbol and the period signifying the start of the unit type. For example, the above template unit file could be used to create an instance unit that looks like this:

example@instance1.service

An instance file is usually created as a symbolic link to the template file, with the link name including the instance identifier. In this way, multiple links with unique identifiers can point back to a single template file.

Template Specifiers

The power of template unit files is mainly seen through its ability to dynamically substitute appropriate information within the unit definition according to the operating environment. This is done by setting the directives in the template file as normal, but replacing certain values or parts of values with variable specifiers.

The following are common specifiers that will be replaced when an instance unit is interpreted with relevant information:

%n: Anywhere where this appears in the template file, the full resulting unit name will be inserted.
%N: This is the same as the above, but any escaping, such as those present in file path patterns, will be reversed.
%p: This references the unit name prefix. this is the portion of the unit name that comes before the @ symbol.
%P: This is the same as the above, but with any escaping reversed.
%i: This references the instance name, which is the identifier following the @ in the instance unit. This is one of the most commonly used identifiers because it will always be dynamic.
%I: This specifier is the same as the above, but with escaping reversed
%f: This will be specified with the unescaped instance name of the prefix name, prepended with a /
%c: The control group of the unit, with /sys/fs/cgroup/systemd/ removed
%u: The name of the user configured to run the unit.
%U: The same as above, but as a numeric UID instead of name.
%H: The host name of the system that is running the unit.
%%: This is used to insert a literal percentage sign.

Manage Systemd Services and Units

Service Management

The fundamental purpose of an init system is to initialize components that must be started after the Linux kernel is booted (traditionally known as "userland" components). The init system is also used to manage services and daemons for the server at any point while the system is running. With that in mind, we will start with some basic service management operations.

In systemd, the target of most actions are "units", which are resources that systemd knows how to manage. Units are categorized by the type of resource they represent and they are defined with files known as unit files. The type of each unit can be inferred from the suffix at the end of the file.
For most service management commands, you can leave off the .service suffix, since systemd is smart enough to know that you probably want to operate on a service management commands.

Starting and Stopping Services

# Start a systemd service, executing instructions in the service's unit files
$ sudo systemctl start application.service
# you can leave off the ".service" for service management commands - though try to use .service for clarity
$ sudo systemctl start application
# Stop a currently running service
$ sudo systemctl stop application.service

Restarting and Reloading

# Restart a service
$ sudo systemcl restart application.service
# If the appllication in question is able to reload its configuration files (without restarting), you can issue the reload command to initiate that process
$ sudo systemctl reload application.service
# If you are unsure whether the service has the functionality to reload its configuration, you can issue the reload-or-restart command. This will reload the configuration in place if availabl, otherwise, it will restart the service
$ sudo systemctl reload-or-restart application.service

Enabling or Disabling Services

# To tell systemd to start services automatically at ot, you must enable them
# This will create a symbolic link from the system's cop of the service file (usually in /lib/systemd/system or /etc/systemd/system) into the location on disk where systemd looks for autostart files
$ sudo systemctl enable application.service
# Disable the service from starting automatically
$ sudo systemctl disable application.service

Checking the Status of Services

# Check the status of a service
# This command provides you with the service state, the cgroup hierarchy, and the first few log lines. Below is an example of an nginx server
$ systemctl status application.service
nginx.service - A high performance web server and a reverse proxy server
	Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
  Active: active (running) since Tue 2015-01-27 19:41:23 EST; 22h ago
 	Main PID: 495 (nginx)
   CGroup: /system.slice/nginx.service
   	├─495 nginx: master process /usr/bin/nginx -g pid /run/nginx.pid; error_log stderr;
    └─496 nginx: worker process
Jan 27 19:41:23 desktop systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 27 19:41:23 desktop systemd[1]: Started A high performance web server and a reverse proxy server.
# Check if the unit is currently active
$ systemctl is-active application.service
# Check is the unit is enabled
$ systemctl is-enabled application.service
# Check if the unit is in a failed state
$ systemctl is-failed application.service

System State Overview

Commands for exploring the current state of the system not just a single unit

# Lsit othe cyrrent units that are active. The bwlow command is the same as just `systemctl`
$ systemctl list-units
UNIT                                      LOAD   ACTIVE SUB     DESCRIPTION
atd.service                               loaded active running ATD daemon
avahi-daemon.service                      loaded active running Avahi mDNS/DNS-SD Stack
dbus.service                              loaded active running D-Bus System Message Bus
dcron.service                             loaded active running Periodic Command Scheduler
dkms.service                              loaded active exited  Dynamic Kernel Modules System
getty@tty1.service                        loaded active running Getty on tty1

The output has the following columns:

UNIT: The systemd unit name
LOAD: Whether the unit's configuration has been parsed by systemd. The configuration of loaded units is kept in memory.
ACTIVE: A summary about whether the unit is active. This is usually a fairly basic way to tell if the unit has started successfully or not.
SUB: This is a lower-level state that indicates more detailed information about the unit. This often varies by unit type, state, and the actual method in which the unit runs.
DESCRIPTION: A short textual description of what the unit is/does

Use the --all flag to list all units, not just active units.

See certain units based on their state by using the --state= flag where state is equal to LOAD, ACTIVE, or SUB

Keep the --all flag when using the --state flag

Use the --type= filter to filter units by type, e.g. --type=service

# See every available unit file on the system
$ systemctl list-unit-files
UNIT FILE                                  STATE
proc-sys-fs-binfmt_misc.automount          static
dev-hugepages.mount                        static
dev-mqueue.mount                           static
proc-fs-nfsd.mount                         static
proc-sys-fs-binfmt_misc.mount              static
sys-fs-fuse-connections.mount              static
sys-kernel-config.mount                    static
sys-kernel-debug.mount                     static
tmp.mount                                  static
var-lib-nfs-rpc_pipefs.mount               static
org.cups.cupsd.path                        enabled
. . .

The state will usually be enabled, disabled, static, or masked. In this context, static means that the unit file does not contain an install section, which is used to enable a unit. As such, these units cannot be enabled. Usually, this means that the unit performs a one-off action or is used only as a dependency of another unit and should not run by itself.

Unit Management

Find more information about units themselves

Displaying a Unit File

$ systemctl cat atd.service
[Unit]
Description=ATD daemon
[Service]
Type=forking
ExecStart=/usr/bin/atd
[Install]
WantedBy=multi-user.target

The output is the unit file as known to be the currently running systemd process.

Displaying Dependencies

See a unit's dependency tree:

$ systemctl list-dependencies sshd.service
sshd.service
├─system.slice
└─basic.target
  ├─microcode.service
  ├─rhel-autorelabel-mark.service
  ├─rhel-autorelabel.service
  ├─rhel-configure.service
  ├─rhel-dmesg.service
  ├─rhel-loadmodules.service
  ├─paths.target
  ├─slices.target
. . .

Use the --all flag to list all dependencies

To show reverse dependencies (units that depend on the specified unit), you can use the --reverse flag to the command.

Checking Unit Properties

You can see the low level properties of a unit using the show command.

$ systemctl show sshd.service
Id=sshd.service
Names=sshd.service
Requires=basic.target
Wants=system.slice
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=syslog.target network.target auditd.service systemd-journald.socket basic.target system.slice
Description=OpenSSH server daemon
. . .

If you want to display a single property, you can pass the -p flag with the property name. Example:

$ systemctl show sshd.service -p Conflicts
Conflicts=shutdown.target

Masking and Unmasking Units

Making a Unit - marks the unit as completely unstartable, automatically or manually, by linking the unit to /dev/null

# Prevents the unit from being started, automatically or manually, for as long as it has been masked
$ sudo systemctl mask nginx.service
# Unmask a service returns it to its previous state, allowing it to be started or enabled:
$ sudo systemctl unmask nginx.service

Editing Unit Files

The edit command, by default, will open file snippet for the unit in question:

$ sudo systemctl edit nginx.service

The above command creates a blank file that can be used to override or add directives to the unit definition. A directory will be created within the /etc/systemd/system directory which contains the name of the unit with .d appended. For instance, for the nginx.service, a directory called nginx.service.d will be created.

Within this directory, a snipped will be created called override.conf. When the unit is loaded, systemd will, in memory, merge the override snippet with the full unit file. The snippet's directives will take precedence over those found in the original unit file.

If you wish to edit the full unit file instead of creating a snippet, use the --full flag.

This will load the current unit file into the editor, where it can be modified. When the editor exits, this changed file will be written to /etc/systemd/system, which will take precedence over the system's unit definition (usually found somewhere in /lib/systemd/system).
To remove any additions you have made, either delete the unit's .d configuration directory or the modified service file from /etc/systemd/system.
After deleting the file or directory, you should reload the systemd process so taht it no longer attempts to reference these files and reverts back to using the system copes. You can do this by typing:

$ sudo systemctl daemon-reload

Adjusting the System State (Runlevel) With Targets

Target are special unit files that describe a system state or synchronization point. Like other units, the files that defined targets can eb identified by their suffix, which in this case is .target. Targets do not do much themselves, but are instead used to group other units together.

This can be used in order to bring the system to certain states, much like other init systems use runlevels. They are used as a reference for when certain functions are available, allowing you to specify the desired state instead of the individual units needed to produce the state.

Getting and Setting the Default Target

The systemd process has a default target that it uses when starting the system. Satisfying the cascade of dependencies from that single target will bring the system into the desired state.

# Find the default target for your application
$ systemctl get-default
# Set the default target
$ sudo systemctl set-default graphical.target

Listing Available Targets

$ systemctl list-unit-files --type=target

Unlike runlevels, multiple targets can be active at one time. An active target indicates that systemd has attempted to start all of the units tied to the target and has not tries to tear them down again.

Isolating Targets

It is possible to start all of the units associated with a target and stop all units that are not part of the dependency tree. The command that we need to do this is called, appropriately, isolate.

$ sudo systemctl isolate multi-user.target

Using Shortcuts for Important Events

# Put the system into rescue (single-user) mode
$ sudo systemctl rescue
# Halt the system
$ sudo systemctl halt
# Initiate a full shutdown
$ sudo systemctl powerofff
# Restart
$ sudo systemctl reboot

Journalctl to View and Manipulate Systemd Logs

Some of the most compelling advantages of using systemd are those involved with process and system logging. When using other tools, logs are usually dispersed throughout the system, handled by different daemons and processes, and can be fairly difficult to interpret when they span multiple applications. systemd attempts to address these issues by providing a centralized management solution for logging all kernel and userland processes. The system that collects and manages these logs is known as the journal.

The journal is implemented with the journald daemon, which handles all of the messages produced by the kernel, initrd, services, etc. In this guide, we will discuss how to use the journalctl utility, which can be used to access and manipulate data within the journal.

General Idea

One of the impetuses behind the systemd journal is to centralize the management of logs regardless of where the messages are originating. The journald daemon collects data from all available sources and stores them in a binary format for easy and dynamic manipulation.

Advantages of This Approach:

Administrators can dynamically display log data according to their needs.
Storing in binary format means that the data can be displayed in arbitrary output formats depending on what you need at the moment. For daily log management you may be used to viewing logs in the standard syslog format

The systemd journal can either be used with an existing syslog implementation, or it can replace the syslog functionality, depending on you needs. While the systemd journal will cover most of the administrator's logging needs, it can also complement existing logging mechanisms.

Setting the System Time

systemd will display results in local time by default
You can set it up to display in UTC time as well

# see available timezones
$ timedatectl list-timezones
# Set the timezone
$ sudo timedatectl set-timezone zone
# Check that the time is correct
$ timedatectl status
	Local time: Fri 2021-07-09 14:44:30 EDT
	Universal time: Fri 2021-07-09 18:44:30 UTC
	RTC time: Fri 2021-07-09 18:44:31
	Time zone: America/New_York (EDT, -0400)
	System clock synchronized: yes
  NTP service: active
 	RTC in local TZ: no

Basic Log Viewing

To see the logs that the journald demon has collected, use the journalctl command

Oldest entries will be shown at the top.

$ journalctl
-- Logs begin at Tue 2015-02-03 21:48:52 UTC, end at Tue 2015-02-03 22:29:38 UTC. --
Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (max allowed 49.
Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (max allowed 49.
Feb 03 21:48:52 localhost.localdomain systemd-journald[139]: Received SIGTERM from PID 1 (systemd).
Feb 03 21:48:52 localhost.localdomain kernel: audit: type=1404 audit(1423000132.274:2): enforcing=1 old_en
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.
Feb 03 21:48:52 localhost.localdomain kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/
Feb 03 21:48:52 localhost.localdomain kernel: SELinux:  8 users, 102 roles, 4976 types, 294 bools, 1 sens,
Feb 03 21:48:52 localhost.localdomain kernel: SELinux:  83 classes, 104131 rules

. . .

This journal includes logs from the early boot process, the kernel, the initrd, and application standard error and out. Use the --utc flag to display times in UTC time.

Journal Filtering By Time

Use the -b flag to only display journal entries from the most recent boot.

Some distributions enable saving previous boot information by default, while others disable this feature. To enable persistent boot information, you can create the directory to store the journal by typing

$ sudo mkdir -p /var/log/journal
# Edit the journal configuration file
$ sudo nano /etc/systemd/journald.conf

Under the [Journal] set the Storage= option to "persistent" to enable persistent logging:

When saving previous boots is enabled, journalctl provides some commands to help you work with boots as a unit of division. To see the boots that journald knows about, use the --list-boots option with journalctl:

$ journalctl --list-boots
-2 caf0524a1d394ce0bdbcff75b94444fe Tue 2015-02-03 21:48:52 UTC—Tue 2015-02-03 22:17:00 UTC
-1 13883d180dc0420db0abcb5fa26d6198 Tue 2015-02-03 22:17:03 UTC—Tue 2015-02-03 22:19:08 UTC
 0 bed718b17a73415fade0e4e7f4bea609 Tue 2015-02-03 22:19:12 UTC—Tue 2015-02-03 23:01:01 UTC

To see the journal from a previous boot , use the relative pointer (first column above), or the boot ID:

$ journalctl -b -1
$ journalctl -b caf0524a1d394ce0bdbcff75b94444fe

You can filter b arbitrary time limits using the --since and --until options, which restrict the entries displayed to those after or before the given time, respectively.

The time values can come in various formats. For absolution time values, you should use the format in the example below:

$ journalctl --since "2015-01-10" --until "2015-01-11 03:00"
# Example of relative formats
$ journalctl --since yesterday
$ journalctl --since 09:00 --until "1 hour ago"

Filtering by Message Interest

You can use the -u option to filter by what unit you are interested in:

$ journalctl -u nginx.service
# Can combine multiple unit logs
$ journalctl -u nginx.service -u php-fpm.service --since today
# Filtering by process, user, or group ID
$ journalctl _PID=8088
$ id -u www-data # gets the user id of the www-data user
$ hournalctl _UID=33 --since today

You can find out about all of the available journal fields by typing:

man systemd.hournal-fields

You can filter by component path:

journalctl /usr/bin/bash # journalctlwill display all of the entries that involve the exectable in question

journalctl -k # Display kernel messages

Filter by message priority using the -p flag:

$ journalctl -p err -b

The err in the above command is the priority name of the message priority. You can use the priority name or the number corresponding to the priority name:

0: emerg
1: alert
2: crit
3: err
4: warning
5: notice
6: info
7: debug

Modifying the Journal Display

$ journalctl --no-full # truncate the output
$ journalctl -a # Tell journalctl to display all info at once
$ journalctl --no-pager # output to standard output, not to pager

Use the -o flag to output the data in certain formats:

$ journalctl -b -u nginx -o json
$ journalctl -b -u nginx -o json-pretty

List of Available Formats:

cat: Displays only the message field itself
export: A binary format suitable for transferring or backing up
json: Standard JSON with one entry per line
json-pretty: JSON formatted for readability
json-sse: JSON formatted output wrapped to make server-sent event compatible
short: The default syslog style output
short-iso: The default format augmented to show ISO 8601 wallclock timestamps
short-monotonic: The default format with monotonic timestamps
short-precise: The default format with microsecond precision
verbose: Shows every journal field available for the entry, including those usually hidden internally

Active Process Monitoring

$ journalctl -n # display the ten most recent logs
$ journalctl -n 20 # Display the 20 most recent logs
$ journalctl -f # actively follow the logs as they are being written

Journal Maintenance

You may be wondering about what the cost is of storing all of the data we've seen so far. Furthermore, you may be interested in cleaning up some older logs and freeing up space.

$ journalctl --disk-usage # Find out the current disk usage
$ sudo journalctl --vacuum-size=16 # Removes old enries until the total journal space taken up on disk is at the requested size
$ sudo journalctl --vacuum-time=1years # Only keep entries from the last year

You can configure your server to place limits on how much space the journal can take up. This can be done by editing the /etc/systemd/journald.conf file.

The following items can be used to limit the journal growth:

SystemMaxUse=: Specifies the maximum disk space that can be used by the journal in persistent storage
SystemKeepFree=: Specifies the amount of space that the journal should leave free when adding journal entries to persistent storage
SystemMaxFileSize=: Controls how large individual journal files can grow to in persistent storage before being rotated
RuntimeMaxUse=: Specifies the maximum disk space that can be used in volatile storage (within the /run filesystem)
RuntimeKeepFree=: Specifies the amount of space to be set aside for other uses when writing data to volatile storage (within the /run filesystem)
RuntimeMaxFileSize=: Specifies the amount of space that an individual journal file can take up in volatile storage (within the /run filesystem) before being rotated