Problem

You want to enter your system’s BIOS/UEFI setup.

Solution

Enter your BIO/UEFI setup by pressing the appropriate Fn key at startup. On Dell, ASUS, and Acer systems this is usually F2, and Lenovo uses F1. However, this varies; for example, some systems use the Delete key, so check your machine’s documentation. Some systems tell which key to press on their startup screens. It can be a bit tricky to press the key at the right time, so start pressing it right after you press the power button, just like banging on an elevator button to make it arrive faster.

Every UEFI looks different; for example, Lenovo’s is bright and well organized (Figure 1-1).

The ASRock UEFI on my test system is dark and dramatic (Figure 1-2). This particular motherboard targets gamers and has many settings for overclocking the CPU and other performance enhancements. This screen shows the motherboard browser; hover the cursor over any item and it provides information about that item.

Figure 1-1. Lenovo’s UEFI on a new ThinkPad

Figure 1-2. ASRock UEFI has a motherboard browser

Discussion

When you boot up your computer, the first startup instructions come from the BIOS or UEFI firmware, stored on the computer’s motherboard. BIOS is the old legacy system that has been with us since 1980. UEFI is its modern replacement. UEFI includes legacy BIOS support. Nearly all computers made after the mid-2000s have UEFI.

UEFI has considerably more features than the old BIOS and is like a little operating system. The UEFI setup screens control boot order, boot devices, security options, Secure Boot, overclocking, displaying hardware health, networking, and many more functions.

See Also

• The documentation for your motherboard

• Unified Extensible Firmware Interface Forum

Problem

You want to find and download a Linux installation image.

Solution

First, you need to choose which Linux you want to try. If you don’t know where to start, I recommend Ubuntu Linux. Fedora Linux and openSUSE Linux are also excellent for anyone new to Linux.

After your download is completed, verify that you have a good download. This is an important step that ensures you have an image that did not get damaged during the download or was otherwise altered in some way.

Every Linux distribution provides signed keys and checksums for its download images. Ubuntu provides copy-and-paste instructions. Open a terminal and change to the directory where you downloaded Ubuntu. For Ubuntu 21.04, verifying the installation image looks like this:

$ echo "fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88 \
*ubuntu-21.04-desktop-amd64.iso" | shasum -a 256 --check

ubuntu-21.04-desktop-amd64.iso: OK

If you see “shasum: WARNING: 1 computed checksum did NOT match,” then your download is no good, assuming you copied the correct checksum. In most cases the image was corrupted during the download, so try downloading it again.

Other Linux distributions provide slightly different verification methods, so follow their instructions.

Discussion

A great site to learn about the hundreds of Linux distributions is Distrowatch.com. Distrowatch provides news and information about more Linux distros than anyone.

See Also

• man 1 sha256sum

• Ubuntu Linux

• Fedora Linux

• openSUSE Linux

Problem

You have downloaded a Linux installation *.iso image, and you want to transfer it to a USB stick to create your own installation media. You prefer a graphical tool to create your installation media.

Solution

Try UNetbootin, Universal Netboot Installer. It runs on Linux, macOS, and Windows, so you can download and create a Linux installation disk on any of these operating systems. UNetbootin creates an installation USB drive from your downloaded *.iso file or downloads a fresh *.iso file (Figure 1-3).

You may use any size USB stick (that is larger than your *.iso file, of course). The *.iso overwrites the entire device, so you can’t use it for anything else, and you must use a separate USB stick for each *.iso file.

Figure 1-3. Using UNetbootin to create an installable Linux USB stick

The UNetbootin site provides downloads and instructions. Some Linux distributions provide UNetbootin packages, but downloading from the UNetbootin site is simple and always up-to-date.

Discussion

Other good graphical apps are USB Creator, ISO Image Writer, and GNOME MultiWriter, which copies to multiple USB drives at one time.

After you have created your installation USB drive, you can view the files on it. The single *.iso expands into a complete filesystem full of files and directories, like this example for Ubuntu:

$  ls -C1 /media/duchess/'Ubuntu 21.04.1 amd64'/
boot
casper
dists
EFI
install
isolinux
md5sum.txt
pics
pool
preseed
README.diskdefines
ubuntu

Every Linux distribution sets up its installers in its own way. This example shows Fedora’s installation files:

$  ls -C1 /media/duchess/Fedora-WS-Live-34-1-6/
EFI
images
isolinux
LiveOS

It would be lovely to have a single USB stick with a herd of Linux installation files, and there are some programs that do this. My favorite is Ventoy. Ventoy supports a large number of Linux distributions. It runs on both Linux and Windows, and you can create a USB stick full of Linux installers for running live Linuxes, and for permanent installations to hard disks.

See Also

• UNetbootin

• Chapter 9

• Ventoy

Problem

You want to create a Linux DVD installation disk with a graphical tool.

Solution

Use K3b (KDE Burn Baby Burn). K3b is the best graphical CD/DVD writing app for Linux.

If you do not have a Linux system, then use any CD/DVD writing program that writes an ISO 9660 image. Your chosen CD/DVD writer will phrase this as something like “burn an existing image to disk.”

You will see something like Figure 1-4 on K3b. Click “Burn Image,” and note the confirmation on the bottom left that says “Write an ISO 9660…image to an optical disk.”

On the next screen (Figure 1-5), in the top left drop-down selector, select your *.iso image. Then on the top right select “ISO 9660 filesystem image.” At the bottom under Settings, check “Verify written data.” This computes a checksum after your image is written and compares it to the checksum of the original *.iso. This is an important step because if the checksums do not match you have a corrupted disk, which is unusable.

Figure 1-4. Creating an installation DVD with K3b

Figure 1-5. Configuring the burn

When the disk is successfully written, you will see a success message like the one shown in Figure 1-6. If there were any errors, this screen will show some helpful error messages.

Figure 1-6. Success!

Discussion

Brasero and XFBurn are also great CD/DVD writing apps for Linux, with simpler interfaces than K3b, but still plenty of functionality.

The tech world changes fast. Just a few years ago I was burning CDs and DVDs for everything. Then USB devices swept the market, and I haven’t burned a disk for years, until I started writing this chapter.

Take heart, for CDs and DVDs are not obsolete, despite the efforts of computer makers to make them obsolete by not including CD/DVD drives in their machines. This is not a problem because you can purchase an external USB CD/DVD drive. You can even find bus-powered drives, so you only need a USB cable, and don’t have to hassle with a power cable. CD/DVD blanks are still good quality, so if you prefer optical disks, they are a reliable choice.

See Also

• K3b

• Brasero

Problem

You want a command-line tool to create a bootable CD/DVD.

Solution

Try the wodim command. Your optical drive is most likely /dev/cdrom, symlinked to /dev/sr0. Use the symlink because it has correct permissions:

$ ls -l /dev | grep cdr
lrwxrwxrwx  1 root root           3 Mar  7 12:38 cdrom -> sr0
lrwxrwxrwx  1 root root           3 Mar  7 12:38 cdrw -> sr0
crw-rw----+ 1 root cdrom    21,   2 Mar  7 08:34 sg2
brw-rw----+ 1 root cdrom    11,   0 Mar  7 12:57 sr0

Then copy your installation image to disk:

$ wodim dev=/dev/cdrom -v ubuntu-21.04-desktop-amd64.iso

Discussion

In the ls -l example, there are sg2 and sr0 devices. sg2 is a character device, and sr0 is a block device. Character devices provide raw access to a hardware device using the raw kernel drivers. Block devices provide buffered access to a hardware device through various software programs that handle reading and writing to physical media. Users interact with storage devices, like DVDs and hard disks, through the kernel’s block device drivers. You can see raw and block kernel modules listed in your /boot/config-* file.

See Also

• man 1 wodim

Problem

You want to create your installation USB drive from the command line, rather than using a graphical tool.

Solution

Use the dd command. dd is on every Linux and works the same way on all of them.

First, verify the dev name for your USB stick with the lsblk command, so that you copy your image to the correct device. In the following example, that is /dev/sdb:

$ lsblk -o NAME,FSTYPE,LABEL,MOUNTPOINT

NAME   FSTYPE  LABEL       MOUNTPOINT
sda
├─sda1 vfat                /boot/efi
├─sda2 xfs     osuse15-2   /boot
├─sda3 xfs                 /
├─sda4 xfs                 /home
└─sda5 swap                [SWAP]
sdb
└─sdb1 xfs     32gbusb
sr0

The following example creates a USB installation stick and shows its progress:

$ sudo dd status=progress if=ubuntu-20.04.1-LTS-desktop-amd64.iso of=/dev/sdb
211509760 bytes (212 MB, 202 MiB) copied, 63 s, 3.4 MB/s

This takes a few minutes. When it’s finished, it looks like this:

2782257664 bytes (2.8 GB, 2.6 GiB) copied, 484 s, 5.7 MB/s
5439488+0 records in
5439488+0 records out
2785017856 bytes (2.8 GB, 2.6 GiB) copied, 484.144 s, 5.8 MB/s

Remove the drive, then reinsert it and take a quick look at the files. Figure 1-7 shows the installation files for Ubuntu Linux in the Thunar file manager.

Figure 1-7. Ubuntu Linux installation files in Thunar

The files are marked with padlocks because the Ubuntu installer uses the SquashFS read-only filesystem. You can read them, but not delete or edit them.

The installation USB stick is ready to use.

Discussion

Identifying the correct device to copy your installation to is super important. In the lsblk example, there are only two storage devices. Note the LABEL column; you can create labels on filesystems so you know what they are. (See Recipe 9.2 and the filesystem creation recipes in Chapter 11 to learn how to create filesystem labels.)

The graphical installation media creators are good, but I prefer the dd command because it is simple and reliable. dd is short for Disk Duplicator. This is one of those ancient useful GNU commands, in the GNU coreutils package, that has been around forever.

See Also

• man 1 dd

Problem

You want to try a simple Ubuntu installation. You have your installation medium ready, and you know how to boot to your installation medium. There is nothing on the computer that you want to keep, so Ubuntu can take over the hard drive.

Solution

The following example demonstrates a fast, simple installation of Ubuntu Linux 21.04, Hirsute Hippo. All Ubuntu releases have alliterative animal names.

Insert your installation device, power on the machine, and open your system’s one-time boot menu. Select the installation device and boot up (Figure 1-8).

Figure 1-8. Booting to an installation USB stick

When the GRUB menu appears, select the default option. For Ubuntu 21.04 this is Ubuntu, and it boots by default when you make no selection (Figure 1-9).

Figure 1-9. Ubuntu installer’s GRUB boot menu

Then you will have a choice of Try Ubuntu and Install Ubuntu. Try Ubuntu launches the live version, and Install Ubuntu opens the installer (Figure 1-10). It doesn’t matter which one you select because there is a big installation button on the live desktop.

Figure 1-10. Choosing the live image or the installer

When you launch the installer, it walks you through a few steps. First, language and keyboard layout.

Then, if you have a wireless network interface, you have the option to set it up or wait until after installation.

Then configure your installation. On the “Updates and other software” screen, select “Normal installation” (Figure 1-11).

Figure 1-11. Select Normal installation

On the next screen, select “Erase disk and install Ubuntu,” then click Install Now (Figure 1-12).

Figure 1-12. Select Erase disk and install Ubuntu

The next screen asks “Write the changes to disk?” Click Continue. There are a few more screens: set your time zone, create your username, password, and hostname, and then the installation starts. You do not have to do anything until the installation is finished, then restart the computer, remove your installation device when prompted, and press the Enter key. After restart you will have a few setup screens, then you can play with your nice new Ubuntu Linux.

Discussion

Most Linux distributions have a similar installation process: boot your installation medium, then choose a default simple installation or choose a customizable installation. Some ask all the questions at the beginning, such as username and password; others do the final setup after reboot.

Linux installers typically have back buttons to go back and make changes. You may quit at any time, though this may leave your system in an unusable state. This is not fatal; just start over and do a complete installation.

You may reinstall as many times on as many machines as you want without worrying about license keys, except for the enterprise distros that require registration keys (Red Hat, SUSE, or Ubuntu with paid support).

See Also

• Ubuntu documentation

Problem

You want to set up your own partitioning scheme.

Solution

In this recipe we will redo the example Ubuntu installation in Recipe 1.7 and set up our own partitioning scheme.

You can set up partitioning in any number of ways. Table 1-1 shows how I prefer to set up my Linux workstations.

When you get to the Installation Type screen, select “Something else” to start your customized installation (Figure 1-13).

Figure 1-13. Selecting the installation type

When you get to the partitioning screen, erase the entire disk by clicking New Partition Table. Then you will see something like Figure 1-14.

Figure 1-14. Creating a new partition table

To create new partitions click on the “free space” line to select it, then click the plus sign, +, to add a new partition. Set the size, filesystem, and mountpoint. Figure 1-15 creates a 500 MB /boot partition.

Figure 1-15. Creating the boot partition

Click on “free space” again, click the plus sign, and keep going until you have created all of your partitions. Figure 1-16 shows the result: /boot, /home, /var, /tmp, and the swap file are all on their own partitions.

Figure 1-16. Partitions set up and ready to continue the installation

Discussion

The examples are from a virtual machine, so the hard disk is vda instead of sda.

The example in the recipe uses the Ext4 filesystem on all the partitions. You may use whatever filesystems you want; see Chapter 11 to learn more.

Disk partitions are like having a bunch of separate physical disks. Each one is an independent section of the hard disk, and each partition can have a different filesystem. The filesystems you select, and their sizes, all depend on how you use your system. If you need a lot of data storage, then /home needs to be large. It could even be a separate disk.

Giving /boot its own partition makes managing multiboot systems easier because this makes the boot files independent of whatever operating systems you install or remove. 500 MB is more than enough.

Putting /, root, in its own partition makes it easy to restore or to nuke and replace with a different Linux. 30 GB is more than enough for most distros, except when you use the Btrfs filesystem, then you should make it 60 GB to make room for storing snapshots.

Put /home on its own partition to isolate it from the root filesystem, so you can replace your Linux installation without touching /home. /home could even be on a separate disk.

/var and /tmp can fill up from runaway processes. Putting them on their own partitions prevents them from crashing the other filesystems. Mine are usually 20 GB each, and for a busy server they need to be larger.

Putting a swap file equal to the size of your RAM on its own partition enables suspend-to-disk.

See Also

• The discussion in Recipe 3.9 to learn about suspend and sleep states

• Chapter 8

• Chapter 9

Problem

You have /home on its own partition and want to preserve it for the new Linux installation.

Solution

In Recipes 1.7 and 1.8 we erased the existing installation by creating a new partition table. When you have partitions that you want to preserve, such as /home or any shared directory, do not create a new partition table. Instead, edit the existing partitions. You can delete existing partitions, create new partitions, and reuse existing partitions.

In the following example in the Ubuntu installer, /dev/sda3 is a separate /home partition. Right-click on it, then click “Change…” Then you can set its mountpoint to /home, and make sure that the “Format?” box is NOT checked (Figure 1-17). If you format it or change the filesystem type, all data on the partition will be erased.

Figure 1-17. Saving /dev/sda3, instead of overwriting it

Discussion

The Discussion in Recipe 1.8 goes into some detail on customizing your partitioning scheme and which filesystems benefit from being isolated on their own partitions.

See Also

• Chapter 8

• Chapter 9

Problem

You don’t want the default package installation, but prefer to select your own software to install. For example, you might want to set up a development workstation, a web server, a central backup server, a multimedia production workstation, a desktop publishing workstation, or choose your own office productivity applications.

Solution

Every Linux distro manages installation options a little differently. In this recipe you will see examples for openSUSE and Fedora Linux. openSUSE supports multiple installation types from a single installation image, and Fedora Linux has several different installation images.

These two examples are typical of the general-purpose Linux distributions.

Remember, you can install and remove software all you want after installation.

openSUSE

The openSUSE installer supports both a simple installation from defaults and extensive customization options. It has two screens that control package selection. The first screen (Figure 1-18) provides a selection of system roles to choose from, such as a desktop system with the KDE or GNOME graphical environment, a generic desktop with the IceWM window manager, a server with no graphical environment, or a transactional server with no graphical environment. Each role comes with a prefab set of packages. You can install one of these as is or select packages to install or remove.

Each role is customizable, as you will see a few screens later (Figure 1-19).

Figure 1-18. openSUSE installation roles

Figure 1-19. openSUSE installation settings

Click Software to open the package selection screen. This screen displays the openSUSE patterns, which are related groups of packages you can install with a single command. I like the Xfce desktop, so I am adding it to my installation (Figure 1-20).

Note the Details button at the bottom left. Click this to open a screen with multiple tabs for more fine-grained package selection (Figure 1-21). In this screen you will find a massive amount of information: the individual packages in each pattern, package groups, download repositories, installation summary, dependencies, and information on every package. Use the window on the right to select or deselect packages from each pattern. The installer will automatically resolve dependencies after your changes.

Figure 1-20. openSUSE software patterns

Figure 1-21. openSUSE individual package selection

When you are finished with software selection, you are returned to the Installation Summary screen, giving you another chance to change your installation settings. Click the green Next button to complete the installation.

Fedora Linux

The Fedora Linux Workstation and Server installers provide only partitioning options and no package customization. You need the 600 MB network installer image for a customizable installation, from Fedora Alternative Downloads. It is labeled as Fedora Server, but it provides complete package selection for any type of Fedora installation. Set up all your installation choices from the Installation Summary screen (Figure 1-22).

Take note of all the installation options on the Installation Summary screen: software selection, user creation, partitioning, keyboard and language, time zone, network, and hostname. Click Software Selection to open the screen for selecting the packages you want to install (Figure 1-23).

Figure 1-22. Fedora Linux network installer

Figure 1-23. Fedora Linux package selection

When you are finished, click Done; you are returned to the Installation Summary screen. When you are finished setting up your installation, click Begin Installation, and the rest of the installation runs unattended.

Discussion

Whatever Linux you want to try, read its documentation and release notes. This is important information that saves a lot of aggravation. Also look for forums, mailing lists, and wikis to find help.

You may install as many desktop environments as you want, and then select the one you want to use when you log in. The button to select desktops is usually small and not obvious; for example, the default Ubuntu login screen (Figure 1-24) hides the desktop selector button until you click on a username. Xfce, Lxde, GNOME, and KDE are some of the popular graphical desktops. GNOME is the default on Ubuntu, openSUSE, and Fedora.

Figure 1-24. Selecting a different graphical environment

See Also

• openSUSE documentation

• SUSE Transactional Updates

Problem

You want to install more than one Linux distribution on your computer in a multiboot setup, then select the one you want to run from your boot menu.

Solution

No worries, for you can install as many Linuxes as your hard disk (or disks) will hold. You must already have one Linux installed, and it must have a separate /boot partition. Then the steps are:

1. Provide sufficient free disk space for the new Linux, which can be on the same hard disk as your existing Linux, or on a separate hard disk, either internal or external.

2. Make careful note of the partitions that belong to your first installed Linux, so that you do not accidentally overwrite or delete any partitions you want to keep.

3. Mount the /boot partition in every new Linux that you install, and do not format it.

4. Boot to your installation medium, then configure the new installation to install on the free disk space.

The installer will automatically find your existing Linux installation and add the new Linux to the boot menu. After the installation is completed, you will see a boot menu like Figure 1-25, which has options to boot to Linux Mint or Ubuntu.

Figure 1-25. New boot menu with Linux Mint and Ubuntu

Discussion

If you need to free up space on your hard disk, you can shrink existing partitions safely (see Recipe 9.7) before you launch the installer. It is safest to do this on unmounted partitions, and some filesystems cannot be shrunk while they are mounted. Use SystemRescue to shrink partitions, see Recipe 19.12.

Most Linux installers are smart enough to recognize existing Linux installations and to offer the option to preserve them. In Figure 1-26 you see the Linux Mint installer, providing the options to take over your whole hard disk or to install next to Ubuntu without deleting it.

Figure 1-26. Installing Linux Mint next to Ubuntu

See Also

• Recipe 8.9

• Recipe 9.7

• Recipe 19.12

• The installation documentation for your Linux distribution

Problem

You want to dual-boot Linux and Windows on your computer.

Solution

Dual-booting Linux and Windows installs both systems on one computer, and then you choose the one you want to use from your boot menu at startup.

It is best to install Windows first, if it is not already installed, and install Linux second. Windows likes to control the bootloader, so installing Linux second allows Linux to take control.

As always, make sure you have fresh backups and Windows recovery media.

After Windows is installed, start your Linux installation. You will install Linux in whatever way you want: a simple installation or a customized installation where you set up partitioning and package selection. There is an important option specific to multibooting:

1. If you have a single hard drive, then the “Device for boot loader installation” is /dev/sda.

2. If you have Windows on one hard disk and are installing Linux to a second hard disk, the “Device for boot loader installation” is the Linux disk. Use the device name, for example /dev/sdb, not a partition name, like /dev/sdb1.

In Figure 1-27, there are two hard drives, with Windows on /dev/sda and Linux on /dev/sdb.

Figure 1-27. Installing Ubuntu next to Windows

Be very certain you are installing Linux to the correct location and not overwriting Windows. You may partition for Linux just as you would if you were installing it standalone, and again, be careful which partitions you change.

Once you have your partitioning set up and are satisfied with your installation configuration, go ahead and complete your Linux installation. After it is finished and you have rebooted, your GRUB menu will have entries for both systems (Figure 1-28).

Figure 1-28. openSUSE and Windows in the GRUB boot menu

Discussion

You can install as many Linux and Windows systems on your system in a multiboot setup as you have room for on your hard drives.

There are other ways to run Linux and Windows on the same machine. Windows 10 includes the Windows Subsystem for Linux 2 (WSL 2), which runs supported Linux distributions in a virtual environment. You can run Windows in virtual machines on Linux if you have Windows installation media. Virtual machines are lovely because you can run multiple operating systems at the same time, though you need higher-end CPUs and lots of memory.

VirtualBox and QEMU/KVM/Virtual Machine Manager are good free virtual machine hosts that run on Linux.

See Also

• Windows Subsystem for Linux Documentation

• VirtualBox

• KVM

• Virtual Machine Manager

• QEMU

Problem

You purchased a computer with Windows 8 or 10 preinstalled, and you can’t find your product key.

Solution

Let Linux find it for you. Run the following command from a Linux system installed on the same computer with Windows, or from SystemRescue:

$ sudo cat /sys/firmware/acpi/tables/MSDM
MSDMU
DELL  CBX3
AMI
FAKEP-RODUC-TKEY1-22222-33333

And there it is on the last line.

If you can log in to Windows, run the following command in Windows to retrieve your product key:

C:\Users\Duchess> wmic path softwarelicensingservice get OA3xOriginalProductKey
OA3xOriginalProductKey
FAKEP-RODUC-TKEY1-22222-33333

Discussion

If you have no recovery media, Windows 10 is a free download. You will need your 25-digit OEM product key for a fresh installation.

See Also

• Download Windows 10

Problem

You have downloaded a Linux *.iso file and are curious to see what it looks like after it is unpacked. You could go ahead and create a bootable DVD or USB stick, and then inspect the files, but you really want to unpack it without copying it to another device.

Solution

Linux has a pseudodevice called the loop device. This makes your *.iso image accessible like any other filesystem. Follow these steps to mount your *.iso image file in a loop device.

First, create a mountpoint in your home directory, giving it whatever name you want. In the example it is called loopiso:

$ mkdir loopiso

Mount your *.iso in this new directory. In the example it is a Fedora Linux installation image:

$ sudo mount -o loop Fedora-Workstation-Live-x86_64-34-1.2.iso loopiso
mount: /home/duchess/loopiso: WARNING: device write-protected, mounted read-only

See the mounted filesystem in a file manager (Figure 1-29).

Figure 1-29. A mounted .iso in Fedora Linux 34

You can enter the directories and read the files. You won’t be able to edit any files because they are mounted read-only.

When you are finished, unmount it:

$ sudo umount loopiso

Discussion

The loop device maps a regular file to a virtual partition, and you can set up a virtual filesystem in this file. If you want try to creating your own, a little bit of web searching will find a lot of how-tos. Start with man 8 losetup.

See Also

• man 8 mount

• man 8 losetup

Problem

Whenever you change your GRUB configuration, you need to rebuild it.

Solution

The command to rebuild your GRUB configuration varies. On Fedora and openSUSE, use this command:

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Some distros, such as Ubuntu, use:

$ sudo grub-mkconfig -o /boot/grub/grub.cfg

Ubuntu Linux also has a script that runs grub-mkconfig, update-grub:

$ sudo update-grub

Discussion

Some Linux distributions helpfully provide the correct command at the top of /etc/default/grub.

Remember to always verify your correct filenames and paths when you edit your GRUB configuration because they vary on the different Linuxes.

See Also

• Your motherboard documentation, to learn about your system’s BIOS/UEFI

• GNU GRUB Manual

• GRUB has multiple single-purpose man pages; run man -k grub to see all of them

• info grub or info grub2

Problem

Your favorite Linux distribution hides the GRUB menu when you have only one operating system installed on your computer, and you want it to appear every time you boot up.

Solution

Several Linux distros do this, including Ubuntu and Fedora. You can unhide your GRUB menu temporarily by pressing and holding the Shift key at startup.

Edit /etc/default/grub to unhide it permanently with the following options:

GRUB_TIMEOUT="10"
GRUB_TIMEOUT_STYLE=menu

If the lines GRUB_HIDDEN_TIMEOUT=0 and GRUB_HIDDEN_TIMEOUT_QUIET=true are in your file, comment them out.

After changing /etc/default/grub, rebuild your GRUB configuration (Recipe 2.1).

Discussion

GRUB_HIDDEN_TIMEOUT=0 means do not display the GRUB menu, while GRUB_HIDDEN_TIMEOUT_QUIET=true means do not display the countdown timer.

If you install another operating system in a multiboot configuration, then the GRUB menu should unhide itself.

See Also

• Recipe 2.1

• GNU GRUB Manual

• GRUB has multiple single-purpose man pages; run man -k grub to see all of them

• info grub or info grub2

Problem

You are wondering about the extra entries in your GRUB menu that reference specific Linux kernel versions, like in Figure 2-3. You want to know what they are for and what to do with them.

Figure 2-3. GRUB kernel boot options

Solution

Over time, as you update your Linux system, older Linux kernels are retained and added to your GRUB menu. This provides an easy way to boot to a known good older kernel if something goes awry with a newer kernel. You don’t have to keep the older kernels and can remove them with your package manager.

Discussion

In olden times kernel updates were a big deal, because they often meant bug fixes, additional support for hardware, such as video, network, and audio interfaces, and support for software features, such as proprietary file formats and new protocols. The rate of change was rapid, and it was not unusual for a new kernel to not work correctly, so keeping the option to boot to an older kernel was routine. These days such troubles are less common, and kernel updates are generally undramatic.

See Also

• GNU GRUB Manual

• The Linux Kernel Archives

Problem

You know that configuring GRUB is done a little differently than most programs, and you want to know where the GRUB configuration files are and which ones you use to manage GRUB.

Solution

GRUB configuration files are in /boot/grub/, /etc/default/grub, and /etc/grub.d/. The GRUB configuration is complex, with many scripts and modules.

/etc/default/grub is for configuring the appearance of the GRUB menu you see at startup, such as hiding or showing the boot menu, applying themes and background images, menu timeout, and kernel options.

The files in /etc/grub.d/ support more complex configurations, and /boot/grub/ stores image and theme files for customizing the appearance of your GRUB menu.

The main GRUB configuration file is /boot/grub/grub.cfg, which GRUB reads at startup. You do not edit this file because it is built from /etc/grub.d/ and /etc/default/grub; every time you make configuration changes you must rebuild the GRUB configuration.

The GRUB configuration is automatically rebuilt when you install any updates that affect the boot process, such as installing newer kernels and removing older kernels.

Discussion

If you are interested in scripting, studying the GRUB files is an excellent course in organizing large numbers of interdependent scripts.

The files in /etc/grub.d/ are called drop-in files. Rather than dealing with a giant single configuration file, each drop-in file contains a configuration for a specific task. These files are numbered in the order that GRUB should read them, with lower numbers indicating higher priority. The following example is from Fedora 32:

$ sudo ls -C1 /etc/grub.d/
00_header
01_users
08_fallback_counting
10_linux
10_reset_boot_success
12_menu_auto_hide
20_linux_xen
20_ppc_terminfo
30_os-prober
30_uefi-firmware
40_custom
41_custom
backup
README

Each of these files is a script, and each must have the executable bit set. You can disable any of them by clearing the executable bit, like this:

$ sudo chmod -x 20_linux_xen

Re-enable a script by adding the executable bit:

$ sudo chmod +x 20_linux_xen

See Also

• GNU GRUB Manual

• GRUB has numerous man pages; run man -k grub to see all of them

• info grub or info grub2

• Chapter 6

Problem

You want to write the most minimal working GRUB configuration.

Solution

This is the most basic /etc/default/grub file, with only the necessary entries to boot a Linux system and show the GRUB menu. The following example is for openSUSE Leap 15.2:

# If you change this file, run 'grub2-mkconfig -o /boot/grub2/grub.cfg'
# afterwards to update /boot/grub2/grub.cfg.

GRUB_DEFAULT=0
GRUB_TIMEOUT=10
GRUB_TIMEOUT_STYLE=menu

Remember Figure 2-1? Figure 2-4 is the same system, but with a minimal GRUB configuration.

Figure 2-4. Minimal GRUB menu

There are more options you can try, such as different ways to set the default boot option, change background images and themes, change the colors, and change the screen resolution. See the Discussion to learn about these.

Discussion

There are numerous options you can use in /etc/default/grub, and you can ignore most of them. These are the options that I think are the most useful:

GRUB_DEFAULT=

Sets the default boot entry. The boot entries are counted from 0 in grub.cfg, but not numbered. How do you know what numbers each boot option has? There is no obvious way to figure this out; you have to count your boot options manually. Count the “menuentry” sections to figure out the numbering. A menu entry looks like this:

menuentry 'openSUSE Leap 15.2'  --class opensuse --class gnu-linux
  --class gnu --class os
menuentry_id_option 'gnulinux-simple-102a6fce-8985-4896-a5f9-e5980cb21fdb' {
        load_video
        set gfxpayload=keep
        insmod gzio
        [...]

Or use the awk command to list them for you, like this example for Ubuntu 20.04:

$ sudo awk -F\' '/menuentry / {print i++,$2}' /boot/grub/grub.cfg
0 Ubuntu
1 Ubuntu, with Linux 5.8.0-53-generic
2 Ubuntu, with Linux 5.8.0-53-generic (recovery mode)
3 Ubuntu, with Linux 5.8.0-50-generic
4 Ubuntu, with Linux 5.8.0-50-generic (recovery mode)
5 UEFI Firmware Settings

You probably don’t want to use a recovery mode entry as the default, or a memory test, though it doesn’t hurt anything if you do. UEFI Firmware Settings is a shortcut to your system’s BIOS/UEFI.

GRUB_TIMEOUT=10

Sets the number of seconds the GRUB menu waits before booting the default, and GRUB_TIMEOUT_STYLE=menu displays the menu during the countdown. GRUB_TIMEOUT=0 boots immediately without displaying the menu, and GRUB_TIMEOUT=-1 disables automatic booting and waits for the user to select a boot entry.

GRUB_DEFAULT=saved

Together with GRUB_SAVEDEFAULT=true, GRUB_DEFAULT=saved makes the last menu entry that you booted the default for the next boot.

GRUB_CMDLINE_LINUX=

Adds Linux kernel options for all menu entries.

GRUB_CMDLINE_LINUX_DEFAULT=

Passes kernel options only to the default menu entries. G⁠R⁠U⁠B⁠_⁠C⁠M⁠D⁠L⁠I⁠N⁠E⁠_⁠L⁠I⁠N​U⁠X⁠_⁠D⁠E⁠F⁠A⁠U⁠L⁠T⁠=⁠"⁠q⁠u⁠i⁠e⁠t⁠ ⁠s⁠p⁠l⁠a⁠s⁠h⁠" is a common default option that disables the verbose output at startup and displays a graphical splash screen. Figure 2-5 shows what the verbose output looks like. If you have G⁠R⁠U⁠B⁠_⁠C⁠M⁠D⁠L⁠I⁠N⁠E⁠_⁠L⁠I⁠N⁠U⁠X⁠_⁠D⁠E⁠F​A⁠U⁠L⁠T⁠=⁠"⁠q⁠u⁠i⁠e⁠t⁠ ⁠s⁠p⁠l⁠a⁠s⁠h⁠" configured, you can see this output without changing your configuration by pressing the Esc key during startup.

GRUB_TERMINAL=gfxterm

Sets the your GRUB screen to a graphical mode that supports colors and images. GRUB_TERMINAL=console disables graphical mode.

GRUB_GFXMODE=

Sets the screen resolution for the graphical mode, for example, GRUB_GFXMODE=1024x768. Run the set pager=1 command, and then run videoinfo from your GRUB command line to see your supported modes (Figure 2-6). set pager=1 lets you use the arrow keys to page up and down long command output. GRUB_GFXMODE=auto calculates a reasonable default.

Figure 2-5. Startup messages

Figure 2-6. Supported video modes

GRUB_BACKGROUND=

Sets a background image on the GRUB menu, using the image of your choice (see Recipe 2.6).

GRUB_THEME=

Decorates your GRUB menu with a complete theme (see Recipe 2.8).

See Also

• Recipe 2.6

• Recipe 2.8

• GNU GRUB Manual

• GRUB has multiple single-purpose man pages; run man -k grub to see all of them

• info grub or info grub2

Problem

You don’t care for the appearance of your GRUB menu, and you want to pretty it up.

Solution

You need an image in PNG, 8-bit JPG, or TFA format. It can be any size, and GRUB will scale it to fit. In the following example, a photo of Duchess enjoying the bookshelves graces our GRUB menu.

Copy your image to /boot/grub/, and add the full filepath of your image to /etc/default/grub. The photo of Duchess is /boot/grub/duchess-books.jpg:

GRUB_BACKGROUND="/boot/grub/duchess-books.jpg"

If there is a GRUB_THEME= line, make sure it is commented out, then rebuild your GRUB configuration (Recipe 2.1).

You should see a line like “Found background: /boot/grub/duchess-books.jpg” in the output of your rebuild command. If you do not see this, there is an error in your configuration.

When it looks correct, rebuild, reboot, and enjoy your new GRUB menu background (Figure 2-7).

Figure 2-7. Duchess the literacy cat gracing the GRUB menu

The fonts in the example are barely readable, so jump ahead to Recipe 2.7 to learn how to change their colors.

Discussion

You may use any image on your system; it does not have to be in /boot/grub/. Putting your image files in /boot/grub/ keeps all your GRUB customizations in one place and makes them available to all installed Linux systems on a multiboot setup.

See Also

• GNU GRUB Manual

• GRUB has multiple single-purpose man pages; run man -k grub to see all of them

• info grub or info grub2

Problem

Your new background is lovely (Figure 2-7), but your fonts are barely visible, and you need to change the colors so you can read your GRUB menu.

Solution

This is fun because you can quickly preview colors from the GRUB command line. Then, when you know what colors you want, edit /etc/default/grub and create a new file in /etc/grub.d/ to load your colors, then rebuild /boot/grub/grub/cfg. Reboot to enjoy your background image with pretty colored fonts.

Start up your computer, and when the GRUB menu appears, press C to open the GRUB command line (Figure 2-8).

Figure 2-8. The GRUB command line

The following two commands set the colors in Figure 2-9:

grub> menu_color_highlight=cyan/blue
grub> menu_color_normal=yellow/black

Figure 2-9. Setting GRUB menu colors from the GRUB command line

You may set and test each color pair one pair at a time. From the GRUB menu, press C to open the GRUB command shell. Type your command (sorry, no copy-paste), press Enter, then press Esc to return to the menu to see how it looks. You can list your previous commands with the up and down arrow keys, and edit and reuse them rather than retyping everything.

You must specify two colors in this order: foreground/background. All the colors are solid, with no transparency, with one exception: when you select black as a background color it is transparent. That is why menu_color_normal= must have black as the background color, when you have a background image. If you use any other color, your image will be covered by the background color. The menu_color_highlight= background color only applies to whatever line is currently selected.

When you figure out your colors, make them permanent. Boot up and create a new script in /etc/grub.d/. In the following example, it is called 07_font_colors. Copy this exactly:

#!/bin/sh

        if [ "x${GRUB_BACKGROUND}" != "x" ] ; then
                if [ "x${GRUB_COLOR_NORMAL}" != "x" ] ; then
                echo "set color_normal=${GRUB_COLOR_NORMAL}"
                fi

                if [ "x${GRUB_COLOR_HIGHLIGHT}" != "x" ] ; then
                echo "set color_highlight=${GRUB_COLOR_HIGHLIGHT}"
                fi
        fi

Then make it executable:

$ sudo chown +x 07_font_colors

Now add these lines to your /etc/default/grub file, using your colors:

export GRUB_COLOR_NORMAL="yellow/black"
export GRUB_COLOR_HIGHLIGHT="cyan/blue"

Rebuild your GRUB configuration (Recipe 2.1) and reboot to see if it worked.

Discussion

See Recipe 2.4 to learn about the files in /etc/grub.d/ and why the filenames must start with numbers. In my experience it doesn’t matter if your fonts script starts early or late, but if it doesn’t work for you, try changing the priority. Make sure it is executable. The options are as follows:

• menu_color_highlight controls the colors of the highlighted lines inside the menu box.

• menu_color_normal controls the colors of the not-highlighted lines.

Use these colors exactly as they are written in Table 2-1, all lowercase and with this exact spelling.

See Also

• GNU GRUB Manual

• GRUB has multiple single-purpose man pages; run man -k grub to see all of them

• info grub or info grub2

• Recipe 2.4

Problem

You like prettying up your GRUB menu, and you want to know if there are themes for the GRUB menu and how to install them.

Solution

You are in luck, for there are many themes for GRUB. Start by using your package manager with the grep command to search for package names. This example is for Ubuntu Linux:

$ apt search theme | grep grub

Using theme | grep grub to filter your results will find all relevant packages, like grub-theme-breeze, grub2-themes-ubuntu-mate, and grub-breeze-theme. Install themes just as you would any package.

Discussion

Your new theme should be installed in /boot/grub/themes. Find your new theme, for example /boot/grub/themes/ubuntu-mate, and look for the theme.txt file. Enter the full path in /etc/default/grub, like this example for the ubuntu-mate theme:

GRUB_THEME=/boot/grub/themes/ubuntu-mate/theme.txt

Be sure to comment out any other configuration lines pertaining to appearance that you may have, such as GRUB_BACKGROUND=, any custom font colors, and any other themes. Then rebuild your GRUB configuration (Recipe 2.1). You should see a line like “Found theme: /boot/grub/themes/ubuntu-mate/theme.txt” in your command output.

If all goes well, reboot and you will be rewarded with a screen like Figure 2-10. If it does not display correctly, recheck your configuration and commands.

Figure 2-10. The Ubuntu MATE GRUB theme

See Also

• GNOME Themes

• KDE Themes

• GNU GRUB Manual

Problem

When you start your system, it stops at a GRUB prompt, grub>, and does not boot. You need to know how to boot your system and then repair your configuration.

Solution

When the boot process stops at the grub> prompt (Figure 2-11), this means it found /boot/grub/ but cannot find the root filesystem.

Figure 2-11. The GRUB command shell

You need to find the root filesystem, the Linux kernel, and its matching initrd file. When you are in the GRUB command shell, the entire filesystem is open to you.

The first command you should run is to invoke the pager, so that you can page up and down long output:

grub> set pager=1

List your disks and partitions. GRUB has its own way of identifying hard disks and partitions. It numbers disks from 0, partitions from 1, and labels all hard disks as hd. On a running Linux system, hard disks are identified as /dev/sda, /dev/sdb, and so on.

In the following example, GRUB lists two hard disks, hd0 and hd1, which are the same as /dev/sda and /dev/sdb. hd0,gpt5 is the same as /dev/sda5, and hd1,msdos1 is the same as /dev/sdb1:

grub> ls
(hd,0) (hd0,gpt5) (hd0,gpt4) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1)
(hd1) (hd1,msdos1)

This output shows that hd0 has a gpt partition table, and hd1 has the old-fashioned msdos partition table. It is not necessary to use the gpt and msdos labels when you are listing partitions and files.

GRUB tells you the filesystem types, universally unique identifiers (UUIDs), and other information on the partitions:

grub> ls (hd0,3)
  Partition hd0,3: filesystem type ext* - Last modification time 2021-12-29
  01:17:58 Tuesday, UUID 5c44d8b2-e34a-4464-8fa8-222363cd1aff - Partition start
  at 526336KiB -
Total size 20444160KiB

You need to find /boot. Suppose you remember that it is in the root filesystem on the second partition; start looking there. A forward slash following the partition name means list all files and directories on the partition:

grub> ls (hd0,2)/
bin   dev  home  lib64   media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var

All boot files are in the /boot directory:

grub> ls (hd0,2)/boot
efi/ grub/ System.map-5.3.18-lp152.57-default config-5.3.18-lp152.57-default
initrd-5.3.18-lp152.57-default vmlinuz vmlinuz-5.3.18-lp152.57-default
sysctl.conf-5.3.18-lp152.57-default vmlinux-5.3.18-lp152.57-default.gz

Everything you need to boot your system is there. Set the root filesystem partition, kernel, and initrd image:

grub> set root=(hd0,2)
grub> linux /boot/vmlinuz-5.3.18-lp152.57-default root=/dev/sda2
grub> initrd /boot/initrd-5.3.18-lp152.57-default
grub> boot

If there are multiple vmlinuz and initrd files, use the two with the newest matching version numbers. If all the commands are correct, your system will boot and you can fix your GRUB configuration (Recipe 2.11).

Discussion

When /boot is in its own partition, you won’t see any other directories because it is not in the root filesystem.

vmlinuz-5.3.18-lp152.57-default is the compressed Linux kernel.

initrd-5.3.18-lp152.57-default is the initial ramdisk, a temporary root filesystem used only to start up your system.

Boot failures are caused by corrupted files; adding, removing, or moving hard disks; installing or removing operating systems; or repartitioning. If you can’t get to a GRUB prompt, see Chapter 19 to learn how to rescue your system with SystemRescue.

You can practice using the grub> shell by pressing C when your GRUB menu appears. This is safe because changes you make do not survive a reboot.

See Also

• GNU GRUB Manual

Problem

When you start your system, it stops at a GRUB prompt, grub rescue>, and does not boot. You need to know how to boot your system and then repair your configuration.

Solution

The grub rescue> prompt (Figure 2-12) is the rescue shell, which means GRUB could not find /boot. Not to worry, you can find it from the GRUB prompt, boot your system, and then make a permanent fix.

Figure 2-12. The GRUB rescue shell

List your partitions:

grub rescue> ls
(hd0) (hd0,gpt5) (hd0,gpt4) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1)
(hd1) (hd1, msdos1)

At this point there is no tab-completion or paging, so you have to type everything.

GRUB tells you the filesystem types, UUIDs, and other information on the partitions:

grub rescue> ls (hd0,3)
    Partition hd0,3: filesystem type ext* - Last modification time 2021-12-29
    01:17:58
Tuesday, UUID 5c44d8b2-e34a-4464-8fa8-222363cd1aff - Partition start at
526336KiB -
Total size 20444160KiB

If you don’t know which partition contains /boot, you will have to list the files and directories in each one until you find it. You do not have to use the gpt and msdos labels. A forward slash following the device name means list all files and directories:

grub rescue> ls (hd0,2)/
bin   dev  home  lib64   media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var

Hurrah, there it is in the root filesystem. List the files in /boot:

grub rescue> ls (hd0,2)/boot
efi/ grub/ System.map-5.3.18-lp152.57-default config-5.3.18-lp152.57-default
initrd-5.3.18-lp152.57-default vmlinuz vmlinuz-5.3.18-lp152.57-default
sysctl.conf-5.3.18-lp152.57-default vmlinux-5.3.18-lp152.57-default.gz

There are a few additional commands for grub rescue>. You have to tell it where /boot/grub is, and then load the normal and linux kernel modules, which are in /boot/grub/i386-pc (along with many other kernel modules that GRUB uses at startup). normal changes the boot mode from rescue to normal, and linux starts the system loader:

grub rescue> set prefix=(hd0,2)/boot/grub
grub rescue> set root=(hd0,2)
grub rescue> insmod normal
grub rescue> insmod linux

After loading normal and linux, you have tab completion. You could also turn on paging, set pager=1, to enable using the arrow keys to scroll to your previous commands. Now tell GRUB where to find the kernel and initrd file:

grub> linux /boot/vmlinuz-5.3.18-lp152.57-default root=/dev/sda2
grub> initrd /boot/initrd-5.3.18-lp152.57-default
grub> boot

If there are multiple vmlinuz and initrd files, use the two with the newest matching version numbers. If all the commands are correct, your system will boot and you can fix your GRUB configuration (Recipe 2.11).

Discussion

When /boot is on its own partition, you won’t see any other directories because it is in its own filesystem.

See Also

• GNU GRUB Manual

Problem

You were able to boot your system from the GRUB prompt, and now you need to know how to make a permanent repair.

Solution

Check your GRUB configuration carefully for errors. When it looks correct, rebuild your GRUB configuration (Recipe 2.1). Then you need to reinstall GRUB. In the following example, it is reinstalled to /dev/sda:

$ sudo grub-mkconfig -o /boot/grub/grub.cfg
$ sudo grub-install /dev/sda

Be sure to install it to the correct disk, if you have more than one, and use only the device name (for example, /dev/sda ) and not a partition (such as /dev/sda1).

Discussion

Be sure to have good current backups. If your rescue efforts fail, try reinstalling GRUB from SystemRescue (Recipe 19.9).

See Also

• GNU GRUB Manual

• Chapter 19

Problem

You want to use the systemctl commands to shut down and reboot your system.

Solution

Halt the system and power off the machine:

$ systemctl poweroff

Another way to halt the system and power off the machine:

$ systemctl shutdown

Reboot:

$ systemctl reboot

Halt the system without powering off the machine:

$ systemctl halt

Discussion

The systemctl shutdown commands do not have the many options that the legacy commands do, which does not matter all that much because there are a lot of redundant options in the legacy commands. There is one significant difference: systemctl shutdown lacks the timed shutdown options supported by the shutdown command (see Recipe 3.2).

See Also

• man 8 systemd-halt.service

Problem

You want to use timed shutdowns, for example, 10 minutes from now or at a specific time, and warn all logged-in users. Or you want to just shut down now with no frills.

Solution

The shutdown command works the same way, whether it is symlinked to systemctl or the legacy shutdown executable.

The following examples show how to shut down immediately, shut down a certain number of minutes from now, cancel a shut down, shutdown at a specific time, halt, and reboot.

Shut down immediately, with no notification to other logged-in users:

$ shutdown -h now

Shut down in 10 minutes with notifications:

$ shutdown -h +10
Shutdown scheduled for Sun 2021-05-23 11:04:43 PDT, use 'shutdown -c' to cancel.

Other users on the system may see this message, depending on which Linux they are using, and if they have a terminal open:

Broadcast message from duchess@client4 on pts/4 (Sun 2021-05-24 10:54:43 PDT):

The system is going down for poweroff at Sun 2021-05-24 11:04:43 PDT!

Cancel the shutdown:

$ shutdown -c

Logged-in users may see this message:

Broadcast message from duchess@client4 on pts/4 (Sun 2021-05-24 10:56:00 PDT):

The system shutdown has been cancelled

Create your own message:

$ shutdown -h +6 "Time to stop working and go outside to play!"

Rather than specifying minutes to shutdown, you may set a shutdown time in 24-hour hh:mm format. The following example shuts down the system at 10:15 P.M.:

$ shutdown -h 22:15

Reboot:

$ shutdown -r

Halt the system without powering off:

$ shutdown -H

Running shutdown with no options is equivalent to shutdown -h +1.

Discussion

shutdown sends messages only when you use the -h option, except for the -h now option, and when you use the -k option. These are called wall messages, which is short for “write to all logged-in users.” Linux distributions vary in their support of this feature, so other users on your particular Linux may not see these messages.

• --help displays a summary of options.

• -H, --halt performs a clean shutdown but does not power off the machine; you must press and hold the power button to power off your machine.

• -P, --poweroff performs a clean shutdown and powers off the machine.

• -r, --reboot performs a clean shutdown and reboots the machine.

• -k sends a wall message without shutting down the machine.

• --no-wall disables wall messages.

See Also

• man 8 shutdown

• man 1 wall

• man 8 systemd-halt.service

Problem

You understand the shutdown command, and now you want to know what halt, reboot, and poweroff are for, and how to use them.

Solution

These are all pretty much the same.

halt performs a clean shutdown, stopping all services and processes and unmounting filesystems, but it does not power off the machine. After the halt command is finished, you must press and hold the machine’s power button to complete the shutdown.

reboot performs a clean shutdown and restarts the system.

poweroff performs a clean shutdown and powers off the machine:

$ halt
$ reboot
$ poweroff

The halt and poweroff commands can reboot the system:

$ halt --reboot
$ poweroff --reboot

Discussion

If you’re thinking this all looks a little weird and redundant, you’re right. As software ages, cruft accumulates and never goes away. Linux has been around since 1991, and started out as a free clone of Unix, which was born in 1969. That is a lot of years for the various contributors to tweak code and add their own favorite features.

halt and poweroff are the same commands and support the same options:

• --help displays a summary of options.

• --halt performs a clean shutdown but does not power off the machine (yes, halt and halt --halt do the same thing).

• -p, --poweroff performs a clean shutdown and powers off the machine (yes, poweroff and poweroff --poweroff do the same thing) .

• --reboot performs a clean shutdown and restarts the machine.

• -f, --force forces an immediate halt or poweroff. Shutdown of all running services is skipped, all processes are killed, and all filesystems are unmounted or mounted read-only. Run it twice to force an unclean shutdown; for example, poweroff -f -f, which you should use only when normal shutdown commands fail.

• -w, --wtmp-only does not perform a shutdown, but only writes an entry in /var/log/wtmp.

• -d, --no-wtmp prevents writing a wtmp entry.

See Also

• man 8 halt

• man 8 poweroff

• man 8 systemd-halt.service

Problem

Your Linux system has systemd, and you want to use systemctl to manage system sleep modes.

Solution

systemctl provides these power saving modes: suspend, hibernate, hybrid-sleep, and suspend-then-hibernate.

Put your system into suspend mode:

$ systemctl suspend

This stores your current session in RAM and puts all hardware into a suspended state. Wake your system up by pressing any key, moving the mouse, or opening your laptop lid.

Put your system into hibernate mode:

$ systemctl hibernate

This stores your session on disk and powers off the machine. Wake it up by pressing the power button, and when it resumes, which can take a minute or two, your session will be restored where you left off.

Put your system into hybrid-sleep mode:

$ systemctl hybrid-sleep

This suspends your system to both memory and disk, and shuts off all devices except RAM. If your system RAM loses power, the system resumes from disk. Wake it up by pressing the power button.

Put your system into suspend-then-hibernate mode:

$ systemctl suspend-then-hibernate

suspend-then-hibernate first goes into suspend mode, then enters hibernation after the period of time specified by the HibernateDelaySec= setting in /etc/systemd/sleep.conf. Wake it up by pressing the power button.

Discussion

See the Discussion in Recipe 3.9 for details on the different sleep states.

Your graphical desktop should have buttons for entering power saving modes and a graphical configuration tool for controlling events such as screen blanking, screen locking, power button, mouse, and laptop lid actions such as entering a sleep state or powering off.

Power management tends to work best on laptops and may not behave as expected on your Linux distribution. Power management is affected by your UEFI, CPU capabilities, udev, Advanced Configuration and Power Interface (ACPI), kernel compilation options, and possibly other devices and programs; so much depends on how your particular Linux has implemented power management. Check your Linux distribution documentation.

See Also

• man 1 systemctl

• man 8 systemd-halt.service

Problem

You want a reliable method of rebooting that always works, even when you are having problems such as crashes and runaway processes.

Solution

The good old “three-key salute,” Ctrl-Alt-Delete, was made for this. Press and hold these three keys in sequence, and they will override most problems and reboot your system. Ctrl-Alt-Delete is disabled in some Linux distributions, and you can change this.

Ctrl-Alt-Delete is controlled by systemd in the Linux console. See Recipe 3.6 to learn about managing Ctrl-Alt-Delete in systemd.

For systems without systemd, see the Discussion in this recipe.

Graphical environments have their own configuration tools for Ctrl-Alt-Delete, independent of systemd. For example, there is a keyboard configuration module in the Xfce4 Settings Manager (Figure 3-1); in GNOME, use the Keyboard Settings module in the GNOME Settings utility.

Figure 3-1. Settings → Keyboard → Applications, configuring keyboard shortcuts in Xubuntu

If you prefer a different key combination, use any keys you want. You could even use a single key, though that risks rebooting with an accidental key press.

Discussion

On Linux systems that do not use systemd, Ctrl-Alt-Delete is controlled by the /etc/inittab file. This example, from MX Linux, shows a typical configuration:

# What to do when CTRL-ALT-DEL is pressed.
ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

12345 makes it active in runlevels 1, 2, 3, 4, and 5. -t1 means wait one second, -a calls /etc/shutdown.allow, and -r is reboot. Configure it to power off the system with the -h option, just like running shutdown from the command line (see Recipe 3.2). To disable Ctrl-Alt-Delete, comment out the line with the shutdown command.

Note that -t1 and -a are not present in all shutdown command implementations. The preceding example is from MX Linux. MX Linux supports both the Unix System V initialization system (SysV init) and systemd, and you select the one you want to use from the boot menu.

Ctrl-Alt-Delete is coded into the IBM PC BIOS/UEFI, and it should always reboot a system before the operating system is launched, up to the moment before GRUB launches the operating system.

Ctrl-Alt-Delete was created by IBM engineer David Bradley for the IBM PC BIOS. Orignally it was a developer tool and not intended for users. It required two hands by design, to make it difficult to press by accident.

Then Microsoft adopted it to bring up the Task Manager on the first press, and to reboot on the second press. Then in Windows NT it was used to access the Windows login screen. Supposedly this was a security measure that prevented users from being deceived by fake login screens, which I never knew was a risk, but that was a long time ago. There is a funny exchange about the invention and use of Ctrl-Alt-Delete between Mr. Bradley and Bill Gates preserved in a YouTube video, which hopefully will stay up forever: “Control-Alt-Delete: David Bradley & Bill Gates”.

See Also

• man 7 systemd.special

Problem

systemd controls the behavior of Ctrl-Alt-Delete in the Linux console, and you want to know how to manage it.

Solution

You can check the status, disable or enable Ctrl-Alt-Delete, or change it to power off the system.

The Ctrl-Alt-Delete unit file is not a service, but a target, so it does not run as a daemon. If the /etc/systemd/system/ctrl-alt-del.target symlink exists, Ctrl-Alt-Delete is enabled.

The following example disables and masks ctrl-alt-del.target:

$ sudo systemctl disable ctrl-alt-del.target
Removed /etc/systemd/system/ctrl-alt-del.target.

$ sudo systemctl mask ctrl-alt-del.target
Created symlink /etc/systemd/system/ctrl-alt-del.target → /dev/null.

Unmask and re-enable it:

$ sudo systemctl unmask ctrl-alt-del.target
Removed /etc/systemd/system/ctrl-alt-del.target.

$ sudo systemctl enable ctrl-alt-del.target
Created symlink /etc/systemd/system/ctrl-alt-del.target →
/lib/systemd/system/reboot.target.

The changes take effect immediately.

Change it to power off the system by linking the ctrl-alt-del.target unit to the poweroff.target unit. First, disable it to remove the existing symlink, then create the new symlink:

$ sudo systemctl disable ctrl-alt-del.target
Removed /etc/systemd/system/ctrl-alt-del.target.

$ sudo ln -s /lib/systemd/system/poweroff.target \
  /etc/systemd/system/ctrl-alt-del.target

Now it will power off the system instead of rebooting.

Discussion

Use the stat command to see symlinks:

$ stat /lib/systemd/system/ctrl-alt-del.target
  File: /lib/systemd/system/ctrl-alt-del.target -> reboot.target
  Size: 13              Blocks: 0          IO Block: 4096   symbolic link
Device: 802h/2050d      Inode: 136890      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 0/ root)   Gid: ( 0/ root)

You should not change any /lib/systemd/system/ links. Instead, create the new symlink in /etc/systemd/system/ so that your change is not overwritten by system updates.

See Also

• man 7 systemd.special

Problem

You want your machine to turn itself off at night so you can walk away and not worry about it. Or, your users are careless watt wasters who refuse to develop the habit of shutting their PCs down at night.

Solution

Use cron to create scheduled shutdowns. For example, add this line to /etc/crontab to shut down every night at 10:30 P.M., with a 20-minute warning. Editing /etc/crontab requires root permissions, and the following example uses the nano text editor:

$ sudo nano /etc/crontab
#  m   h   dom mon dow   user    command
  10  22    *   *   *    root    /sbin/shutdown -h +20

This example runs only at 11 P.M. on weekdays and shuts down immediately:

#  m   h   dom mon dow   user    command
  00  23    *   *  1-5   root    /sbin/shutdown -h now

Another way is to use the crontab command as root or with sudo:

$ sudo crontab -e
# m   h     dom mon  dow command
  00  23    *   *    1-5 /sbin/shutdown -h now

There is no name field when you run crontab, as there is in /etc/crontab. The previous example opens the root user’s crontab in edit mode. Edit and save, then you’re done.

Don’t try to name the file yourself. During editing it is a temporary file, which is automatically renamed by crontab when you save it. It will be saved in /var/spool/cron/crontabs.

Discussion

/etc/crontab has a name field, so any user can have entries in this file, but only root can edit /etc/crontab. Users who want to control their own personal crontabs should use the crontab command, as personal crontabs do not require root permissions.

The fields in /etc/crontab can take a little getting used to (Table 3-1), so here are more examples and explanations.

The asterisk * is a wildcard for “all.”

Shut down only on weekends:

# shutdown at 1:05 am Saturdays and Sundays
00 01   * * 7,0   root /sbin/shutdown -h +5

There is a quirk in cron that Sunday is 0 or 7. This dates back to very olden times, and I have no idea why it persists. You’ll have to test it to see what works, so you may need to use 6,7 for Saturday, Sunday:

00 01   * * 6,7   root /sbin/shutdown -h +5

It would be nice to use sat,sun, but you can only enter one day by name, and cannot use names in lists. Days of the week and months are named with the first three letters: sat, sun, jan, feb. Case does not matter.

You can use ranges: 1–4 means 1, 2, 3, and 4.

Ranges and lists can be mixed: 1, 3, 5, 6-10.

Step values follow ranges:

• 10–23/2 is every second hour in the range

• */2 in the dow field means every other day

• 2–6/2 equals 2, 4, 6

The following strings are nice shortcuts that replace the first five fields:

@reboot
@yearly
@annually
@monthly
@weekly
@daily
@midnight
@hourly

See Also

• man 8 cron

• man 1 crontab

• man 5 crontab

Problem

Scheduled shutdowns are so wonderful, you want scheduled wake-ups as well.

Solution

You’re in luck, because Linux supports scheduled wake-ups. There are three methods to try: Wake-on-LAN, real-time clock (RTC) wake-ups, or your computer’s UEFI setup, if it has a scheduled wake-up feature.

UEFI wake-ups are the most reliable. Figure 3-2 shows the scheduled wake-up screen on a Lenovo ThinkPad.

Figure 3-2. Scheduling wake-ups in a Lenovo UEFI

Enter your UEFI setup by pressing the appropriate Fn key at startup. On Dell, ASUS, and Acer systems this is usually F2; Lenovo uses F1. This varies, however. For example, some systems use the Delete key, so check your machine’s documentation. Some systems tell which key to press on their startup screens. It can be a bit tricky to press the key at the right time, so start pressing it right after you press the power button, just like banging on an elevator button to make it arrive faster.

If your system does not have a UEFI wake-up feature, try either Wake-on-LAN (see Recipe 3.10, which requires a second device to send a wake-up signal), or RTC wake-ups (see Recipe 3.9).

Discussion

Let’s discuss briefly what BIOS and UEFI mean. When you boot up your computer, the first startup instructions come from the Basic Input Output System (BIOS), or the Unified Extensible Firmware Interface (UEFI) firmware stored on the computer’s motherboard. BIOS is the old legacy system that has been with us since 1980. UEFI is its modern replacement. UEFI includes legacy BIOS support, though someday this will be removed. Nearly all computers made after the mid-2000s have UEFI.

UEFI has considerably more features than the old BIOS, and is like a little operating system. The UEFI setup screens control boot order, boot devices, security options, Secure Boot, overclocking, displaying hardware health, networking, and many more functions.

See Also

• Recipe 3.9

• Recipe 3.10

• Recipe 3.11

Problem

You want to set up scheduled wake-ups with RTC because your UEFI setup does not have a scheduled wake-up feature, or because you just want to.

Solution

Use the rtcwake command, which should already be present on your system from the util-linux package. rtcwake stops and wakes your system. You may set it to wake up your system after a specified interval, such as 1800 seconds from now, or at a scheduled time and date.

Your system’s real-time clock (RTC) should be set to Coordinated Universal Time (UTC).

When rtcwake stops your system, it sends it into an ACPI sleep state. Look in /sys/power/state to see which sleep states your system supports. In the following example, only three of the six ACPI sleep states are supported:

$ cat /sys/power/state
freeze mem disk

On non-systemd Linuxes, run cat /proc/acpi/info.

In the above example, we have three sleep states to try. Test each one according to the following example:

$ sudo rtcwake -m freeze -s 60

-m specifies the standby mode, and -s is the number of seconds until the system starts up again. When it is successful, you will see your system go into a sleep state, then wake up, and you will see either a success message or an error message.

The following example does a dry run of scheduling a wake-up tomorrow at 8 A.M.:

$ sudo rtcwake -n -m disk no -u -t $(date +%s -d "tomorrow 08:00")
rtcwake: wakeup from "disk" using /dev/rtc0 at Mon Nov 23 08:00:00 2021

Remove -n to disable the dry run and run it for real.

The following is a nice, simple /etc/crontab example to automate shutdowns and wake-ups. The rtcwake command suspends to disk at 11 P.M. on weeknights and wakes up 8 hours later:

#  m   h   dom mon dow   user    command
  00  23    *   *  1-5   root    /usr/sbin/rtcwake -m disk -s 28800

Discussion

Look in your system’s BIOS/UEFI to verify that your hardware clock is set to UTC. If there is no button or some kind of setting to change the real-time clock to UTC, change the time manually to the current UTC time.

The example that includes -u -t +$(date +%s -d “tomorrow 08:00”) converts Unix epoch time to human-readable values. Unix epoch time is the number of seconds elapsed since midnight January 1, 1970 UTC. date +%s reports the current Unix epoch time. The -t option passes Unix epoch time to date for conversion, and -u specifies that your hardware clock is set to UTC.

The no option for rtcwake means do not go into a sleep state, but only set the wake-up time. Remove the no option to immediately go into a sleep state.

Real-time clock (RTC) wake-ups are the least reliable. Your system has to be put into an Advanced Configuration and Power Interface (ACPI) sleep state supported by your Linux. ACPI is the modern power management standard for managing sleep states. It is supposed to be vendor neutral and hardware independent, but the standard is complex and hardware vendors often support only a subset of its capabilities. Adding to the fun, the various Linuxes implement it in different ways.

There are six ACPI sleep states, S0-S5. The Linux kernel is capable of supporting up to four, and distributions vary in which ones and how many:

S0

System is running, monitor may be off, most peripherals are on.

S1

Power-on suspended, CPU stops, power to CPU and RAM is on.

S2

CPU is powered off, dirty cache is flushed to RAM.

S3

Also called standby, sleep, and suspend-to-RAM. Data may not be written to disk.

S4

Hibernation, suspend-to-disk. Everything in RAM is written to disk and the system is powered down.

S5

Similar to powering the system off, except there is still power to the power button and peripherals, such as the keyboard, network interface, and USB devices.

See Also

• man 8 cron

• man 8 rtcwake

• Time and Date

Problem

You want to set up remotely triggered wake-ups with Wake-on-LAN because your UEFI setup does not have a scheduled wake-up feature, or it is too simple for your needs, or you want the flexibility to send a wake-up signal at random times. This recipe is for waking up a machine on the same network as the device you use to send the wake-up signal, and your target machine must use a wired Ethernet interface.

Solution

Configure your PC to listen for wake-up requests, then use a second device, such as another computer, a smartphone, or a Raspberry Pi to send the wake-up signal, which is called the magic packet. Which isn’t really magic, but merely a specialized packet designed specifically for remote wake-ups. (Yes, I too am sad it is not real magic.)

Start by booting into your system’s UEFI setup and look for settings for enabling Wake-on-LAN.

Then exit and finish starting up. Install the wakeonlan and ethtool packages.

Fetch the name of your Ethernet interface, which in this example is enp0s25, and use ethtool to verify that it supports Wake-on-LAN. The output is abbreviated for clarity:

$ ip addr show
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> state UP 0
    link/ether 9c:ef:d5:fe:8f:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.97/24 brd 192.168.1.255 scope global dynamic
    [...]

$ sudo ethtool enp0s25 | grep -i wake-on
        Supports Wake-on: pumbg
        Wake-on: g 

Record the MAC address of your interface. In the preceding ip example, the “ether” line lists the MAC address, 9c:ef:d5:fe:8f:20. Yours will be different, as MAC addresses are unique.

“Supports Wake-on: pumbg” is the magic phrase that verifies your interface has the necessary support, indicated by the g switch. The second line, “Wake-on: g,” tells you that it is already enabled. If it is not, enable it:

$ sudo ethtool -s enp0s25 wol g

If it does not stay enabled after a restart, add an entry like this to /etc/crontab to run the command after every restart:

$ @reboot root /usr/bin/ethtool -s enp0s25 wol g

Shut down the machine, and from a second device on the same network, send the command to wake it up, using the MAC address of your target machine’s Ethernet interface:

$ /usr/bin/wakeonlan 9c:ef:d5:fe:8f:20

If your target machine and second device are on the same network but in different subnets, specify the broadcast address for your target machine:

$ /usr/bin/wakeonlan -i 192.168.44.255 9c:ef:d5:fe:8f:20

Discussion

Wake-on-LAN is an Ethernet standard for remotely waking up a computer by sending it a wake-up signal over a network. wakeonlan is the name of the command and the name of the package on most Linuxes.

When your computer is shut down, it isn’t really turned off, but is in a low power mode, and can receive and act on the magic packet wake-up signal.

wakeonlan sends the magic packet over UDP port 9. The magic packet goes to the network’s broadcast address, and every host on the network receives this packet. The MAC address ensures that only the host with that address will wake up.

The target machine wakes up just as if you had pressed the power button.

See Also

• man 1 wakeonlan

• man 8 ethtool

• Recipe 3.8

• Recipe 3.9

• Recipe 3.11

Problem

Your want to wake up a remote computer via its wireless interface (Wake-on-Wireless LAN, or WoWLAN).

Solution

This recipe is for waking up a machine on the same network as the device you use to send the wake-up signal.

Your machine must have an onboard wireless interface, either integrated on the motherboard or peripheral component interconnect (PCI). It will not work with a USB interface because there is no power to the USB bus when the machine is shut down.

First, enter the UEFI setup of the machine you want to wake up remotely and enable any Wake-on-LAN settings.

Then exit and finish starting up. Install the iw command, and use it to list all of your wireless devices:

$ iw dev
phy#0
        Interface wlxcc3fd5fe014c
                ifindex 3
                wdev 0x1
                addr 9c:bf:25:fe:0e:7c
                ssid accesspointe
                type managed
                channel 11 (2462 MHz), width: 20 MHz, center1: 2462 MHz
                txpower 20.00 dBm 

If there is more than one, query the one you want to use. The following example shows a wireless interface that does not support WoWLAN:

$ iw phy0 wowlan show
command failed: Operation not supported (-95)

The following example is for an interface that supports WoWLAN, and it is not enabled:

$ iw phy0 wowlan show
WoWLAN is disabled

Enable WoWLAN:

$ sudo iw phy0 wowlan enable magic-packet
WoWLAN is enabled:
  * wake up on magic packet

iw dev provides the MAC address, which your second device needs to send the magic packet:

$ /usr/bin/wakeonlan 9c:bf:25:fe:0e:7c

Send the magic packet to a machine on the same network, but in a different subnet, using the broadcast address of the subnet:

$ /usr/bin/wakeonlan -i 192.168.44.255 9c:bf:25:fe:0e:7c

Discussion

This works well when the remote computer is a laptop, as laptops have integrated wireless network interfaces that typically support more features. Desktop machines usually don’t ship with integrated wireless interfaces, so you must shop carefully for a PCI/PCIe wireless adapter that supports WoWLAN and Linux.

See Also

• man 8 iw

• man 1 wakeonlan

• Recipe 3.8

• Recipe 3.9

• Recipe 3.10

Problem

You need to know if your Linux distribution uses systemd or something else.

Solution

Look for the /run/systemd/system/ directory. If this exists, then your init system is systemd.

Discussion

The /run/systemd/ directory may be present on your system if your distribution supports multiple init systems. But systemd is not the active init unless you see /run/systemd/system/.

There are several other ways to learn which init system your system is using. Try querying /sbin/init. Originally this was the SysV executable, and now most Linux distributions preserve the name and symlink it to the systemd executable. This example confirms that the init is systemd:

$ stat /sbin/init
File: /sbin/init -> /lib/systemd/systemd
[...]

On a system using SysV init, it has no symlink:

$ stat /sbin/init
File: /sbin/init
[...]

The /proc pseudofilesystem is an interface to your Linux kernel, and contains the current state of a running system. It is called a pseudofilesystem because it exists only in memory and not on disk. In this example, /proc/1/exe is symlinked to the systemd executable:

$ sudo stat /proc/1/exe
File: /proc/1/exe -> /lib/systemd/systemd
[...]

On a SysV system, it links to init:

$ sudo stat /proc/1/exe
File: /proc/1/exe -> /sbin/init
[...]

The /proc/1/comm file reports your active init system:

$ cat /proc/1/comm
systemd

On a SysV system, it reports init:

$ cat /proc/1/comm
init

The command attached to process ID (PID) 1 is your init. PID 1 is the first process launched at startup, which then starts all other processes. You can see this with the ps command:

$ ps -p 1
  PID TTY          TIME CMD
    1 ?        00:00:00 systemd

When the init is SysV, it looks like this:

$ ps -p 1
  PID TTY          TIME CMD
    1 ?        00:00:00 init

See Recipe 4.2 for more information on PID 1.

Linux support for systemd varies. Most of the major Linux distributions have adoped systemd, including Fedora, Red Hat, CentOS, openSUSE, SUSE Linux Enterprise, Debian, Ubuntu, Linux Mint, Arch, Manjaro, Elementary, and Mageia Linux.

Some popular distributions that do not support systemd, or include it but not as the default init, are Slackware, PCLinuxOS, Gentoo Linux, MX Linux, and antiX.

See Also

• Distrowatch for information on hundreds of Linux distributions

• man 5 proc

• man 1 pstree

• man 1 ps

Problem

You want a better understanding of services and processes on Linux.

Solution

PID 1 is the mother of all processes on Linux systems. This is the first process to start, and then it launches all other processes.

Processes are one or more running instances of a program. Every task in a Linux system is performed by a process. Processes can create independent copies of themselves, that is, they can fork. The forked copies are called children, and the original is the parent. Each child has its own unique PID, and its own allocation of system resources, such as CPU and memory. Threads are lightweight processes that run in parallel and share system resources with their parents.

Some processes run in the background and do not interact with users. Linux calls these processes services or daemons, and their names tend to end with the letter D, such as httpd, sshd, and systemd.

Every Linux system starts PID 1 first, which then launches all other processes. Use the ps command to list all running processes in PID order:

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 10:06 ?        00:00:01 /sbin/init splash
root         2     0  0 10:06 ?        00:00:00 [kthreadd]
root         3     2  0 10:06 ?        00:00:00 [rcu_gp]
root         4     2  0 10:06 ?        00:00:00 [rcu_par_gp]
[...]

The pstree command organizes this mass of information into a tree diagram. This example shows all processes, their child processes, PIDs, and threads, which are enclosed in curly braces:

$ pstree -p
systemd(1)─┬─ModemManager(925)─┬─{ModemManager}(944)
           │                   └─{ModemManager}(949)
           ├─NetworkManager(950)─┬─dhclient(1981)
           │                     ├─{NetworkManager}(989)
           │                     └─{NetworkManager}(991)
           ├─accounts-daemon(927)─┬─{accounts-daemon}(938)
           │                      └─{accounts-daemon}(948)
           ├─acpid(934)
           ├─agetty(1103)
           ├─avahi-daemon(953)───avahi-daemon(970)
[...]

The full pstree output is quite large. You can view a single process, identified by its PID, and its parents, children, and threads, like the following example for the Kate text editor:

$ pstree -sp 5193
systemd(1)───kate(5193)─┬─bash(5218)
                        ├─{kate}(5195)
                        ├─{kate}(5196)
                        ├─{kate}(5197)
                        ├─{kate}(5198)
                        ├─{kate}(5199)
[...]

This shows that systemd(1) is Kate’s parent, bash(5218) is Kate’s child, and all the processes in curly braces are Kate’s threads.

Discussion

Processes always exist in one of several states, and these states change according to system activity. The following pstree example displays the PID, user, state, and command fields:

$ ps -eo pid,user,stat,comm
  PID USER       STAT COMMAND
    1 root       Ss   systemd
    2 root       S    kthreadd
   32 root       I<   kworker/3:0H-kb
   68 root       SN   khugepaged
11222 duchess    Rl   konsole

• R is either currently running or waiting in the run queue.

• l means the process is multithreaded.

• S is interruptable sleep; the process is waiting for an event to complete.

• s is a session leader. Sessions are related processes managed as a unit.

• I is an idle kernel thread.

• < means high priority.

• N is low priority.

There are several rarely used states you can read about in man 1 ps.

See Also

• Recipe 4.7
• man 5 proc
• man 1 pstree
• man 1 ps

Problem

You want to list all services installed on your system, and you want to know the states of the services: whether they are running, not running, or in an error state.

Solution

systemctl, the systemd manager command, tells all. Run it with no options to see a detailed list of all loaded units. A systemd unit is any related batch of processes defined in a unit configuration file and managed by systemd:

$ systemctl

This prints a giant pile of information: 177 active loaded units on my test system with the full unit names, status, and long descriptions. Redirect the the output to a text file for easier study:

$ systemctl > /tmp/systemctl-units.txt

Treat yourself to more information overload by listing all units, active and inactive:

$ systemctl --all

This results in 349 loaded units listed on my test system, including not-found and inactive units. How many total unit files? The following example shows 5 out of 322:

$ systemctl list-unit-files
UNIT FILE                                      STATE
proc-sys-fs-binfmt_misc.automount              static
-.mount                                        generated
mount                                          generated
dev-hugepages.mount                            static
home.mount                                     generated
[...]
322 unit files listed.

We are interested in service files because Linux users and administrators interact mainly with service files and rarely need to bother with any other type of unit file. How many are installed? Let’s see:

$ systemctl list-unit-files --type=service
UNIT FILE                                  STATE
accounts-daemon.service                    enabled
acpid.service                              disabled
alsa-state.service                         static
alsa-utils.service                         masked
anacron.service                            enabled
[...]
212 unit files listed.

The preceding example displays the four most common states that a service can be in: enabled, disabled, static, or masked.

List only enabled services:

$ systemctl list-unit-files --type=service --state=enabled
UNIT FILE                                  STATE
accounts-daemon.service                    enabled
anacron.service                            enabled
apparmor.service                           enabled
autovt@.service                            enabled
avahi-daemon.service                       enabled
[...]
62 unit files listed.

List only disabled services:

$ systemctl list-unit-files --type=service --state=disabled
UNIT FILE                            STATE
acpid.service                        disabled
brltty.service                       disabled
console-getty.service                disabled
mariadb@.service                     disabled
[...]
12 unit files listed.

List only static services:

$ systemctl list-unit-files --type=service --state=static
UNIT FILE                              STATE
alsa-restore.service                   static
alsa-state.service                     static
apt-daily-upgrade.service              static
apt-daily.service                      static
[...]
106 unit files listed.

List only masked services:

$ systemctl list-unit-files --type=service --state=masked
UNIT FILE                    STATE
alsa-utils.service           masked
bootlogd.service             masked
bootlogs.service             masked
checkfs.service              masked
[...]
36 unit files listed.

Discussion

Service unit files are in /usr/lib/systemd/system/ or /lib/systemd/system/, according to where your Linux distribution puts them. These are plain-text files you can read.

enabled

This shows that the service has become available and is managed by systemd. When a service is enabled, systemd creates a symlink in /etc/systemd/system/ from the unit file in /lib/systemd/system/. It can be started, stopped, reloaded, and disabled by the user with the systemctl command.

Note

Enabling a service does not immediately start it, and disabling a service does not immediately stop it (see Recipe 4.6).

disabled

Diabled means that there is no symlink in /etc/systemd/system/, and it will not start automatically at boot. You can stop and start it manually.

masked

This means the service is linked to /dev/null/. It is completely disabled and cannot be started by any means.

static

This means that the unit file is a dependency of other unit files, and cannot be started or stopped by the user.

Some less-common service states you will see:

indirect

Indirect states belong to services that are not meant to be managed by users, but are meant to be used by other services.

generated

Generated states indicate that the service has been converted from a nonnative systemd initialization configuration file, either SysV or LSB init.

See Also

• man 1 systemctl

Problem

You want to know the status of one service or a few specific services.

Solution

systemctl status provides a nice little bundle of useful status information. The following example queries the CUPS service. CUPS, the Common Unix Printing System, should be on all Linux systems:

$ systemctl status cups.service
● cups.service - CUPS Scheduler
     Loaded: loaded (/lib/systemd/system/cups.service; enabled; vendor preset:
             enabled)
     Active: active (running) since Sun 2021-11-22 11:01:48 PST; 4h 17min ago
TriggeredBy: ● cups.path
             ● cups.socket
       Docs: man:cupsd(8)
   Main PID: 1403 (cupsd)
      Tasks: 2 (limit: 18760)
     Memory: 3.8M
     CGroup: /system.slice/cups.service
             ├─1403 /usr/sbin/cupsd -l
             └─1421 /usr/lib/cups/notifier/dbus dbus://

Nov 22 11:01:48 host1 systemd[1]: Started CUPS Scheduler.

Query multiple services with a space-delimited list:

$ systemctl status mariadb.service bluetooth.service lm-sensors.service

Discussion

There is a lot of useful information in this little bit of output (Figure 4-2).

Figure 4-2. systemctl status output for the CUPS printer service

The dot next to the service name is a quick status indicator. It appears in colors on most terminals. White is an inactive or deactivating state. Red is a failed or error state. Green indicates an active, reloading, or activating state. The rest of the information in the output is described in the following:

Loaded

Verifies that the unit file has been loaded into memory, displays its full path, the service is enabled (see the Discussion about states in Recipe 4.3), and vendor preset: disabled/enabled indicates if the installation default is to start at boot or not. When it is disabled, the vendor default is to not start at boot. This only shows the vendor preference and does not indicate if it is currently enabled or disabled.

Active

Tells you if the service is active or inactive, and how long it has been in that state.

Process

Reports the PIDs and their commands and daemons.

Main PID

This is the process number for the cgroup slice.

Tasks

Reports how many tasks the service has started. Tasks are PIDs.

CGroup

Shows which unit slice the service belongs to and its PID. The three default unit slices are user.slice, system.slice, and machine.slice.

Linux control groups (cgroups) are sets of related processes and all of their future children. In systemd, a slice is a subdivision of a cgroup, and each slice manages a particular group of processes. Run systemctl status to see a diagram of the cgroup hierarchy.

By default, service and scope units are grouped in /lib/systemd/system/system.slice.

User sessions are grouped in /lib/systemd/system/user.slice.

Virtual machines and containers registered with systemd are grouped in /lib/systemd/system/machine.slice.

The remaining lines are the most recent log entries from journalctl, the systemd log manager.

See Also

• man 1 systemctl

• man 5 systemd.slice

• man 1 journalctl

• Kernel cgroups documentation

Problem

You want to stop and start services with systemd.

Solution

This is a job for systemctl. The following examples use the SSH service to demonstrate service management.

Start a service:

$ sudo systemctl start sshd.service

Stop a service:

$ sudo systemctl stop sshd.service

Stop and then restart a service:

$ sudo systemctl restart sshd.service

Reload the service’s configuration. For example, you made a change to sshd_config and want to load the new configuration without restarting the service:

$ sudo systemctl reload sshd.service

Discussion

All of these commands also work with multiple services, space-delimited, for example:

$ sudo systemctl start sshd.service mariadb.service firewalld.service

If you’re curious about the commands that systemd runs behind the scenes to start, reload, or stop the individual daemons, look in their unit files. Some services have start, reload, stop, and other instructions in their unit files, like this example for httpd:

ExecStart=/usr/sbin/httpd/ $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

You don’t have to do anything special with this information; it is there when you want to know how systemctl is managing a particular service.

See Also

• Recipe 4.6

• man 1 systemctl

Problem

You want a service or services to automatically start at boot, or you want to prevent a service from starting at boot, or to disable it completely.

Solution

Enabling a service configures it to automatically start at boot.

Disabling a service stops it from starting at boot, but it can be started and stopped manually.

Masking a service disables it so that it cannot be started at all.

The following example enables the sshd service:

$ sudo systemctl enable sshd.service
Created symlink /etc/systemd/system/multi-user.target.wants/sshd.service →
/usr/lib/systemd/system/sshd.service

The output shows that enabling a service means creating a symlink from the service file in /lib/systemd/system/ to /etc/systemd/system/. This does not start the service. You can start the service with systemctl start, or enable and start the service in one command with the --now option:

$ sudo systemctl enable --now sshd.service

This command disables the sshd service. It does not stop the service, so you must stop it manually after disabling it:

$ sudo systemctl disable sshd.service
Removed /etc/systemd/system/multi-user.target.wants/sshd.service
$ sudo systemctl stop sshd.service

Or, disable and stop it with one command:

$ sudo systemctl disable --now sshd.service

This command reenables the mariadb service, which disables and then enables it. If you have created the symlinks manually for a service, this is useful for quickly resetting them to the defaults:

$ sudo systemctl reenable mariadb.service
Removed /etc/systemd/system/multi-user.target.wants/mariadb.service.
Removed /etc/systemd/system/mysqld.service.
Removed /etc/systemd/system/mysql.service.
Created symlink /etc/systemd/system/mysql.service →
/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/mysqld.service →
/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service →
/lib/systemd/system/mariadb.service.

The following command disables the bluetooth service completely by masking it, so that it cannot be started at all:

$ sudo systemctl mask bluetooth.service
Created symlink /etc/systemd/system/bluetooth.service → /dev/null.

Unmasking the bluetooth service does not enable it, so it must be started manually:

$ sudo systemctl unmask bluetooth.service
Removed /etc/systemd/system/bluetooth.service.
$ sudo systemctl start bluetooth.service

Discussion

When you enable, disable, mask, or unmask a service, it remains in its current state unless you use the --now option. The --now option works with enable, disable, and mask to immediately stop or start the service, but it does not work with unmask.

See the Discussion in Recipe 4.3 to learn more about how systemd uses symlinks to manage services.

See Also

• man 1 systemctl

• The Discussion in Recipe 4.3 to learn how systemd uses symlinks to manage services

Problem

You want to know how to stop troublesome processes. A certain service may be unresponsive or running away, spawning forks and causing your system to hang. Your normal stop command is not working. What do you do?

Solution

Stopping a process is called killing the process. On Linux systems with systemd, you should use systemctl kill. On systems without systemd, use the legacy kill command.

systemctl kill is preferable because it stops all processes that belong to a service and leaves no orphan processes, nor any processes that might restart the service and continue to make trouble. First, try it with no options other than the service name, then check the status:

$ sudo systemctl kill mariadb

$ systemctl status mariadb
● mariadb.service - MariaDB 10.1.44 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset:
enabled)
   Active: inactive (dead) since Sun 2020-06-28 19:57:49 PDT; 6s ago
[...]

The service has cleanly stopped. If this does not work, then try the nuclear option:

$ sudo systemctl kill -9 mariadb

The legacy kill command does not recognize service or command names, but rather requires the PID of the offending process:

$ sudo kill 1234

If this does not stop it, use the nuclear option:

$ sudo kill -9 1234

Discussion

Use the top command to identify runaway processes. Run it with no options, and the processes using up the most CPU resources are listed at the top. Press the q key to exit top.

$ top
top - 20:30:13 up  4:24,  6 users,  load average: 0.00, 0.03, 0.06
Tasks: 246 total,   1 running, 170 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.2 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16071016 total,  7295284 free,  1911276 used,  6864456 buff/cache
KiB Swap:  8928604 total,  8928604 free,        0 used. 13505600 avail Mem

  PID USER       PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3504 madmax     20   0 99.844g 177588  88712 S   2.6  1.1   0:08.68 evolution
 2081 madmax     20   0 3818636 517756 177744 S   0.7  3.2   5:07.56 firefox
 1064 root       20   0  567244 148432 125572 S   0.3  0.9  12:54.75 Xorg
 2362 stash      20   0 2997732 230508 145444 S   0.3  1.4   0:40.72 Web Content
[...]

kill sends signals to processes, and the default signal is SIGTERM (signal terminate). SIGTERM is gentle, allowing processes to shut down cleanly. SIGTERM is also ignorable, and processes don’t have to pay attention to it. Signals can be identified by name or number; for most folks the numbers are easier to remember, so spelling out the default looks like this:

$ sudo kill -1 1234

kill -9 is SIGKILL. SIGKILL stops processes immediately and uncleanly, and also attempts to stop all child processes.

Killing services with systemctl kill is easier than with kill, and more reliable. You only need the service name, and you don’t have to hunt down PIDs. It ensures that all processes belonging to the service are stopped, which kill cannot ensure.

There are a ton of signals that have accumulated over the years, and you can read all about them in man 7 signal. In my experience, the most relevant signals are SIGTERM and SIGKILL, but don’t let that stop you from learning more about the others.

If you are uncomfortable with terminology like kill, parents, children, and orphans, so am I. Maybe someday it will change.

See Also

• man 5 systemd.kill

• man 1 systemctl

• man 1 kill

• man 7 signal

Problem

You want to reboot to different system states in a manner similar to using SysV runlevels.

Solution

systemd targets are similar to SysV runlevels. These are boot profiles that start your system with different options, such as multiuser mode with a graphical desktop, multiuser mode with no graphical desktop, and emergency and rescue modes to use when your current target will not boot. (See the Discussion for more information on runlevels.)

The following command checks if the system is running and reports its state:

$ systemctl is-system-running
running

What is the default target?

$ systemctl get-default
graphical.target

Get the current runlevel:

$ runlevel
N 5

Reboot to rescue mode:

$ sudo systemctl rescue

Reboot to emergency mode:

$ sudo systemctl emergency

Reboot to the default mode:

$ sudo systemctl reboot

Reboot to a different target without changing the default:

$ sudo systemctl isolate multi-user.target

Set a different default runlevel:

$ sudo systemctl set-default multi-user.target

List the runlevel target files and their symlinks on your system:

$ ls -l /lib/systemd/system/runlevel*

List the dependencies in a runlevel target:

$ systemctl list-dependencies graphical.target

Discussion

SysV runlevels are different states that your system can boot to, for example, with a graphical desktop, without a graphical desktop, and with emergency runlevels to use when your default runlevel has problems and will not boot.

systemd targets approximately correspond to the legacy SysV runlevels:

• runlevel0.target, poweroff.target, halt

• runlevel1.target, rescue.target, single-user text mode, all local filesystems mounted, root user only, no networking

• runlevel3.target, multi-user.target, multiuser text mode (no graphical environment)

• runlevel5.target, graphical.target, multiuser graphical mode

• runlevel6.target, reboot.target, reboot

systemctl emergency is a special target that is more restricted than rescue mode: no services, no mount points other than the root filesystem, no networking, root user only. It is the most minimal running system for debugging problems. You may see options to boot into a rescue or emergency mode in your GRUB2 bootloader screen.

systemctl is-system-running reports various system states:

• initializing means the system has not completed startup.

• starting means the system is in the final stages of startup.

• running is fully operational, and all processes are started.

• degraded means the system is operational, but one or more systemd units have failed. Run systemctl | grep failed to see which units failed.

• maintenance means that either the rescue or emergency target is active.

• stopping means that systemd is shutting down.

• offline means that systemd is not running.

• unknown means that there is a problem preventing systemd from determining the operational state.

See Also

• man 1 systemctl

• man 8 systemd-halt.service

Problem

systemd promises faster startups, but your system starts up slowly, and you want to find out why.

Solution

You want systemd-analyze blame. Run it with no options to see a list of system processes and how long they took to start:

$ systemd-analyze blame
         34.590s apt-daily.service
          6.782s NetworkManager-wait-online.service
          6.181s dev-sda2.device
          4.444s systemd-journal-flush.service
          3.609s udisks2.service
          2.450s snapd.service
          [...]

Analyze only user processes:

$ systemd-analyze blame --user
          3.991s pulseaudio.service
           553ms at-spi-dbus-bus.service
           380ms evolution-calendar-factory.service
           331ms evolution-addressbook-factory.service
           280ms xfce4-notifyd.service
           [...]

Discussion

It is useful to review everything that starts at boot and perhaps find services you don’t want starting at boot. My favorite to disable is Bluetooth because I don’t use it on my servers or PCs, but many Linux distros enable it by default.

See Also

• man 1 systemd-analyze

Problem

You want to list users’ UIDs and GIDs.

Solution

Use the id command with no options to see your own UID and GIDs. In the following example, the user is Duchess:

duchess@pc:~$ id
uid=1000(duchess) gid=1000(duchess)
groups=1000(duchess),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),118(lpadmin),
126(sambashare),131(libvirt)

Display another user’s UID and GIDs by providing their username as an argument:

duchess@pc:~$ id madmax
uid=1001(madmax) gid=1001(madmax) groups=1001(madmax),1010(composers)

Display your effective ID. This is your ID when you run a command as another user. You can see this with sudo:

duchess@client4:~$ sudo id -un
root

duchess@client4:~$ sudo -u madmax id -gn
madmax

Discussion

There are three types of user IDs in Linux:

• Real UID/GID

• Effective UID/GID

• Saved UID/GID

The real ID is the UID and primary GID assigned to the user at creation. These are what you see when you run the id command, as yourself, with no options.

The effective ID is the UID used to run a process that requires different privileges than the user who launched the process, for example, the passwd command. passwd requires root privileges but uses the special permission modes to allow users to change their own passwords.

You can see this for yourself. First, take a look at the passwd command’s permissions:

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 68208 May 27  2020 /usr/bin/passwd

This shows that passwd is owned by root, both UID and GID. Now type the passwd command and press Enter.

Open a second terminal to find the process for passwd, then print its process ID, effective ID, and real ID:

$ ps -a|grep passwd
12916 pts/1    00:00:00 passwd

$ ps -eo pid,euser,ruser,rgroup | grep 12916
  12916 root     root     root

Even though an unprivileged user is running passwd, it runs with root permissions. (See Recipe 6.11 for information on the special permission modes.)

The saved ID is used by processes that need elevated privileges, usually root privileges. When a process needs to do work that requires fewer privileges, it can temporarily switch to a nonprivileged user ID. The effective UID is changed to the lower privilege value, and the original effective UID is saved to the SUID, saved user ID. When the process needs elevated privileges again, it changes to the SUID.

The id command has a few options:

• -u shows the effective UID number.

• -g shows the effective GID number.

• -G shows all group IDs.

• -n prints the name rather than the number. You can use this in combination with -u, -g, and -G.

• -un shows the effective UID username.

• -gn shows the effective group name.

• -Gn shows all effective GID names.

• -r shows the real ID instead of the effective ID. You can use this in combination with -u, -g, and -G.

See Also

• Recipe 6.11

• man 1 id

• man 1 ps

Problem

You want to create a new user with a user private group and home directory populated with a set of default files like .bashrc, .profile, .bash_history, and any other files you want them to have.

Solution

The useradd command is included in most Linux distributions and is configurable to suit your requirements. The default configuration varies across the various Linux distributions, so the quickest way to learn how your system is set up is to create a new test user:

$ sudo useradd test1

Now run the id command, and then see if useradd created a home directory. The following examples are from Fedora 34:

$ id test1
uid=1011(test1) gid=1011(test1) groups=1011(test1)

$ sudo ls -a /home/test1/
.  ..  .bash_logout  .bash_profile  .bashrc

In this example, the default configuration meets all the requirements listed in the Problem. Now you only need to set a password:

$ sudo passwd test1
Changing password for user test1.
New password: password
Retype new password: password
passwd: all authentication tokens updated successfully.

You may elect to force the user to reset their password at first login, after creating the user’s password:

$ sudo passwd -e test1
Expiring password for user test1.
passwd: Success

Give the login to your user, and they can start using their new account. The new user account is represented like this in /etc/passwd:

test1:x:1011:1011::/home/test1:/bin/bash

Some Linuxes, for example openSUSE, configure useradd to not create the user’s home directory by default and to put all users into the users (100) group. This potentially exposes files to other users, if group permissions on the files allow it. The following example creates a user private group:

$ sudo useradd -mU test2

-m creates the user’s home directory, and -U creates their private group with the same name as their username.

Discussion

All new user accounts are inactive until you set a password.

The first group created for a user, whether it is a user private group or a common group for all users, is their primary group. All other groups the user is assigned to are supplementary groups.

There are some additional useful options:

• -G, --groups is for adding the user to multiple supplemental groups in a comma-delimited list. The groups must already exist:

$ sudo useradd -G group1,group2,group3 test1

• -c, --comment accepts any text string. Use this for the user’s full name, or any comment or description:

$ useradd -G group1,group2,group3 -c 'Test 1,,,,' test1

The four commas define five fields: full name, room number, work phone, home phone, and other. Way back in olden times this was called the GECOS data. GECOS is short for General Electric Comprehensive Operating Supervisor, a mainframe operating system. You may enter any text string in these fields, or nothing, though it is useful to include the user’s full name. Study your /etc/passwd file to see how other entries use the GECOS fields.

The useradd defaults are scattered across multiple configuration files; see Recipe 5.4 to learn how to change the defaults.

See Also

• man 8 useradd

• man 5 login.defs

• /etc/default/useradd

• /etc/skel

• /etc/login/defs

Problem

You want to create a system user with the useradd command.

Solution

The following example creates a new system user with no home directory, no login shell, and uses the correct UID numbering range for system users:

$ sudo useradd -rs /bin/false service1

-r means create a system user with a real ID in the correct numerical range for system users, and -s specifies the login shell. /bin/false is a command that does nothing and prevents the user from logging into the system.

See the Discussion in Recipe 5.6 for information about UID and GID numbering.

Discussion

In olden times, most services ran as the nobody user. Now it is a common practice for services to have their own unique users, as this provides stronger security than the nobody user owning multiple services. You will rarely have to create a system user, as services should create their own unique users when they are installed.

The nobody user is always assigned UID 65534 and GID 65534.

See Also

• man 8 useradd

• man 1 false

• The Discussion in Recipe 5.6

Problem

The default useradd settings are not right for you, and you want to change them.

Solution

The useradd configuration is spread across multiple configuration files: /etc/default/useradd, /etc/login.defs, and files in the /etc/skel directory.

The following values appear in /etc/default/useradd. This example shows the openSUSE defaults:

$ useradd -D
GROUP=100
HOME=/home
INACTIVE=-1
EXPIRE=
SHELL=/bin/bash
SKEL=/etc/skel
CREATE_MAIL_SPOOL=yes

GROUP=100 sets a single shared group as the default for all new users, traditionally 100. The group must first exist, and USERGROUPS_ENAB no must be set in /etc/login.defs. Then set GROUP= in /etc/default/useradd to the GID of the user group. If our Duchess user is in a shared group, her id output shows uid=1000(duchess) gid=100(users).

Enable private user groups by setting USERGROUPS_ENAB yes in /etc/login.defs, then comment out GROUP= in /etc/default/useradd. This creates a nonshared private group for each user. If our Duchess user has her own private group, her id output shows uid=1000(duchess) gid=1000(duchess).

HOME=

sets the default directory for all user home directories. The default is /home.

INACTIVE=-1

sets the number of days after a password expires until the account is locked. A value of 0 disables the account as soon as the password expires, and a value of –1 disables locking the account.

EXPIRE=

sets an expiration date on the account, in YYYY-MM-DD format. For example, if you set it to 2021-12-31, the account will be disabled on that date. Leaving EXPIRE= empty means the account will not expire.

SHELL=/bin/bash

sets the default command shell. /bin/bash is the most commonly used Linux shell. Other values are any installed shell on the user’s system, such as /bin/zsh or /usr/bin/tcsh. cat /etc/shells lists all installed shells.

SKEL=/etc/skel

sets the location for the files that you want automatically distributed to new users. Most Linuxes put them in /etc/skel. These are files such as .bash_logout, .bash_profile or .profile, .bashrc, and any other files you want new users to have. You may edit these files to suit your own requirements. SKEL is short for skeleton.

CREATE_MAIL_SPOOL=yes

is a relic of olden times, and should be set to yes, as there may be some legacy processes that still need it.

The following values in /etc/login.defs are relevant to user creation defaults:

• USERGROUPS_ENAB yes enables private user groups.

• CREATE_HOME yes configures useradd to automatically create private user home directories. This does not apply to system users (see Recipe 5.3).

Discussion

The UID numbering range is defined in /etc/login.defs. Every UID must be unique, so the user account creation commands assign UIDs from the range defined in this file. Typically, human UIDs start at 1000, and are automatically assigned by useradd. You can override this with the -u option, but you must select an unused number that follows the configured numbering scheme (see the Discussion in Recipe 5.6).

A mandatory password change at first login is a simple precaution against the original password possibly falling into the wrong hands as it passes from the administrator to the user.

See Also

• man 8 useradd

• man 5 login.defs

• /etc/default/useradd

• /etc/skel

• /etc/login/defs

Problem

You followed Recipe 5.2 to create a new user, now you want to customize the Documents, Music, Video, Pictures, and Downloads directories for new users.

Solution

Creating these directories is not a function of useradd, but rather the X Desktop Group (XDG) user directories tool. The Documents, Music, Video, etc., directories are called the well-known user directories. These directories are set up from the /etc/xdg/user-dirs.defaults configuration file, which establishes the default configuration for all users:

$ less /etc/xdg/user-dirs.defaults
# Default settings for user directories
#
# The values are relative pathnames from the home directory and
# will be translated on a per-path-element basis into the users locale
DESKTOP=Desktop
DOWNLOAD=Downloads
TEMPLATES=Templates
PUBLICSHARE=Public
DOCUMENTS=Documents
MUSIC=Music
PICTURES=Pictures
VIDEOS=Videos
# Another alternative is:
#MUSIC=Documents/Music
#PICTURES=Documents/Pictures
#VIDEOS=Documents/Videos

These are name-value pairs. The names cannot be changed. The values are the directories that the names are mapped to, and they are relative to users’ home directories. For example, DOCUMENTS is mapped to /home/username/Documents. The directories are created automatically for every new user when they start their graphical desktop environments for the first time. You may comment out any directories you wish to exclude or change the directories the names are mapped to.

Users may create their own personal configurations in ~/.config/user-dirs.dirs. The directories must exist before applying the changes. The following example was created by our example user Duchess, who does not care for the boring default values. Note that the name-value pair syntax is different in ~/.config/user-dirs.dirs:

XDG_DESKTOP_DIR="$HOME/table"
XDG_DOWNLOAD_DIR="$HOME/landing-zone"
XDG_DOCUMENTS_DIR="$HOME/omg-paperwork"
XDG_MUSIC_DIR="$HOME/singendance"
XDG_PICTURES_DIR="$HOME/piccies"

When your changes are complete and the new directories have been created, use the xdg-user-dirs-update command to apply your changes:

duchess@pc:~$ xdg-user-dirs-update --set DOWNLOAD $HOME/landing-zone
duchess@pc:~$ xdg-user-dirs-update --set DESKTOP  $HOME/table
duchess@pc:~$ xdg-user-dirs-update --set DOCUMENTS  $HOME/omg-paperwork
duchess@pc:~$ xdg-user-dirs-update --set MUSIC  $HOME/singendance
duchess@pc:~$ xdg-user-dirs-update --set PICTURES  $HOME/piccies

Log out, then log back in, and you will see something like Figure 5-1. XDG applies special icons to the well-known directories.

Figure 5-1. Custom well-known directories

The shortcuts in the side pane will not change, and the old directories are unchanged except they do not have the special icons. You will have to change the shortcuts and migrate the old directory contents manually.

Restore to the defaults in /etc/xdg/user-dirs.defaults with this command:

$ xdg-user-dirs-update --force

Log out and back in to see the changes. Again, none of your directories are removed or changed in any way, except for the special icons that mark the well-known user directories.

Discussion

When you run the xdg-user-dirs-update --set command, you must use only the names as listed in man 5 user-dirs.default:

DESKTOP
DOWNLOAD
TEMPLATES
PUBLICSHARE
DOCUMENTS
MUSIC
PICTURES
VIDEOS

Only the values, which are the target directories, are configurable. Target directories must be relative to users’ home directories. If you want to use directories outside of your home, create symlinks. For example, Duchess owns /users/stuff/duchess and stores music files in it. The following example links this directory to /home/duchess/singendance:

duchess@pc:~$ ln -s /users/stuff/duchess /home/duchess/singendance

See Also

• man 5 user-dirs.defaults

• man 1 xdg-user-dirs-update

• man 5 user-dirs.conf

• xdg-user-dirs at freedesktop

Problem

You want to create groups with groupadd.

Solution

The following example creates a new user group musicians:

$ sudo groupadd musicians

Use groupadd with the -r option to create a system group:

$ sudo groupadd -r service1

Discussion

System groups differ from human user groups in the UID and GID numbering ranges assigned to them. This is configured in /etc/login.defs for groupadd and useradd, as this example from Fedora 34 shows:

# Min/max values for automatic uid selection in useradd(8)
#
UID_MIN                  1000
UID_MAX                 60000
# System accounts
SYS_UID_MIN               201
SYS_UID_MAX               999
# Extra per user uids
SUB_UID_MIN                100000
SUB_UID_MAX             600100000
SUB_UID_COUNT               65536

#
# Min/max values for automatic gid selection in groupadd(8)
#
GID_MIN                  1000
GID_MAX                 60000
# System accounts
SYS_GID_MIN               201
SYS_GID_MAX               999
# Extra per user group ids
SUB_GID_MIN                100000
SUB_GID_MAX             600100000
SUB_GID_COUNT               65536

These define the number ranges available to the system administrator. All others are reserved for and managed by the system.

GID numbering is managed automatically by groupadd, according to the number ranges defined in /etc/login.defs You may override this with the -g option, but your chosen GID must fall within the defined range, and must not already be used.

See Also

• man 8 groupadd

• /etc/login.defs

Problem

You want to assign users to groups.

Solution

Use the usermod command. The following example adds Duchess to the musicians group:

$ sudo usermod -aG musicians duchess

This example adds Duchess to multiple groups:

$ sudo usermod -aG musicians,composers,stagehands duchess

Alternatively, you could edit /etc/group and type Duchess’s name after the appropriate group or groups. When you list multiple group members, the list must be comma-delimited, with no spaces between the names.

musicians:x:900:stash,madmax,duchess

When you change group memberships for logged-in users, users must log out and then log back in to activate the changes. There are various workarounds to activate a new group assignment without logging out, but they all have limitations, such as being limited to the current shell. Groups are enumerated at login, so the most reliable solution is to log out and log back in.

Discussion

The -a option means append, and -G is group or groups.

See Also

• man 8 usermod

Problem

You are running Debian or a Debian-based Linux, and need to know how to create new users with adduser.

Solution

adduser walks you through a complete new user setup, like this example for Stash Cat:

$ sudo adduser stash
Adding user 'stash' ...
Adding new group 'stash' (1009) ...
Adding new user 'stash' (1009) with group 'stash' ...
Creating home directory '/home/stash' ...
Copying files from '/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for stash
Enter the new value, or press ENTER for the default
        Full Name []: Stash Cat
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n]

Stash looks like this in /etc/passwd:

stash:x:1009:1009:Stash Cat,,,:/home/stash:/bin/bash

Discussion

The adduser defaults are managed in /etc/adduser.conf. This provides a number of useful defaults such as:

DSHELL=

sets the default login shell. /bin/bash is the most commonly used Linux shell. Other values are any installed shell on the user’s system, such as /bin/zsh or /usr/bin/tcsh. cat /etc/shells lists all installed shells.

USERGROUPS=yes

creates user private groups, no puts all users into the same group.

USERS_GID=100

is required when USERGROUPS=no is set.

EXTRA_GROUPS=

is your list of supplemental groups for new users, for example, EXTRA_GROUPS="audio video plugdev libvirt"‍.

ADD_EXTRA_GROUPS=1

makes the groups listed in EXTRA_GROUPS= the default for new users.

/etc/adduser.conf contains the following user and group numbering scheme:

FIRST_SYSTEM_UID=100
LAST_SYSTEM_UID=999

FIRST_SYSTEM_GID=100
LAST_SYSTEM_GID=999

FIRST_UID=1000
LAST_UID=59999

FIRST_GID=1000
LAST_GID=59999--

Fedora Linux includes adduser, but it is not really adduser, just a symlink to useradd:

$ stat /usr/sbin/adduser
  File: /usr/sbin/adduser -> useradd
  Size: 7    Blocks: 0    IO Block: 4096   symbolic link
[...]

See Also

• man 5 adduser.conf

Problem

You want to create a system user with adduser on your Ubuntu (or Debian, Mint, or other Debian derivative) system.

Solution

The following example creates a new system user, service1, with adduser, without a home directory, and with its own unique primary group:

$ sudo adduser --system  --no-create-home --group service
Adding system user 'service1' (UID 124) ...
Adding new group 'service1' (GID 135) ...
Adding new user 'service1' (UID 124) with group 'service1' ...
Not creating home directory '/home/service1'.

This is how it looks in /etc/passwd:

service1:x:124:135::/home/service1:/usr/sbin/nologin

Discussion

System users do not have home directories.

Back in olden times, it was common for services to run as the nobody user and group, except Debian which uses nobody and nogroup. Recycling the same user for multiple services is a security weakness. It is unlikely you will ever have to create a system user, as the common practice now is for the package installer to create a unique user and group when you install a new service. But now you know how, just in case you ever need to.

nobody and nogroup always have a real ID of 65534.

See Also

• man 8 adduser

Problem

You want to know how to create user and system groups with Debian’s addgroup command.

Solution

The following example creates a human user group:

$ sudo addgroup composers
Adding group 'composers' (GID 1010) ...
Done.

It looks like this in /etc/group:

composers:x:1010:

This example creates a new system group:

$ sudo addgroup --system service1
Adding group 'service1' (GID 136) ...
Done.

Discussion

The difference between user and system groups is they each have different numbering ranges for UIDs and GIDs, which are configured in /etc/adduser.conf.

See Also

• man 8 addgroup

Problem

There is a lot going on in all these user files and group files, and you want to know if there is some kind of checker to verify that these files are written correctly.

Solution

The pwck command checks the integrity of /etc/passwd and /etc/shadow, and grpck checks /etc/group and /etc/gshadow. They look for correct format, valid data, valid names, and valid GIDs (see the man pages for a complete list). When you run these with no options, they report both warnings and errors:

$ sudo pwck
user 'news': directory '/var/spool/news' does not exist
user 'uucp': directory '/var/spool/uucp' does not exist
user 'www-data': directory '/var/www' does not exist
user 'list': directory '/var/list' does not exist

$ sudo grpck
group mail has an entry in /etc/gshadow, but its password field in /etc/group is
not set to 'x'
grpck: no changes

Add the -q option to report only errors:

$ sudo pwck -q

$ sudo grpck -q
group mail has an entry in /etc/gshadow, but its password field in /etc/group
is not set to 'x'

This shows an error in /etc/gshadow. This is not a very helpful message because it’s not really an error. It is not a common practice to place passwords on user groups, so reporting this an as error is needlessly confusing. The other checks are useful, such as the correct number of fields, and a unique valid group name.

You will never edit /etc/shadow or /etc/gshadow, but only /etc/passwd and /etc/group.

Discussion

The following example shows an error that must be corrected. Type n, when prompted, to prevent deleting the entries. The first “delete line” example is from /etc/passwd, and the second is in /etc/shadow:

$ sudo pwck -q
invalid password file entry
delete line 'fakeservice:x:996:996::/home/fakeservice'? n
delete line 'fakeservice:!:18469::::::'? n
pwck: no changes

Then correct the line in /etc/passwd, and that will fix both error messages. In the example, fakeservice:x:996:996::/home/fakeservice is missing the last field, and should be fakeservice:x:996:996::/home/fakeservice:/bin/false.

The “directory does not exist” warnings for /etc/passwd usually refer to default system users that are not in use. For example:

user 'www-data': directory '/var/www' does not exist

The www-data user is unused when you are not running an HTTP server, and there is no /var/www directory until you install an HTTP server.

The “no changes” message means that no changes were made to the password file.

See the man pages for a complete list of checks.

See Also

• man 8 pwck

• man 8 grpck

Problem

You want to disable a user account without deleting it.

Solution

To temporarily deactivate an account, disable the user’s password with the passwd command:

$ sudo passwd -l stash
passwd: password expiry information changed.

Now the user cannot log in. The following example unlocks the user’s account:

$ sudo passwd -u stash
passwd: password expiry information changed.

This does not prevent a user from logging in via a different authentication method, such as an SSH key. To completely disable a user account, use usermod:

$ sudo usermod --expiredate 1 stash

When the user tries to log in, they see a “Your account has expired; please contact your system administrator” message. Restore their account:

$ sudo usermod --expiredate -1 stash

Discussion

Another way to disable a user is to replace the x in the password field in /etc/passwd with an asterisk (*):

stash:*:1009:1009:Stash Cat,,,:/home/stash:/bin/bash

Reenable Stash by replacing the asterisk with x.

See Also

• man 1 passwd

Problem

You need to delete a user, and possibly their home directory and its contents.

Solution

The following example uses the userdel command to delete the user Stash from /etc/passwd, Stash’s primary group and all group memberships, and the shadow files:

$ sudo userdel stash

If Stash belongs to a shared primary user group (discussed in Recipe 5.4), the group will not be deleted.

Use the -r option to delete the user’s home directory and its contents, and their mail spool:

$ sudo userdel -r stash

If the user owns files outside of their home directory, you will have to find and take care of them separately (see Recipe 5.16).

Discussion

Read your /etc/passwd and /etc/group files before and after deleting a user to see the user disappear.

It is a good practice to clean up after removing a user.

See Also

• man userdel

Problem

You’re running Ubuntu (or another Debian derivative) and want to use deluser to delete a user.

Solution

The following example deletes the user Stash from /etc/passwd, Stash’s primary group from /etc/group, and their corresponding shadow files:

$ sudo deluser stash
Removing user 'stash' ...
Warning: group 'stash' has no more members.
Done.

deluser will not remove the primary group of an existing user, so if Stash belongs to a shared primary group it will not be removed.

This example removes Stash’s home directory and makes a backup of all the deleted files:

$ sudo deluser --remove-all-files --backup stash

Discussion

--backup creates a compressed archive of the user’s files in the current directory. Use the --backup-to option to select a different directory:

$ sudo deluser --remove-all-files --backup-to /user-backups stash

If the user owns files outside of their home directory, you will have to hunt them down and deal with them manually (see Recipe 5.16).

See Also

• man 8 deluser

Problem

You have an Ubuntu system and want to use the delgroup command to delete groups.

Solution

The following example removes the musicians group:

$ sudo delgroup musicians

delgroup will not remove the primary group of an existing user. It will remove supplemental groups even when they have members. If you do not want to remove groups that have members, use the --only-if-empty option:

$ sudo delgroup --only-if-empty musicians

Discussion

The default behavior of delgroup is configured in /etc/deluser.conf and /etc/adduser.conf.

See Also

• man 8 delgroup

Problem

You want to delete a user, but you don’t want a bunch of orphaned files left over, and you need to find all of them.

Solution

The find command will locate all files on the local system by UID or GID. The following example searches the entire root directory for all files owned by the user’s UID:

$ sudo find / -uid 1007

This can take some time if there are lot of files to search. If you are certain you do not have to search the entire filesystem, you can narrow your search to specific subdirectories, such as /etc, /home, or /var:

$ sudo find /etc -uid 1007
$ sudo find /home -uid 1007
$ sudo find /var -uid 1007

You may also search by GID, username, or group name:

$ sudo find / -gid 1007
$ sudo find / -name duchess
$ sudo find / -group duchess

Now that you know where all the files are, what to do with them? One option is to change their ownership to another user and let the new user deal with them:

$ sudo find /backups -uid 1007 -exec chown -v 1010 {} \;
changed ownership of '/backups/duchess/' from 1007 to 1010
changed ownership of '/backups/duchess/bin' from 1007 to 1010
changed ownership of '/backups/duchess/logs' from 1007 to 1010

You could combine find and cp to find and copy all the files to a different directory:

$ sudo find / -uid 1007 -exec cp -v {} /orphans \;

Using cp -v prints progress messages and copies only the files and not their parent directories. If you wish to copy the parent directories, use the -r option:

$ sudo find / -uid 1007 -exec cp -rv {} /orphans \;

Copying leaves the original files in place. After they are safely copied, you may wish to delete the originals. One way to do this is to run find again and use rm to delete the original files:

$ sudo find / -uid 1007 -exec rm -v {} \;

This deletes the files but not the directories. Use the -r option to delete the directories if you are certain there are no other files in those directories that you want to keep:

$ sudo find / -uid 1007 -exec rm -rv {} \;

One more option is to use find and mv to move the files to a different location:

$ sudo find / -uid 1007 -exec mv {} /orphans \;

If you see a “No such file or directory” message, usually that is because the file or directory was moved, which you can verify by looking in the directory they were moved to.

Find files owned by a nonexistent user or group:

$ find / -nouser
$ find / -nogroup

Discussion

Be careful with mv and rm because there is no undo. If you make a mistake, your best hope of recovery is from backup.

Cleaning up after departed users can be a chore, because computers make it too easy to create as many files as your storage will hold. If you find yourself thinking that find is taking too long, keep in mind it will find everything while you go do something else.

See Also

• man 1 find

• man 1 mv

• man 1 cp

• man 1 rm

Problem

You need to know how to get root permissions to perform some administration chores.

Solution

Use the su command to change to the root user when you need to do system chores:

duchess@pc:~$ su -l
Password:
root@pc:~#

If you do not know root’s password, or if there is no root password, see Recipe 5.21 to learn how to use sudo to set a root password.

When you are finished, exit root and return to your own account:

root@pc:~# exit
logout
duchess@pc:~$

The -l option invokes the root user’s environment, changing to root’s home directory and loading root’s environment variables. Omit -l to keep your own environment:

duchess@pc:~$ su
Password:
root@pc:/home/duchess~#

Discussion

You can change to any user, as long as you have their password.

Using su to change to root gives you absolute power over your system, and every command that you run is run as root. Consider using sudo (see Recipe 5.18), which provides some safety features, such as protecting root’s password and leaving an audit trail.

See Also

• man 1 su

Problem

You want to delegate some system administration chores to other users, and you want to limit their root powers to what is needed for their specific tasks.

Solution

Use the sudo command. sudo is safer than su because it grants limited root powers to specific users for specific tasks, logs activity, and caches the user’s password for a limited amount of time, with a default of 15 minutes. After 15 minutes, the user must provide sudo with their password again. The caching duration is configurable. sudo protects root’s password because sudo users use their own passwords.

/etc/sudoers is the configuration file, and you should edit it with a special command, visudo. This opens /etc/sudoers with your default text editor, and you can review and edit the default configuration. Once again, Duchess demonstrates for us:

duchess@pc:~$ sudo visudo
[sudo] password for duchess:
[...]
##Allow root to run any commands
root    ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL) ALL
[...]

%sudo ALL=(ALL) ALL means that any user you add to the sudo group gets full sudo powers just like root. The percent sign indicates %sudo is a group from /etc/group, and not a group configured in /etc/sudoers.

Suppose you have a junior admin, Stash, whose job is installing and removing software and keeping the system updated. You could create a system group for Stash. Or you could configure Stash for this task in /etc/sudoers. The following example gives Stash sudo powers to run the listed commands. You need the username, the hostname of the local machine, and a comma-delimited list of allowed commands:

stash server1 = /bin/rpm, /usr/bin/yum, /usr/bin/dnf

Suppose you want to give Stash more admin chores, such as managing services. The allowed commands list will get long, so you could create some command aliases instead. The following example aliases the software management commands to SOFTWARE, and the service management commands to SYSTEMD:

Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/yum, /usr/bin/dnf
Cmnd_Alias SYSTEMD = /usr/bin/systemctl start, /usr/bin/systemctl stop,
/usr/bin/systemctl reload, /usr/bin/systemctl restart, /usr/bin/systemctl
status, /usr/bin/systemctl enable, /usr/bin/systemctl disable,
/usr/bin/systemctl mask, /usr/bin/systemctl unmask

Now Stash’s configuration looks like this:

stash server1 = SOFTWARE, SYSTEMD

You may create user groups in /etc/sudoers (not related to system groups in /etc/group), then assign them some command aliases:

User_Alias  JRADMIN = stash, madmax

JRADMIN server1 = SOFTWARE, SYSTEMD

You may create a Host_Alias to give a user sudo rights on multiple machines:

Host_Alias SERVERS = server1, server2, server3

Then bring in the JRADMINs:

JRADMIN SERVERS = SOFTWARE, SYSTEMD

Discussion

When your limited sudo users try to run a not-allowed command, they see this message: “Sorry, user duchess is not allowed to execute /some/command as root on server2.”

Don’t put too much faith in limiting users to a specific set of commands. Many everyday applications provide a means for privilege escalation via shell escape, and your users can gain full root powers. This example shows how it works with awk:

$ sudo awk 'BEGIN {system("/bin/bash")}'
root@client4:/home/duchess# 

And just like that, Duchess has full root powers. The humble less command also provides a shell escape. Read a file with less that is large enough to require paging:

$ sudo less /etc/systctl.conf
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
/etc/sysctl.conf

Type !sh, then when the prompt changes, type whoami:

duchess@client4:~$ sudo less /etc/systctl.conf
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
!'sh'
duchess@client4:~$ sudo less /etc/sysctl.conf
# whoami
root

Type exit to return your normal shell.

In my experience, it is extremely difficult to keep track of the many applications that can provide a shell escape. journalctl records everything, should you wish to monitor your sudo users (see Recipe 20.1).

In some Linuxes, such as Fedora, the wheel group is the default sudo group. Check your /etc/sudoers file to see how your distribution configures this. You can also create your own sudo group, and name it whatever you want.

The /etc/sudoers file controls users only on the local machine. Including other machines, like the SERVERS alias, allows you to share a single configuration file on multiple machines. sudo ignores any items, such as hosts or users, that are not present on the local machine.

Let’s dissect root ALL=(ALL) ALL to understand what all those ALLs mean.

root

is in the user field, and this field holds any single user, user alias, or system group.

ALL=

is in the host field. ALL means any host anywhere, or you could use a host alias, or name a single host.

(ALL)

is in the optional users field. (ALL) means the user or users can run commands as any other user, or you can specify certain users.

ALL

in is the command field. ALL is unrestricted, or you can specify a list of allowed commands.

See Also

• man 8 sudo

• man 5 sudoers

Problem

On most Linux distributions, sudo caches passwords for a default interval of 15 minutes. Then after 15 minutes you have to enter your password again. You are tired of having to enter your password so often when you have a lot of work to do, and want to make the caching interval longer.

Solution

Change the caching interval in /etc/sudoers. Open the file for editing with visudo:

$ sudo visudo

Then look for a Defaults line and set your new cache interval. The following example sets it to 60 minutes:

$ Defaults timestamp_timeout=60

If you set it to 0, sudo asks for your password every time you use it.

If you set timestamp_timeout to a negative number, like -1, your password never expires.

Discussion

The sudo password caching is a useful protection against accidents, such as forgetting you are running as root or wandering away and allowing someone else to have some fun with your computer.

See Also

• man 8 sudo

• man 5 sudoers

Problem

You want to set some different sudo configurations for your users; for example, you want your junior admins to have a different password timeout than you. Yours is long, and you want theirs to be short.

Solution

You may create individual configurations in /etc/sudoers.d. The following example creates a 30-minute password timeout for Stash:

$ cd /etc/sudoers.d/
$ sudo visudo -f stash

Type Defaults timestamp_timeout=30, save the file, and you’re done. You can see the new file:

$ sudo ls /etc/sudoers.d/
README stash

You only need to enter configuration items that are different from the entries in /etc/sudoers, not to replicate the whole file.

Discussion

This is a nice feature for managing multiple users. Instead of managing one big configuration file, break it up into smaller per-user files.

See Also

• man 8 sudo

• man 5 sudoers

Problem

Your Linux distribution set you up as system administrator during installation, with unrestricted sudo privileges, and did not create a root password. Or, your root user had a password but you forgot it. You need to know how to give root a new password.

Solution

When you want to run as “real” root, use sudo to su to root:

duchess@pc:~$ sudo su -l
[sudo] password for duchess:
root@pc:~#

At this point you can use the passwd command to give root a password so you can log in as root directly, or reset a lost root password.

Discussion

There are times when you need a root password and not sudo; for example, when you boot to an emergency runlevel.

See Also

• man 8 sudo

• man 5 sudoers

• man 1 passwd

Problem

You want your sudo users to authenticate with their own passwords, but your Linux system asks for the root user’s password, like the following example:

$ sudo visudo
[sudo] password for root:

Solution

This is the default behavior on some Linux distributions, such as openSUSE.

When you install Ubuntu Linux and make your user an administrator during installation, Ubuntu configures your user appropriately with full sudo powers, equivalent to root but using your own password. openSUSE does not, but instead configures your user to use the password of the target user, which is root.

To set up sudo users to always be asked for their own passwords, edit /etc/sudoers by commenting out these two lines:

duchess@pc:~$ sudo visudo

# Defaults targetpw
# ALL   ALL=(ALL) ALL 

In openSUSE and Fedora, create sudo users with full root powers by adding them to the wheel group in /etc/group. (For limited users, refer to Recipe 5.18.)

The change takes effect immediately after saving your changes and closing the file.

Discussion

Protecting the root user’s password is a primary reason to use sudo, rather than su.

See Also

• Recipe 5.18

Problem

You want to organize your files by placing them in directories.

Solution

Use the mkdir command to create directories. The following example creates a new subdirectory in the current directory:

$ mkdir -v presentations
mkdir: created directory 'presentations'

Create a subdirectory two levels down inside the current directory, and its parent directories, with the -p (parent) option:

$ mkdir -p presentations/2020/august
mkdir: created directory 'presentations/2020'
mkdir: created directory 'presentations/2020/august'

Create a new top-level directory, which is relative to root, /. You need root privileges to do this:

$ sudo mkdir -v /charts
mkdir: created directory '/charts'

You can set permissions when you create a directory:

$ mkdir -m 0700 /home/duchess/dog-memes

Files are created by applications, such as word processors and image editors, and special commands like touch. The touch command creates a new empty file:

$ touch newfile.txt

See Recipe 6.2 to learn how to use touch to quickly create batches of files for testing.

Discussion

If you are having trouble visualizing file trees, and how all directories are relative to /, try the tree command. Root, /, is at the top:

$ tree -L 1 /
/
├── backups
├── bin
├── boot
[...]

You have probably noticed that this is upside down. In the real world, trees branch from the root, but the tree command displays the directory tree branching downward. There is a reason for this: we read screens from the top down.

This example lists only the top-level directories under root. -L 2 shows second-level directories, -L 3 goes to three levels, and so on.

See Also

• Recipe 6.2

• man 1 mkdir

• man 1 touch

• man 1 yes

• man 1 tree

Problem

You want to create batches of files to use for testing file permissions, and for any testing that needs a lot of files in a hurry.

Solution

Use the touch command. The following example creates a single new empty file:

$ touch newfile.txt

Create 100 new empty files:

$ touch file{00..99}

This creates 100 new files named file00, file01, file02, and so on. You may give them file extensions and name them anything:

$ touch test{00..99}.doc
$ ls
test00.doc
test01.doc
test02.doc
[...]

Put the numbers first in the filename for easy ordering:

$ touch {00..99}test.doc
$ ls
00test.doc
01test.doc
02test.doc
[...]

A fast way to populate the files with content is to use the yes command. The following example creates a 500 MB file filled with the repeated line “This is a test file”:

$ yes This is a test file | head -c 500 MB > testfile.txt

Create a batch of 100 files with 1 MB of content in each file:

$ for x in {01..100};
> do yes This is a test file | head -c 1MB > $x-testfile.txt;
> done 

The new files look like this:

001-testfile.txt
002-testfile.txt
003-testfile.txt
[...]

Discussion

You may customize this command in a number of ways: filenames, file sizes, numbering, and the text for yes.

The examples in the recipe pad the numbers in the filenames with leading zeroes so they will order correctly. Most graphical file managers handle ordering numbered filenames correctly, but the default for ls is lexicographic order. The following example demonstrates this with a 1- to 3-digit numbering range:

$ touch {0..150}test.doc
$ ls -C1
0test.doc
100test.doc
101test.doc
102test.doc
103test.doc
104test.doc
105test.doc
106test.doc
107test.doc
108test.doc
109test.doc
10test.doc
110test.doc
111test.doc
112test.doc
113test.doc
114test.doc
115test.doc
116test.doc
117test.doc
118test.doc
119test.doc
11test.doc
120test.doc
121test.doc
[...]

Lexicographic ordering treats the filenames as text strings instead of integers and characters, and compares each number and letter individually, from left to right. Lexicographic ordering doesn’t know that 10 is smaller than 100, only that 101 follows 100, 102 follows 101, and 10t follows 109 because letters follow numbers, so the t follows the 9.

You can use leading zeroes to make all the numbers the same number of characters, or list your files with ls -v. This treats the numbers in filenames as integers and not characters, so they are listed in correct numerical order.

See Also

• man 1 ls

• man 1 touch

• man 1 yes

Problem

You need to understand the difference between relative and absolute filepaths, and how to find where you are in the filesystem.

Solution

Absolute filepaths always start at the root, /, such as /boot and /etc. Relative filepaths are relative to your current directory and do not have a leading slash. Suppose you are in your home directory and it contains the following subdirectories:

madmax@client2:~$ ls --group-directories-first
 Audiobooks
 bin
 Desktop
 Documents
 Downloads
 games
 Music
 Pictures
 Public
 Templates
 Videos

In this example, the absolute path to Audiobooks is /home/madmax/Audiobooks, and the relative path is Audiobooks. Use the cd command to enter this directory with either the absolute path:

$ cd /home/madmax/Audiobooks

Or the relative path:

$ cd Audiobooks

The directory you are in is the current working directory, cwd. Confirm your cwd with the pwd (print working directory) command:

$ pwd
/home/madmax

Discussion

Absolute and relative filepaths are a common source of confusion. Remember that when the filepath begins with a slash (/), it is an absolute path. When there is no leading slash, it is relative to your current working directory.

Some applications and commands require relative paths; for example, rsync include and exclude lists use filepaths that are relative to the directories being copied.

See Also

• man 1 pwd

• Chapter 7

Problem

You had fun creating a bunch of files and directories, and now you want to get rid of them.

Solution

Use the rm (remove) command with caution, because rm will happily delete everything you tell it to, so be sure you tell it the right files or directories to delete.

Delete a single file, with verbose output:

$ rm -v aria.ogg
removed 'aria.ogg'

Use the -i flag to prompt for confirmation first:

 $ rm -iv intermezzo.wav
rm: remove regular file 'intermezzo.wav'? y
removed 'intermezzo.wav' 

Add the -r (recursive) flag to delete a directory and all of its files and subdirectories. Combining -r with -i will prompt you for confirmation before each deletion:

$ rm -rvi rehearsals
rm: descend into directory 'rehearsals'? y
rm: remove regular file 'rehearsals/brass-section'? y
[...]

If you are confident you don’t need to be prompted for every deletion, omit the -i option.

This example deletes only the jan subdirectory:

$ rm -rv rehearsals/2020/jan

This example deletes the rehearsals directory and all of its files and subdirectories:

$ rm -rv rehearsals

Use wildcards to match file names to delete, for example by file extension:

$ rm -v *.txt

Or by files named with the same text strings:

$ rm -v aria*

If rm refuses to delete a file or directory, and you are certain you want to delete it, add the -f (force) option.

Discussion

rm -rf / will erase your entire root filesystem (if you have root privileges). Some folks think it is a funny prank to tell newbies to do this. It is not funny. It is fun to run it on a test machine, or on a virtual machine, and observe how long the system keeps running because processes in memory are still running, even though the filesystem is erased from disk.

See Also

• man 1 rm

Problem

You have directories, and you have files. You want to move files into the directories, change filenames, and make copies.

Solution

Use the cp command for copying, and the mv command for moving or renaming.

This example copies two files from the current working directory into the ~/songs2 directory:

$ cp -v aria.ogg solo.flac ~/songs2/
'aria.ogg' -> '/home/duchess/songs2/aria.ogg'
'solo.flac' -> '/home/duchess/songs2/solo.flac' 

Copy a directory and all of its contents with the -r (recursive) option:

$ cp -rv ~/music/songs2 /shared/archives

The recursive example only copies the directory and its files. Use the --parents option to preserve parent directories. The following example copies songs1 and its contents, and preserves the filepath duchess/music/songs2/:

$ cp -rv --parents duchess/music/songs2/ shows/
duchess -> shows/duchess
duchess/music -> shows/duchess/music
'duchess/music/songs2' -> 'shows/duchess/music/songs2'
'duchess/music/songs2/intro.flac' -> 'shows/duchess/music/songs2/intro.flac'
'duchess/music/songs2/reprise.flac' -> 'shows/duchess/music/songs2/reprise.flac'
'duchess/music/songs2/solo.flac' -> 'shows/duchess/music/songs2/solo.flac'

The other contents of duchess and music are not copied, only songs2 and its contents.

Use the mv command to move and rename files. This example moves two files to another directory:

$ mv -v aria.ogg solo.flac ~/songs2/
renamed 'aria.ogg' -> '/home/duchess/songs2/aria.ogg'
renamed 'solo.flac' -> '/home/duchess/songs2/solo.flac' 

The following example moves a directory into another directory:

$ mv -v ~/songs2/ ~/music/

Discussion

Some useful cp options are:

• -a, --archive preserves all the file attributes, such as mode, ownership, and timestamps.

• -i, --interactive prompts before overwriting destination files.

• -u, --update overwrites an existing destination file only if the source file is newer. This saves time when you’re recopying a batch of files, and some of the copies are unchanged. (rsync is better for efficient file transfers by copying only changes, see Chapter 7.)

mv has some useful options:

• -i, --interactive prompts before overwriting destination files.

• -n, --no-clobber prevents overwriting destination files.

• -u, --update moves your files only when they are newer than the destination files, or when they are moved for the first time.

See Also

• man 1 cp

• man 1 mv

Problem

You know that the chmod (change mode) command supports both octal and symbolic notation, and you want to use octal notation to manage file permissions.

Solution

The following examples show how to set different permissions on files using octal notation. The first example grants read-write access to the owner of the file.txt file, and excludes all access for group and world:

$ chmod -v 0600 file.txt
mode of 'file.txt' changed from 0644 (rw-r--r--) to 0600
(rw-------)

The file owner can read, edit, and delete the file, while other users can do nothing with it, not even read it, though they can see it listed in a file manager.

Make a file world readable and writeable, allowing everyone to do whatever they want to it:

$ chmod 0666 file.txt

In the next example, file.txt is changed to read-write for the file owner and read-only for group and world:

$ chmod -v 0644 file.txt
mode of 'file.txt' changed from 0666 (rw-rw-rw-) to 0644 (rw-r--r--)

A common permission set is to give the owner and group the same permissions, such as read-write, and to exclude other:

$ chmod 0660 file.txt

Commands and scripts require the executable bit to be set. This example makes the backup.sh script executable and read-write for the owner, executable and readable for group, and inaccessible to other:

$ chmod 0750 backup.sh

Octal notation has four fields, but you probably will use the last three fields the most often, and the first field rarely. The first field is reserved for the special modes (see Recipe 6.8).

Discussion

Octal notation uses integers 0-7. Table 6-1 shows the relationship between owners and permissions.

A file or directory has one user owner, and one group owner. Other is everyone else. A directory or executable that is unrestricted to everyone is mode 0777, and an unrestricted file is mode 0666.

When you’re not familiar with Linux file permissions, it might help to see them in another view, like in Table 6-2.

See Also

• man 1 chmod

• Recipe 6.8

Problem

You know that permissions are managed a little differently on directories, and you want to manage them with chmod’s octal notation.

Solution

Directories must have the executable bit set. This might sound a little strange, but it is necessary for entering the directory with the cd command or with a file manager.

The following examples creates a shared directory:

$ sudo mkdir /shared

This example makes /shared read-write for the owner and read-only for everyone else:

$ chmod 0755 /shared

The owner has unrestricted privileges to the directory. Group and world may enter the directory and read files, but not edit or add files.

This example applies the same permissions to the existing contents of the directory, using the -R (recursive) option:

$ chmod -R 0755 /shared

The next example restricts the directory and its existing contents to the directory owner. Files and directories inside the directory may have different owners and permissions, but are still inaccessible to group and world:

$ chmod 0700 /shared

A common permission set is to give the owner and group the same permissions, such as read-write-execute, and to exclude other:

$ chmod 0770 /shared

Discussion

You have a lot of power with groups and directories to control file access. Set up groups according to function, for example various teams could each have their own exclusive shared directories. Most shops don’t need super-fine-grained control and default to more sharing rather than less. Whatever your needs are, the old chmod command is still the fundamental tool for controlling file permissions.

See Also

• man 1 chmod

Problem

You want to set some permissions not supported by the traditional user-group-other set of permissions, such as allowing unprivileged users to run a command that requires elevated permissions, protecting files in a directory shared by multiple users, or enforcing certain file permissions in a directory.

Solution

The special modes are sticky bit, setuid, and setgid (see Table 6-3). The sticky bit is applied to directories that contain files owned by multiple users, to prevent users from moving, renaming, or deleting files they do not own:

$ chmod -v 1770 /home/duchess/shared
mode of '/home/duchess/shared changed from 0770 (rwxrwx---) to 1770 (rwxrwx--T)

setuid is applied to executable files, to elevate any user running the command to the same permissions as the owner:

$ chmod 4750 backup-script
mode of 'backup-script' changed from 0750 (rwxrw----) to 4770 (rwsrwx---)

Apply setgid to a directory, so that all newly created files in the directory are assigned to the same group as the directory’s group owner. This is a nice trick for enforcing correct ownership in a shared directory:

$ chmod 2770 /home/duchess/shared
mode of '/home/duchess/shared' changed from 0770 (rwxrwx---) to 2770 (rwxrws---)

setgid may also be applied to files, changing the effective group of the user to the same group as the file owner.

Discussion

setgid and setuid have the potential to create security holes for an intruder or an untrustworthy user. It is a best practice to use them only when you can’t think of a safer way to accomplish what you want to do, such as using group assignments or sudo.

setuid is useful for executable files.

setgid is useful for directories and files.

The sticky bit is only for directories. Table 6-3 shows the relationship of permissions to owners.

The special mode values may be combined (see Table 6-4).

A more descriptive name for the sticky bit is restricted deletion bit. This bit prevents unprivileged users from removing or renaming a file in a directory, unless they own the file. You can see this on your /tmp directory, which is world readable and writeable, and contains files owned by multiple users. Using the sticky bit prevents users from moving, renaming, or deleting files they do not own, even if they have write privileges on some files they do not own:

 $ stat --format=%a:%A:%U:%G /tmp
1777:drwxrwxrwt:root:root

The sticky bit is the 1 in 1777.

setgid means set group user identification, and setuid is set user identification. These are used to elevate the permissions of an unprivileged user to the same as the user or group owner. This is how unprivileged users can use the passwd command to change their own passwords, even though only root has write permissions on /etc/passwd, and everyone else has only read and execute permissions:

 $ stat --format=%a:%A:%U:%G /usr/bin/passwd
4755:-rwsr-xr-x:root:root

In /etc/passwd the 4 in 4755 is setuid, which means all users have root powers when they run the command, though their powers are limited to changing their own passwords.

See Also

• man 1 chmod

Problem

You want to remove the special modes from a file or directory.

Solution

Removing a special mode is a little different from setting it because you need to use an extra leading zero, as in the following example:

$ chmod -v 00770 backup.sh
mode of 'backup.sh' changed from 1770 (rwxrwx--T) to 0770 (rwxrwx---)

Or replace the leading zeroes with a leading equals sign:

$ chmod -v =770 backup.sh
mode of 'backup.sh' changed from 1770 (rwxrwx--T) to 0770 (rwxrwx---)

See Also

• man 1 chmod

Problem

You know that the chmod (change mode) command supports both octal and symbolic notation, and you want to use symbolic notation to manage file permissions.

Solution

Symbolic notation is more complex than octal notation and behaves differently according to which operator you use.

There are three operators: +, -, and =. You can change permissions for everyone with the a flag, or individually with u for the file owner, g for the group, and -o for other, which is everyone else:

• + adds to existing permissions.

• - subtracts from existing permissions.

• = adds new permissions, and removes any permission bits not listed.

Suppose that file.txt is owner read-write, group read, and other read, or -rw-r--r--:

$ stat --format=%a:%A:%U:%G file.txt
664:-rw-r--r--:stash:stash

You want to change it to -rw-rw-rw-. Add write permissions to group and other:

$ chmod -v g+w,o+w file.txt
mode of 'file.txt' changed from 0644 (rw-r--r--) to 0666 (rw-rw-rw-)

You could also use a=rw.

In the next example the owner of file.txt changes it from world readable and writeable to only the file owner can edit it, and group and world can only read it:

$ chmod -v g-w,o-w file.txt
mode of 'file.txt' changed from 0666 (rw-rw-rw-) to 0644 (rw-r--r--)

A common permission set is to give the owner and group the same permissions, such as read-write, and to exclude other:

$ chmod -v u=rw,g=rw,o-r file.txt
mode of 'file.txt' changed from 0644 (rw-r-r--) to 0660 (rw-rw----)

Commands and scripts require the executable bit to be set. This example adds the executable bit to the existing permissions for the file owner:

$ chmod -v u+x file.txt
mode of 'file.sh' changed from 0660 (rw-rw----) to 0760 (rwxrw----)

The = operator is useful for overwriting existing permissions:

$ chmod -v u=rw,g=rw,o=r file.txt
mode of 'file.sh' changed from 0760 (rwxrw----) to 0664 (rw-rw-r--)

Discussion

The key to using chmod’s symbolic notation reliably is to always be explicit and to be mindful of the existing permissions. Add and subtract from existing permissions (except with the = operator, which overwrites), and specify u, g, o, or -a.

symbolic notation is designed to be mnemonic, r for read, w for write, and x for execute (Table 6-5).

The notation for users and groups is also mnemonic (Table 6-6).

Just like octal notation, symbolic notation also supports the special modes (see Recipe 6.11).

There are 10 values in symbolic notation, and unset values (which mean no permissions) are represented by a dash, like this example for Duchess’s home directory:

$ stat --format=%a:%A:%U:%G /home/duchess
755:drwxr-xr-x:duchess:duchess

In drwxr-xr-x, the d indicates that this is a directory. There is no comparable value in octal notation.

The remaining nine values are divided into three triads, and the three values in each triad represent read, write, and execute.

See Also

• man 1 chmod

Problem

You want to set special modes with chmod’s symbolic notation.

Solution

The special modes include the sticky bit, setuid, and setgid. These are all set in the executable fields. (See the end of the Discussion in Recipe 6.10 if you are not sure what the executable fields are.)

The sticky bit is applied to directories that contain files owned by multiple users to prevent nonowners from moving, renaming, or deleting the files:

$ chmod o+t /shared/stickydir
mode of '/shared/stickydir' changed from 0775 (rwxrwxr-x) to 1775 (rwxrwxr-t)

Apply setgid to a directory to set all newly created files in the directory to the same group as the directory. This is a nice trick for enforcing correct ownership in a shared directory:

$ chmod -v g+s /shared
mode of '/shared' changed from 0770 (rwxrwx---) to 2770 (rwxrws---)

Apply setuid to an executable file to allow nonroot users to run the executable:

$ chmod -v u+s backup-script
mode of 'backup-script' changed from 0755 (rwxr-xr-x) to 4755 (rwsr-xr-x)

setuid and setgid have the potential to open security holes; see the Discussion to learn more.

Discussion

setuid is useful for executable files.

setgid is useful for directories and files.

The sticky bit is only for directories.

Table 6-7 shows the relationship between owners and modes.

A more descriptive name for the sticky bit is restricted deletion bit. This prevents users from removing or renaming a file in a directory unless they own the file. You can see this on your /tmp directory, which is world readable and writeable, and contains files for multiple users. Using the sticky bit prevents users from moving, renaming, or deleting files they do not own:

$ stat --format=%a:%A:%U:%G /tmp
1777:drwxrwxrwt/:root:root 

setgid means set group identification, and setuid is set user identification. These are used to elevate the permissions of an unprivileged user to the same as the file owner. This is how unprivileged users can use the passwd command to change their own passwords, even though only root has write permissions on /etc/passwd, and everyone else has only read and execute permissions:

$ stat --format=%a:%A:%U:%G /usr/bin/passwd
4755:-rwsr-xr-x:root:root

rws in the user fields means read, write, and execute for all users, with the same permissions as the file owner.

setgid and setuid have the potential to create security holes. It is a best practice to use them only when you can’t devise a safer way to accomplish what you want to do, such as using group assignments or sudo.

See Also

• man 1 chmod

Problem

You want to set permissions on more than one file at a time.

Solution

chmod supports operating on lists of files. You can also use the find command and shell wildcards to select the files you want to change.

The following example takes a space-delimited list of files and makes them all read-only for everyone:

$ chmod -v 444 file1 file2 file3

Set permissions for a directory and its contents, including subdirectories, with the -R (recursive) flag:

$ chmod -vR 755 /shared

You may use wildcards to select files; for example, to make all .txt files in the current directory readable and writable to the owner, and to make group and other readable:

$ chmod -v 644 *.txt

Use a wildcard to select all filenames that start with the same string:

$ chmod -v 644 abcd*

This example makes all files in the current directory read-write for the owner and group, without changing permissions on the directory:

$ find . -type f -exec chmod -v 660 {} \;

You can change the mode of all files belonging to a particular user. You may name the user with either their numeric ID or username. This example starts at the root of the filesystem:

$ sudo find / -user madmax -exec chmod -v 660 {} \;
$ sudo find / -user 1007 -exec chmod -v 660 {} \;

Discussion

You need root privileges to search for files in all directories.

The dot (find .) tells find to start its search in the current directory. You can start your search in any directory.

-type limits the results to files, and not directories.

-user looks for files owned by the specified user.

-exec chmod -v 660 {} \; is a fabulous little incantation that takes the results of the find search and runs the chmod -v 660 command on the results. You can use this for pretty much any command that you want to apply to the results of a find search.

See Also

• man 1 chmod

• man 1 find

Problem

You need to change ownership on a file or directory.

Solution

Use the chown (change owner) command to change file ownership. The basic command syntax is chown user:group filename. You may change only the owner, chown user: filename, or only the group, chown :group filename.

Changing the owner requires root privileges:

duchess@client1:~$ sudo chown -v madmax: song.wav
changed ownership of 'song.wav' from duchess:duchess to madmax:duchess

Change the group owner:

$ sudo chown -v :composers song.wav
changed ownership of 'song.wav' from madmax:duchess to :composers

Change both the user and group owner:

$ sudo chown stash:stash song.wav

Discussion

You need root privileges to make changes to files you do not own and to transfer file ownership to another user. You can change group file ownership without root privileges when you belong to both the original group and the new group.

The colon is optional when you change only the owner and required when you change the group.

See Also

• man 1 chown

Problem

You want to change ownership of directories and their contents, or just the contents of directories, a list of files, or change ownership of files from one user to another.

Solution

chown supports operating on lists of files. You can also use the find command and shell wildcards to list the files you want to change.

To change the owner of several files at once with chown, use a space-delimited list:

$ sudo chown -v madmax:share file1 file2 file3

Change files with a certain file extension in the current directory to a new group:

$ sudo chown -v :share *.txt

Give all of a user’s files in a directory to another user, using their numeric UIDs or usernames:

$ chown -Rv --from duchess stash /shared/compositions

$ chown -Rv --from 1001 1005 /shared/compositions

Use the -find_ command to traverse the entire filesystem, or any directory and its subdirectories, to give all of a user’s files to another user:

$ sudo find / -user duchess -exec chown -v stash {} \;

$ sudo find / -user 1001 -exec chown -v 1005 {} \;

Discussion

Transferring ownership of all of a user’s files to another user, or to a different group, is useful for cleaning up after users who no longer have accounts on the system.

See Also

• man 1 chown

Problem

You want to understand why files are created with a certain set of default permissions, and how to configure the defaults yourself.

Solution

The umask (user file-creation mode mask) controls this behavior. To see what yours is, run the umask command:

$ umask
0002

This is how it looks it in symbolic notation:

$ umask -S
u=rwx,g=rwx,o=rx

This sets your default permissions to 0775 for directories and 0664 for files, because the umask “masks” the hardcoded default permissions of 0777 and 0666. Or you can think of it as subtraction, 0777 - 0002 = 0775.

To change your umask temporarily for the duration of your current session, set it this way:

$ umask 0022

Set the umask permanently by inserting the line umask 0022, or whatever value you want, in your ~/.bashrc file.

Set the default umask for all of your users in /etc/login.defs:

UMASK 022

Table 6-8 shows some common umask values.

Discussion

umask is a Bash shell built-in, and not an executable program stored in /bin, /usr/bin, or any of the other bin (binary) directories.

Table 6-8 lists some commonly used umask values.

See Also

• man 1 chmod

• See the Shell Builtin Commands section of man 1 bash to learn more about umask and other Bash built-in commands

Problem

You want to create shortcuts or links to files.

Solution

There are two types of links in Linux: soft links and hard links. Soft links are for files and directories. Hard links are only for files.

Use the ln (link) command to create soft and hard links. The following example creates a soft link to an external directory, /files/userstuff, in Mad Max’s home directory:

$ ln -s /files/userstuff stuff

/files/userstuff is the target, and stuff is the destination, or soft link name. You can name your soft links anything you want, and move and delete them without affecting their targets. When you open a soft link, it behaves the same way as opening the target.

Hard links are copies of files. The default for the ln command is to create hard links:

$ ln /files/config1.txt myconf.txt

Discussion

Soft links are for files and directories, while hard links are only for files.

$ stat stuff
  File: stuff -> /files/userstuff
  Size: 4               Blocks: 0          IO Block: 4096   symbolic link
Device: 804h/2052d      Inode: 877581      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/ madmax) Gid: ( 1000/ madmax)

$ ls -l
[...]
lrwxrwxrwx 1 madmax madmax  4 Apr 26 12:42 stuff -> /files/userstuff

$ ls -li
1353 -rw-rw-r--   3 madmax madmax  11208 Apr 26 13:06  config.txt
1353 -rw-rw-r--   3 madmax madmax  11208 Apr 26 13:06  config2.txt
1353 -rw-rw-r--   3 madmax madmax  11208 Apr 26 13:06  config3.txt

$ stat config3.txt
  File: config3.txt
  Size: 11208           Blocks: 24         IO Block: 4096   regular file
Device: 804h/2052d      Inode: 1353        Links: 3

$ find /etc -xdev -samefile config3.txt
./config
./config2
./config3

$ df -i /dev/sda4
Filesystem        Inodes  IUsed     IFree IUse% Mounted on
/dev/sda4      384061120 389965 383671155    1% /home

See Also

• man 1 ls

Problem

You want to hide some files and directories so that nobody can see them.

Solution

To hide files so nobody can see them, put them on a storage device only you have access to.

To reduce clutter in your file manager, use dot files to ignore files. You already have these. Look for a setting like “Show hidden files” in your graphical file manager, or use the -a option for ls:

$ ls -a
.
..
Audiobooks
.bash_history
.bash_logout
.bashrc
bin
.bogofilter
.cache
Calibre-Library
cat-memes
.cddb
.cert

Prefixing any file with a dot makes it a hidden file, though it is really not hidden, but ignored until you want to see it. This is used mainly in users’ home directories to reduce clutter by not displaying configuration files. These are normal files you can edit, delete, or whatever you want.

Discussion

Note the single and double dots at the top of the file list. The single dot represents the current directory, and the double dot represents the parent directory. Try it with the cd command. The first example stays in the current directory, the second example changes to the parent directory:

stash@client4:~$ cd .
stash@client4:~$

stash@client4:~$ cd ..
stash@client4:/home$

Run cd with no options to return to your home directory, or cd - to return to the last directory you were in.

See Also

• See the Shell Builtin Commands section of man 1 bash to learn more about cd and other Bash built-in commands

Problem

You’re not sure which files you should back up. Do you need to make backups of your system files? Do you really need to back up all of your personal files? Are there files you should not back up?

Solution

Any file that you would be sorry to lose is a file you need to back up. Your personal files and system data files are the most important. Restoring system files such as commands, applications, and libraries is less important because you can always download and reinstall these.

The following directories contain files such as configurations; data files for servers such as web, FTP, and mail servers; log files; applications installed in nonstandard locations; and shared directories, all of which should be backed up:

• /boot/grub, if it contains any customizations such as themes, background images, or fonts.

• /etc contains system configuration files.

• /home, users’ personal files.

• /mnt, temporary filesystem mountpoints. Back this up if you have mountpoints you want to preserve.

• /opt, for proprietary or other applications not installed the standard way.

• /root, the root user’s personal files.

• /srv, data for servers such as web, FTP, and rsync servers.

• /tmp holds temporary data that is automatically updated or deleted as needed. Some of the data in /tmp is persistent, for example user-created files and some system services, and they should be backed up.

• /var stores many types of data such as log files, mail spools, cron jobs, and data for system services, though most distros have migrated to using /srv for system services.

If you have any shared directories, custom commands and scripts, or any data files or directories not listed previously, back them up.

/proc, /sys, and /dev are pseudofilesystems that exist only in memory and should not be backed up.

/media is for mounting removable storage media and should be managed by the system, so there is no need to back it up. If you are manually creating mountpoints in /media, they really need to be moved to /mnt.

Many databases should not be backed up with simple copying because they have special utilities and procedures for making copies and backups, and for restoring from backups. Use the tools made for your databases. Some examples are PostrgreSQL, MariaDB, and MySQL.

Copying everything is the easy way if your backup storage is large enough. You also have the option of fine-tuning your file selection by creating lists of files to copy or to exclude; see Recipes 7.8 and 7.9.

Discussion

Storage media is so cheap now you may not have to care about conserving storage space. If you need to be mindful of storage limitations, see the recipes in this chapter on file selection.

See Also

• Filesystem Hierarchy Standard

Problem

You are restoring files from backup, and you want to know if there are files that should not be restored.

Solution

Some files should not be restored, depending on the circumstances.

Do not restore /etc/fstab after reinstalling Linux (the file that configures your static filesystem mounts). Every time you install Linux, all the filesystems get new Universal Unique Identifiers (UUIDs), so they will not be recognized and your new installation will fail.

Be careful restoring any file in /etc or the dotfiles (such as /home/.config or /home/.local) in your home directory. If you are restoring from backup to a new installation of a different release, or a different Linux distribution, there may be incompatibilities in configuration options or file locations. Restore them one at a time so you can quickly spot any problems.

See Also

• Chapter 1

Problem

You want to know the easiest, simplest way to make regular backups to a local USB storage device.

Solution

The answer is to use simple copying. Get yourself a nice USB hard drive or USB stick. Plug it in and use your file manager to copy your files. Easy peasey, no muss, no fuss, and it is dead simple to restore your files. Or, use the cp command (see Recipe 7.4).

Discussion

Simple copying doesn’t scale all that well, but for a few devices, like a PC, laptop, and phone, it works fine. The important part is making regular backups, verifying that you can restore files from your backups, and not worrying if you’re being nerdy enough.

Back in olden times, backups were more complicated because storage was expensive, so backup programs used a lot of tricks to save space. Now you can buy multi-terabyte external USB 3.0 hard drives for less than $200.

See Also

• man 1 cp

Problem

You like using simple copying for making your backups to an external USB storage drive, and you want to automate the process.

Solution

This calls for the cp command and crontab to schedule your backups.

You may list individual files and directories to copy with cp, separated by spaces:

duchess@pc:~$ cp -auv Pictures/cat-desk.jpg Pictures/cat-chair.png \
  ~/cat-pics /media/duchess/2tbdisk/backups/

The following example copies Duchess’s entire home directory to the backups directory on an external USB drive named 2tbdisk:

duchess@pc:~$ cp -auv ~ /media/duchess/2tbdisk/backups/

This creates /media/duchess/2tbdisk/backups/duchess/ on the backup device.

Copy the contents of a directory without copying the directory itself:

duchess@pc:~$ cp -auv /home/duchess/* /media/duchess/2tbdisk/backups/

Create a personal cron job to run your backup every night at 10:30 P.M.:

duchess@pc:~$ crontab -e
# m h  dom mon dow   command
30 22  *   *   *   /bin/cp -au /home/duchess /media/duchess/2tbdisk/backups/

Discussion

If you want to preserve file attributes such as ownership and permissions, format your backup drive with a Linux filesystem that supports file attributes, such as Ext4, XFS, or Btrfs (see Chapter 11). The FAT filesystems do not preserve ownership or permissions.

Keep an eye on how long it takes your backup to run. If it takes longer than your scheduled backup interval, cron will start the next backup on schedule, and then you have a mess.

The first run takes the longest because all the files are new. Subsequent backups will go faster as only new files and files with newer timestamps will be copied.

The tilde, ~, is a shortcut for the current user’s home directory, so in this recipe it is short for /home/duchess.

The asterisk in /home/duchess/* means copy all the files in /home/duchess, but not the directory /home/duchess.

The -a, -u, and -v options for cp mean:

• -a, --archive recursively copies and preserves all file attributes: mode, ownership, timestamps, and extended attributes.

• -u, --update tells cp to copy only files with newer timestamps than the copies in the backup directory, or new files that have not been backed up yet.

• -v, --verbose prints activity messages during the copy operation.

Some other useful options:

• -R, -r recursive; use this to copy directories when you do not use the -a option. -a preserves file attributes, -R, -r does not. FAT and exFAT filesystems do not support file attributes, so use -R, -r with these.

• --parents creates missing parent directories on the destination.

• -x, --one-file-system This prevents recursing into other partitions and mounted network filesystems. For example, if you have an NFS share mounted you may not want to add it to your backup.

Most Linux distributions mount USB devices in /run/media or /media. The easy way to find the filepath for your USB drive is to look in your file manager or use the lsblk command:

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
[...]
sdb      8:16   0  1.8T  0 disk
└─sdb1   8:17   0  1.5T  0 part /media/duchess/backups

See Also

• Recipe 3.7 to learn more about using cron

• man 1 crontab

• man 1 cp

Problem

You want to make backups to a USB stick or USB hard drive, and you want something that is faster and more efficient than simple copying. You also want simple file restorations that you can make with standard Linux tools, without needing special software.

Solution

rsync is what you want. It keeps filesystems synchronized, both local and remote. rsync is fast and efficient, as it transfers only the changes in files, and you can restore files with the rsync command, the cp command, your file manager, or whatever copying tool you prefer.

The following example shows how to back up a home directory. First, name your source directory, which is the directory you want to back up, then name the destination directory. This example copies Duchess’s /home to a USB drive named 2tbdisk:

duchess@pc:~$ rsync -av ~ /media/duchess/2tbdisk/
sending incremental file list
duchess/
duchess/Documents/
duchess/Downloads/
duchess/Music/
[...]

sent 27,708,209 bytes  received 20,948 bytes  11,091,662.80 bytes/sec
total size is 785,103,770,793  speedup is 28,313.29

You can specify two or more directories, in a space-delimited list, to transfer to the destination directory:

duchess@pc:~$ rsync -av ~/arias ~/overtures /media/duchess/2tbdisk/duchess/

Copy files from your backup device to your computer by reversing the source and destination:

duchess@pc:~$ rsync -av /media/duchess/2tbdisk/duchess/arias /home/duchess/

You may safely test your rsync command, without copying any files, with the --dry-run option:

duchess@pc:~$ rsync -av --dry-run \
~/Music/scores ~/Music/woodwinds /media/duchess/2tbdisk/duchess/

If any files are deleted from a source directory, rsync will not delete them from the destination directory unless you explicitly tell it to with the delete option:

duchess@pc:~$ rsync -av --delete /home/duchess /media/duchess/2tbdisk/

Discussion

The tilde, ~, is a shortcut for your home directory, so in the examples it means /home/duchess.

Command examples with line breaks use a slash, \, to indicate that the command continues on the next line. You can copy the whole command, with the slashes, and it should work.

If you have networked filesystems mounted on your PC, such as NFS or Samba, use the -x option to copy only from your local filesystem, and not recurse into the remote filesystems.

Adding a trailing slash, ~/, (/home/duchess/) copies only the contents of the duchess/ directory, but not the directory itself, resulting in /media/duchess/2tbdisk/[files]. Omitting the trailing slash transfers the contents of /home/duchess and the duchess directory, resulting in /media/duchess/2tbdisk/duchess/[files]. The trailing slash only matters on the source directory, and it makes no difference on the destination directory.

Don’t feel bad if you have to count on your fingers or make a lot of test runs to remind yourself how the trailing slash behaves, because this vexes everyone. It might help to think of the trailing slash as a little fence that prevents the source directory from escaping.

The -a and -v options for rsync mean:

• -a, --archive retains the mode, timestamps, permissions, and ownership, and copies recursively. This is the same as *-rlptgoD*, which copies recursively, copies symlinks, preserves permissions, preserves file modification times, preserves file ownership, and preserves special files, such as device files.

• -v, --verbose displays activity messages.

You may wish to use some of these options:

• -q, --quiet suppresses nonerror messages.

• --progress shows information on each file as it is transferred.

• -A, --als preserves access control lists (ACLs).

• -X, --xattrs preserves extended file attributes (xattrs).

Your filenames, of course, will be different than the examples. 2tbdisk is a filesystem label created by the user (see Recipe 9.4). It is an abbreviation for “2 terabyte disk.” If you do not create a label, udev creates one, for example /media/duchess/488B-7971/.

You may use the normal Linux tools, such as rsync, your file manager, or the cp command, to restore files.

See Also

• man 1 rsync

Problem

You want to use rsync to copy files to another computer on your local network or over the internet, and you want encrypted transport and authentication.

Solution

rsync uses SSH by default when you transfer files to another machine. The remote machine must be running an SSH server, and the source machine must have an SSH client already set up (see Chapter 12).

This example transfers files over the local network from Duchess’s PC to her laptop. Duchess’s username on her laptop is Empress, and she is copying files from her home directory on her PC to her home directory on her laptop:

duchess@pc:~$ rsync -av ~/Music/arias empress@laptop:songs/
duchess@laptop's password:
building file list ... done
arias/
arias/o-mio-babbino-caro.ogg
arias/deh-vieni-non-tardar.ogg
arias/mi-chiamano-mimi.ogg
wrote 25984 bytes  read 68 bytes  7443.43 bytes/sec
total size is 25666  speedup is 0.99

If the destination directory does not exist, rsync will create it.

To upload files over the internet, use the fully qualified domain name of the server you are logging in to:

duchess@pc:~$ rsync -av ~/Music/woodwinds \
 empress@remote.example.com:/backups/

The syntax for copying files from a remote host is reversed. This example copies the /woodwinds directory and its contents from the remote host to Duchess’s home directory:

duchess@pc:~$ rsync -av empress@remote.example.com:/backups/woodwinds \
 /home/duchess/Music/

Discussion

You might remember when the SSH option had to be explicit—for example, rsync -a -e ssh [options]. This is no longer necessary.

You may find some of these options to be useful:

• --partial preserves partially downloaded files when the network connection is interrupted, and resumes the file transfer from where it left off when the connection is restored.

• -h, --human-readable displays file sizes in kilobytes, megabytes, and gigabytes, rather than bytes.

• --log-file= stores a complete record of each transfer in a text file. Put them all together like this:

duchess@pc:~$ rsync --partial --progress \
 --log-file=/home/duchess/rsynclog.txt \
 -hav ~/Music/arias empress@remote.example.com:/backups/

Both authentication and transport are encrypted by SSH. Users need shell accounts on all machines they are going to transfer files to. See Chapter 12 to learn about secure remote administration with SSH.

Consider setting up a central backup server for simplifying administration. Your users have their own accounts with their own /home directories and can manage their own backups and restores without bothering you.

Another option for a backup server is to run rsync as a service. The advantage of this is your rsync users do not need login accounts on the server. One disadvantage is it does not support encrypted transfers. See Recipe 7.13 to learn about this.

See Also

• man 1 rsync

• Recipe 12.5

• Recipe 12.7

Problem

You want to create crontabs to automatically run your secure rsync transfers.

Solution

You need SSH set up for passwordless authentication on the destination machine (see Recipes 12.10 and 12.11) and network access to the destination machine for the clients.

Then use /etc/crontab for transfers that require root permissions. The following example makes a backup of /etc every night at 10 P.M. to a LAN server named server1:

# m h dom mon dow user  command
00 22 * * * root /usr/bin/rsync -a /etc server1:/system-backups

Use personal crontabs for transferring your own files (see Recipe 3.7).

Discussion

OpenSSH is a lovely tool that provides secure network transfers for a host of tasks. Anything that you run over a network can probably run over SSH.

See Also

• Chapter 12

• Recipe 12.10

• Recipe 12.11

Problem

So far the examples have shown how to transfer entire directories. You want to know how to exclude files and directories from being copied.

Solution

For simplicity, the following examples demonstrate local transfers to a USB drive, but they also work for remote transfers over SSH; see Recipe 7.6.

When it’s just a few files, you can list them on the command line using --exclude=. This example excludes one file from /home/duchess/Music/arias:

duchess@pc:~$ rsync -av --exclude=lho-perduta.wav \
 ~/Music/arias /media/duchess/2tbdisk/duchess/Music/

This is nice and easy and reliable. However, there is a gotcha: if there are multiple files in your source directory with the same name as your excluded file, all of them will be excluded. If you do not want the duplicates to be excluded, you need to specify which one is to be excluded. In the following example you want to exclude only the copy in the arias/ source directory:

duchess@pc:~$ rsync -av --exclude=arias/lho-perduta.wav \
 ~/Music/arias /media/duchess/2tbdisk/duchess/Music/

Exclude more than one file by enclosing them in curly braces, separated with single quotes and commas. There must be no spaces between the equals sign and the curly brace, and no spaces between the commas and single quotes:

duchess@pc:~$ rsync -av \
--exclude={'arias/lho-perduta.wav','non-mi-dir.wav','un-bel-di-vedremo.flac'} \
~/Music/arias /media/duchess/2tbdisk/duchess/Music/

Excluding directories works the same way as excluding files, and you can mix files and directories in your exclude list:

duchess@pc:~$ rsync -av \
--exclude={'soprano/','tenor/','non-mi-dir.wav'} \
~/Music/arias /media/duchess/2tbdisk/duchess/Music/

See Recipe 7.11 to learn how to put your excludes list in a file.

Discussion

The root directory in an rsync transfer is the top-level directory you are transferring files from. In the examples in this recipe, that is ~/Music/arias. rsync checks all the files and directories in your root directory, and compares them to your exclude directives, which rsync calls patterns. Patterns are checked against the files and directories in your root directory, starting at the root and proceeding down through the directory hierarchy. Every time a pattern is matched, it is excluded from the transfer. If the pattern arias/lho-perduta.wav is duplicated in another location, such as 2arias/lho-perduta.wav, it will also be excluded. When a pattern ends with a slash (/), rsync will match only directories.

See Also

• man 1 rsync

• Recipe 7.10

Problem

You want to include a selected set of files in your backup, rather than defining a list of files to exclude.

Solution

When you want to back up just a few files, you can do this on the command line. --include= operates differently than --exclude= because it doesn’t really mean “include,” it means “do not exclude.” It needs two additional options, --include=*/ and --exclude='*', as this example of transferring a single file shows:

duchess@pc:~$ rsync -av --include=*/ --include=lho-perduta.wav \
 --exclude='*' ~/Music/arias /media/duchess/2tbdisk/duchess/Music/

You may transfer a list of files:

duchess@pc:~$ rsync -av --include=*/ \
--include={'lho-perduta.wav','non-mi-dir.wav','un-bel-di-vedremo.flac'} \
--exclude='*' ~/Music/arias /media/duchess/2tbdisk/duchess/Music/

There must be no spaces between the equals sign and the curly brace, and no spaces between the commas and single quotes.

If there are multiple files with the same name in different locations in your source directory, rsync will transfer all of them. In this example only /home/duchess/Music/arias/sopranos/lho-perduta.wav is transferred because the pattern soprano/lho-perduta.wav is unique in /Music/arias:

duchess@pc:~$ rsync -av --include=*/ --include=soprano/lho-perduta.wav
--exclude='*' ~/Music/arias /media/duchess/2tbdisk/duchess/Music/
Music/
Music/arias/
Music/arias/baritone/
Music/arias/soprano/
Music/arias/soprano/lho-perduta.wav
Music/arias/tenor/
[...]

That transfers only a single file, but all the subdirectories in ~/Music/arias are copied. Use the -m, --prune-empty-dirs option to prevent copying empty directories, like this example:

duchess@pc:~$ rsync -avm --include=*/ --include=soprano/lho-perduta.wav
--exclude='*' ~/Music/arias /media/duchess/2tbdisk/duchess/Music/
Music/
Music/arias/soprano/
Music/arias/soprano/lho-perduta.wav

When you have more than a few files to include, store your list in a plain-text file (see Recipes 7.10 and 7.11).

Discussion

--include=*/ tells rsync to traverse your entire source directory.

--include=[files] means do not exclude these files.

--exclude='*' tells rsync to exclude everything not included.

Remember that all filepaths are relative to your source directory and not your system’s root directory.

See Also

• man 1 rsync

• Recipe 7.10

• Recipe 7.11

Problem

Your includes are too many for a command-line incantation, and you want to maintain your list in a file that rsync can read. You also want this to be simple, if possible, thanks to past experience with rsync include/exclude files that never would work right.

Solution

The simplest way to maintain a list, without going crazy over figuring out rsync’s include/exclude syntax, is to create a plain list of files that you provide to the --files-from= option. You don’t have to worry about getting them in the right order, or using rsync’s filter notation, just a plain list with whatever files and directories you want. The only gotcha is every item in your list must be relative to your source directory. In the following example, all list items are relative to /home/duchess:

# include file list
#
/Documents/compositions/jazz/
/Documents/schedule.odt
/Videos/concerts/
.config
.local
/Music/courses/bassoon.avi</strong>
[...]

Then use the list with the --files-from option:

duchess@pc:~$ rsync -av ~ --files-from ~/include-list.txt \
 duchess@remote.example.com:/backups/

Discussion

This is the easiest way to maintain a list of files and directories to back up. There are no excludes, no wildcards, no funny syntax, just a nice clean understandable list.

When you use the tilde to indicate your home directory, leave off the equals sign from --files-from, like the last example in the recipe.

See Also

• man 1 rsync

• Recipe 7.9

• Recipe 7.11

Problem

You like the simple include file concept in Recipe 7.10, but you really want to have both includes and excludes.

Solution

What you want is an rsync exclude file. An exclude file provides more flexibility and contains both includes and excludes. The following example illustrates a basic configuration. Every item must start by including the source root, which in this example is /home/duchess, and end with excluding the source root:

# exclude file list
#
# include home directory
+ /duchess/
#
# include .config and .local, exclude all other dotfiles
+ /duchess/.config
+ /duchess/.local
- /duchess/.*
#
# include jazz/, exclude all other files in Documents
+ /duchess/Documents/
+ /duchess/Documents/compositions/
+ /duchess/Documents/compositions/jazz/
- /duchess/Documents/compositions/*
- /duchess/Documents/*
#
# include schedule.odt, include all .ogg files in
# arias/, exclude all other files in Music
+ /duchess/Music/
+ /duchess/Music/schedule.odt
+ /duchess/Music/arias/*.ogg
- /duchess/Music/arias/*
- /duchess/Music/*
#
# includes courses/, exclude all other files in Videos
+ /duchess/Videos/
+ /duchess/Videos/courses/
- /duchess/Videos/*
#
# exclude everything else
- /duchess/*

Feed it to rsync with the exclude-from= option:

duchess@pc:~$ rsync -av ~ \
 --exclude-from=/home/duchess/exclude-list.txt \
 /media/duchess/2tbdisk/

Discussion

The exclude-list.txt example demonstrates backing up:

• Two dotfiles, .config and .local

• A single subdirectory of /Documents, /jazz

• A single file in /Music, schedule.odt, and only .ogg files in /Music/arias/

• A single directory in /Videos, /courses

There must be no spaces between lines, and comments (#) are useful for reminding you of the purpose of each section, and for adding a bit of whitespace. Preface your includes with the plus sign and the excludes with the minus sign.

All other files in /home/duchess are excluded from backup. Includes must always come first. As the example file shows, you have to be precise in defining each include/exclude. Includes must be listed in their directory hierarchy order, with all subdirectories. For example, this will fail with all files excluded:

+ /duchess/Documents/compositions/
- /duchess/*

Try including /Documents:

+ /duchess/Documents/
+ /duchess/Documents/compositions/
- /duchess/*

Now all the subdirectories and their contents in /Documents are transferred, and not just /compositions. To copy only /compositions you have to exclude /Documents; it is not enough to exclude only /duchess. The following example copies only /duchess/Documents/compositions/ and nothing else:

+ /duchess/Documents/
+ /duchess/Documents/compositions/
- /duchess/Documents/*
- /duchess/*

You may use wildcards to include or exclude files by type. For example, include all .ogg and .flac files, exclude all .wav files, and exclude all cache and temp directories:

# include home directory
+ /duchess/
#
# include all ogg and flac files
+ *.ogg
+ *.flac
#
# exclude wav files, all cache and temp dirs
- *.wav
- cache*
- temp*

You may have multiple source directories.

There is always just one destination directory.

See also

• man 1 rsync

• Recipe 7.10

Problem

Large file transfers can use a lot of network bandwidth and slow down everything. You want a simple way to restrict rsync’s bandwidth use without implementing something complex like traffic shaping.

Solution

Use rsync’s --bwlimit option. This example limits it to 512 Kbps:

$ rsync --bwlimit=512 -ave ssh ~/Music/arias empress@laptop:songs/

Discussion

--bwlimit only accepts values in kilobits.

See Also

• man 1 rsync

Problem

You want your users to back up their own data on a central backup server, but you don’t want to give them shell accounts on your backup server.

Solution

Set up a central backup server, and run rsync in daemon mode. You should have name services already set up, and the hosts on your network have access to the backup server. Users will not need login accounts on the server because you will use rsync’s own access controls and user authorization to control access to the rsync archives.

rsync must be installed on all machines. rsyncd runs on the backup server, and clients will use the rsync command to connect to the server.

On the backup server, edit or create /etc/rsyncd.conf to create an rsync module defining the archive:

# modules
[backup_dir1]
   path = /backups
   comment = "server1 public archive"
   list = yes
   read only = no
   use chroot = no
   uid = 0
   gid = 0

Create your /backups directory, mode 0700, owned by root, to prevent unauthorized access from anyone who has access to the server:

$ sudo mkdir /backups/
$ sudo chmod 0700 /backups/

Start rsyncd on the server in daemon mode with systemd:

$ sudo systemctl start rsyncd.service

On Debian/Ubuntu it is rsync.service.

If your Linux does not have systemd, start it with the rsync command:

admin@server1:~$ sudo rsync --daemon

On the backup server, test that rsyncd is listening and accepting connections:

admin@server1:~$ rsync server1::
backup_dir1     "server1 public archive"

Then test from another PC on your network using the server’s hostname or IP address:

duchess@pc:~$ rsync server1::
backup_dir1     "server1 public archive"

duchess@pc:~$ rsync 192.168.10.15::
backup_dir1     "server1 public archive"

Now you know that it is ready to transfer files. Test that you can copy files to your new rsyncd server:

duchess@pc:~$ rsync -av ~/drawings server1::backup_dir1
building file list.....done
drawings/
drawings/aug_03
drawings/sept_03

wrote 1126399 bytes  read 104 bytes  1522.0 bytes/sec
total size is 1130228  speedup is 0.94

Now view the nice new uploaded files:

duchess@pc:~$ rsync server1::backup_dir1/drawings/
drwx------    4,096  2021/01/04  06:06:55    .
-rw-r--r--   21,560  2021/09/17  08:53:18    aug_03
-rw-r--r--   21,560  2021/10/14  16:42:16    sept_03

Upload a few more files to the server, then download files to a different computer from the rsyncd server:

madmax@buntu:~$ rsync -av server1::backup_dir1/drawings ~/downloads
receiving incremental file list
created directory /home/madmax/downloads
drawings/
drawings/aug_03
drawings/sept_03

sent 123 bytes  received 11562479 bytes  1755.00 bytes/sec
total size is 1141776 speedup is 1.00

Everything works. Take a break and enjoy your success.

Discussion

This is not a secure form of file transfer because there is no encryption, and anyone on the network can access the files. It is suitable to use on your local network for easy archiving and file sharing.

rsync [hostname]:: needs double colons when connecting to an rsync server running in daemon mode. This tells rsync to look for a module name.

These are the command options in the /etc/rsyncd.conf example:

[backup_dir1]

The module name can be anything you want.

path =

Defines the directory for the module to use.

comment =

This is a brief description to remind you who the module belongs to, or what it is for.

list=yes

Permits users to see a list of files in the module. no hides the module.

read only = no

This allows users to upload files to the server.

use chroot = no

Overrides the default of use chroot = yes. chroot is change root, sometimes called a chroot jail. A chroot jail is a separate environment inside your filesystem that contains its own root filesystem, commands, libraries, and everything else it needs to function. This is not a secure environment, though it is commonly thought of as a security tool. For rsync, the man page describes this as useful protection from configuration errors. The trade-off is rsync is blocked from following symlinks to files outside of the chroot environment, and it complicates preserving UIDs and GIDs by name. As they say, your mileage may vary, and it may be a good option for you. See the use chroot section of rsyncd.conf (5).

Set both uid and gid to root or 0. This preserves UIDs and GIDs, and manages permissions correctly.

If any of your transfers fail, look at the rsync error messages. They will tell you if you made a mistake in your filepaths, misspelled something, or couldn’t connect to the server, and give useful hints for correcting the problem.

If you are running a Linux without systemd, consult its documentation to learn how to start and stop rsyncd.

See Recipe 7.14 to learn how to set up access controls.

See Also

• man 5 rsyncd.conf

Problem

You don’t want an open rsyncd server, and you want users to have their own protected modules that other users cannot access.

Solution

rsyncd comes with its own simple authentication and access controls. Create a new file containing username/password pairs, and add auth users and secrets file directives to /etc/rsyncd.conf.

First create the password file. In the following example, /etc/rsyncd-users sets up three users and their passwords:

# rsync-users for server1
duchess:12345
madmax:23456
stash:34567

Set permissions to read-write for root-only:

$ sudo chmod 0600 /etc/rsyncd-users

Now create a module for one of your users in /etc/rsyncd.conf. This example creates a module for Duchess, using the /backups/duchess directory on the rsync server:

[duchess_backup]
   path = /backups/duchess
   comment = Duchess's private archive
   list = yes
   read only = no
   auth users = duchess
   secrets file =/etc/rsyncd-users
   use chroot = no
   strict modes = yes
   uid = root
   gid = root
   

Create your user’s backup directory, like this example for Duchess, with mode 0700:

$ sudo mkdir /backups/duchess/
$ sudo chmod -R 0700 /backups/duchess/

Now try logging in:

$ rsync duchess@server1::duchess_backup
Password: 12345
drwxr-xr-x      4,096 2020/06/29   18:24:43 .

Try transferring some files:

$ rsync -av ~/logs duchess@server1::duchess_backup
Password:
sending incremental file list
logs/
logs/irc.log
logs/irc_#core-standup.log
logs/irc_#core.log
logs/irc_#desktop.log
logs/irc_#engineering.log
logs/irc_#mobile.log

sent 130,507 bytes  received 305 bytes  37,374.86 bytes/sec
total size is 129,383  speedup is 0.99

It worked! If the file transfer fails, check the rsync log to learn why. On systemd Linuxes, read the most recent log entries in the status output:

$ systemctl status rsyncd.service

On other Linux distributions the rsyncd log should be in /var/log.

Discussion

The username/password pairs are arbitrary and are not related to system user accounts. rsyncd users have no access to the host system outside of their rsync shares.

For additional security, add these directives to /etc/rsyncd.conf:

hosts allow

Use this to list hosts that are allowed to access the rsyncd archives. For example, you can limit access to hosts on a single subnet:

hosts allow = *.local.net
hosts allow = 192.168.1.

All hosts not allowed are denied, so you don’t need a hosts deny directive.

hosts deny

This usually isn’t needed, if you use hosts allow. It is useful for denying access to specific hosts that cause annoyance.

The password file is in cleartext, so it must be restricted to the superuser.