Command #1: wc

The wc command prints the number of lines, words, and characters in a file:

$ wc animals.txt
  7  51 325 animals.txt

wc reports that the file animals.txt has 7 lines, 51 words, and 325 characters. If you count the characters by eye, including spaces and tabs, you’ll find only 318 characters, but wc also includes the invisible newline character that ends each line.

The options -l, -w, and -c instruct wc to print only the number of lines, words, and characters, respectively:

$ wc -l animals.txt
7 animals.txt
$ wc -w animals.txt
51 animals.txt
$ wc -c animals.txt
325 animals.txt

Counting is such a useful, general-purpose task that the authors of wc designed the command to work with pipes. It reads from stdin if you omit the filename, and it writes to stdout. Let’s use ls to list the contents of the current directory and pipe them to wc to count lines. This pipeline answers the question, “How many files are visible in my current directory?”

$ ls -1
animals.txt
myfile
myfile2
test.py
$ ls -1 | wc -l
4

The option -1, which tells ls to print its results in a single column, is not strictly necessary here. To learn why I used it, see the sidebar “ls Changes Its Behavior When Redirected”.

wc is the first command you’ve seen in this chapter, so you’re a bit limited in what you can do with pipes. Just for fun, pipe the output of wc to itself, demonstrating that the same command can appear more than once in a pipeline. This combined command reports that the number of words in the output of wc is four: three integers and a filename:

$ wc animals.txt
  7  51 325 animals.txt
$ wc animals.txt | wc -w
4

Why stop there? Add a third wc to the pipeline and count lines, words, and characters in the output “4”:

$ wc animals.txt | wc -w | wc
      1       1       2

The output indicates one line (containing the number 4), one word (the number 4 itself), and two characters. Why two? Because the line “4” ends with an invisible newline character.

That’s enough silly pipelines with wc. As you gain more commands, the pipelines will become more practical.

Command #2: head

The head command prints the first lines of a file. Print the first three lines of animals.txt with head using the option -n:

$ head -n3 animals.txt
python	Programming Python	2010	Lutz, Mark
snail	SSH, The Secure Shell	2005	Barrett, Daniel
alpaca	Intermediate Perl	2012	Schwartz, Randal

If you request more lines than the file contains, head prints the whole file (like cat does). If you omit the -n option, head defaults to 10 lines (-n10).

By itself, head is handy for peeking at the top of a file when you don’t care about the rest of the contents. It’s a speedy and efficient command, even for very large files, because it needn’t read the whole file. In addition, head writes to stdout, making it useful in pipelines. Count the number of words in the first three lines of animals.txt:

$ head -n3 animals.txt | wc -w
20

head can also read from stdin for more pipeline fun. A common use is to reduce the output from another command when you don’t care to see all of it, like a long directory listing. For example, list the first five filenames in the /bin directory:

$ ls /bin | head -n5
bash
bsd-csh
bunzip2
busybox
bzcat

Command #3: cut

The cut command prints one or more columns from a file. For example, print all book titles from animals.txt, which appear in the second column:

$ cut -f2 animals.txt
Programming Python
SSH, The Secure Shell
Intermediate Perl
MySQL High Availability
Linux in a Nutshell
Cisco IOS in a Nutshell
Writing Word Macros

cut provides two ways to define what a “column” is. The first is to cut by field (-f), when the input consists of strings (fields) each separated by a single tab character. Conveniently, that is exactly the format of the file animals.txt. The preceding cut command prints the second field of each line, thanks to the option -f2.

To shorten the output, pipe it to head to print only the first three lines:

$ cut -f2 animals.txt | head -n3
Programming Python
SSH, The Secure Shell
Intermediate Perl

You can also cut multiple fields, either by separating their field numbers with commas:

$ cut -f1,3 animals.txt | head -n3
python	2010
snail	2005
alpaca	2012

or by numeric range:

$ cut -f2-4 animals.txt | head -n3
Programming Python	2010	Lutz, Mark
SSH, The Secure Shell	2005	Barrett, Daniel
Intermediate Perl	2012	Schwartz, Randal

The second way to define a “column” for cut is by character position, using the -c option. Print the first three characters from each line of the file, which you can specify either with commas (1,2,3) or as a range (1-3):

$ cut -c1-3 animals.txt
pyt
sna
alp
rob
hor
don
ory

Now that you’ve seen the basic functionality, try something more practical with cut and pipes. Imagine that the animals.txt file is thousands of lines long, and you need to extract just the authors’ last names. First, isolate the fourth field, author name:

$ cut -f4 animals.txt
Lutz, Mark
Barrett, Daniel
Schwartz, Randal
⋮

Then pipe the results to cut again, using the option -d (meaning “delimiter”) to change the separator character to a comma instead of a tab, to isolate the authors’ last names:

$ cut -f4 animals.txt | cut -d, -f1
Lutz
Barrett
Schwartz
⋮

Command #4: grep

grep is an extremely powerful command, but for now I’ll hide most of its capabilities and say it prints lines that match a given string. (More detail will come in Chapter 5.) For example, the following command displays lines from animals.txt that contain the string Nutshell:

$ grep Nutshell animals.txt
horse	Linux in a Nutshell	2009	Siever, Ellen
donkey	Cisco IOS in a Nutshell	2005	Boney, James

You can also print lines that don’t match a given string, with the -v option. Notice the lines containing “Nutshell” are absent:

$ grep -v Nutshell animals.txt
python	Programming Python	2010	Lutz, Mark
snail	SSH, The Secure Shell	2005	Barrett, Daniel
alpaca	Intermediate Perl	2012	Schwartz, Randal
robin	MySQL High Availability	2014	Bell, Charles
oryx	Writing Word Macros	1999	Roman, Steven

In general, grep is useful for finding text in a collection of files. The following command prints lines that contain the string Perl in files with names ending in .txt:

$ grep Perl *.txt
animals.txt:alpaca      Intermediate Perl       2012    Schwartz, Randal
essay.txt:really love the Perl programming language, which is
essay.txt:languages such as Perl, Python, PHP, and Ruby

In this case, grep found three matching lines, one in animals.txt and two in essay.txt.

grep reads stdin and writes stdout, making it great for pipelines. Suppose you want to know how many subdirectories are in the large directory /usr/lib. There is no single Linux command to provide that answer, so construct a pipeline. Begin with the ls -l command:

$ ls -l /usr/lib
drwxrwxr-x  12 root root    4096 Mar  1  2020 4kstogram
drwxr-xr-x   3 root root    4096 Nov 30  2020 GraphicsMagick-1.4
drwxr-xr-x   4 root root    4096 Mar 19  2020 NetworkManager
-rw-r--r--   1 root root   35568 Dec  1  2017 attica_kde.so
-rwxr-xr-x   1 root root     684 May  5  2018 cnf-update-db
⋮

Notice that ls -l marks directories with a d at the beginning of the line. Use cut to isolate the first column, which may or may not be a d:

$ ls -l /usr/lib | cut -c1
d
d
d
-
-
⋮

Then use grep to keep only the lines containing d:

$ ls -l /usr/lib | cut -c1 | grep d
d
d
d
⋮

Finally, count lines with wc, and you have your answer, produced by a four-command pipeline—/usr/lib contains 145 subdirectories:

$ ls -l /usr/lib | cut -c1 | grep d | wc -l
145

Command #5: sort

The sort command reorders the lines of a file into ascending order (the default):

$ sort animals.txt
alpaca	Intermediate Perl	2012	Schwartz, Randal
donkey	Cisco IOS in a Nutshell	2005	Boney, James
horse	Linux in a Nutshell	2009	Siever, Ellen
oryx	Writing Word Macros	1999	Roman, Steven
python	Programming Python	2010	Lutz, Mark
robin	MySQL High Availability	2014	Bell, Charles
snail	SSH, The Secure Shell	2005	Barrett, Daniel

or descending order (with the -r option):

$ sort -r animals.txt
snail	SSH, The Secure Shell	2005	Barrett, Daniel
robin	MySQL High Availability	2014	Bell, Charles
python	Programming Python	2010	Lutz, Mark
oryx	Writing Word Macros	1999	Roman, Steven
horse	Linux in a Nutshell	2009	Siever, Ellen
donkey	Cisco IOS in a Nutshell	2005	Boney, James
alpaca	Intermediate Perl	2012	Schwartz, Randal

sort can order the lines alphabetically (the default) or numerically (with the -n option). I’ll demonstrate this with pipelines that cut the third field in animals.txt, the year of publication:

$ cut -f3 animals.txt                         Unsorted
2010
2005
2012
2014
2009
2005
1999
$ cut -f3 animals.txt | sort -n               Ascending
1999
2005
2005
2009
2010
2012
2014
$ cut -f3 animals.txt | sort -nr              Descending
2014
2012
2010
2009
2005
2005
1999

To learn the year of the most recent book in animals.txt, pipe the output of sort to the input of head and print just the first line:

$ cut -f3 animals.txt | sort -nr | head -n1
2014

... | sort -nr | head -n1

... | sort -n | head -n1

As another example, let’s play with the file /etc/passwd, which lists the users that can run processes on the system.⁴ You’ll generate a list of all users in alphabetical order. Peeking at the first five lines, you see something like this:

$ head -n5 /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
smith:x:1000:1000:Aisha Smith,,,:/home/smith:/bin/bash
jones:x:1001:1001:Bilbo Jones,,,:/home/jones:/bin/bash

Each line consists of strings separated by colons, and the first string is the username, so you can isolate the usernames with the cut command:

$ head -n5 /etc/passwd | cut -d: -f1
root
daemon
bin
smith
jones

and sort them:

$ head -n5 /etc/passwd | cut -d: -f1 | sort
bin
daemon
jones
root
smith

To produce the sorted list of all usernames, not just the first five, replace head with cat:

$ cat /etc/passwd | cut -d: -f1 | sort

To detect if a given user has an account on your system, match their username with grep. Empty output means no account:

$ cut -d: -f1 /etc/passwd | grep -w jones
jones
$ cut -d: -f1 /etc/passwd | grep -w rutabaga         (produces no output)

The -w option instructs grep to match full words only, not partial words, in case your system also has a username that contains “jones”, such as sallyjones2.

Command #6: uniq

The uniq command detects repeated, adjacent lines in a file. By default, it removes the repeats. I’ll demonstrate this with a simple file containing capital letters:

$ cat letters
A
A
A
B
B
A
C
C
C
C
$ uniq letters
A
B
A
C

Notice that uniq reduced the first three A lines to a single A, but it left the last A in place because it wasn’t adjacent to the first three.

You can also count occurrences with the -c option:

$ uniq -c letters
      3 A
      2 B
      1 A
      4 C

I’ll admit, when I first encountered the uniq command, I didn’t see much use in it, but it quickly became one of my favorites. Suppose you have a tab-separated file of students’ final grades for a university course, ranging from A (best) to F (worst):

$ cat grades
C	Geraldine
B	Carmine
A	Kayla
A	Sophia
B	Haresh
C	Liam
B	Elijah
B	Emma
A	Olivia
D	Noah
F	Ava

You’d like to print the grade with the most occurrences. (If there’s a tie, print just one of the winners.) Begin by isolating the grades with cut and sorting them:

$ cut -f1 grades | sort
A
A
A
B
B
B
B
C
C
D
F

Next, use uniq to count adjacent lines:

$ cut -f1 grades | sort | uniq -c
      3 A
      4 B
      2 C
      1 D
      1 F

Then sort the lines in reverse order, numerically, to move the most frequently occurring grade to the top line:

$ cut -f1 grades | sort | uniq -c | sort -nr
      4 B
      3 A
      2 C
      1 F
      1 D

and keep just the first line with head:

$ cut -f1 grades | sort | uniq -c | sort -nr | head -n1
      4 B

Finally, since you want just the letter grade, not the count, isolate the grade with cut:

$ cut -f1 grades | sort | uniq -c | sort -nr | head -n1 | cut -c9
B

and there’s your answer, thanks to a six-command pipeline—our longest yet. This sort of step-by-step pipeline construction is not just an educational exercise. It’s how Linux experts actually work. Chapter 8 is devoted to this technique.

Where Variables Come From

Variables like USER and HOME are predefined by the shell. Their values are set automatically when you log in. (More on this process later.) Traditionally, such predefined variables have uppercase names.

You also may define or modify a variable anytime by assigning it a value using this syntax:

name=value

For example, if you work frequently in the directory /home/smith/Projects, you could assign its name to a variable:

$ work=$HOME/Projects

and use it as a handy shortcut with cd:

$ cd $work
$ pwd
/home/smith/Projects

You may supply $work to any command that expects a directory:

$ cp myfile $work
$ ls $work
myfile

When defining a variable, no spaces are permitted around the equals sign. If you forget, the shell will assume (wrongly) that the first word on the command line is a program to run, and the equals sign and value are its arguments, and you’ll see an error message:

$ work = $HOME/Projects               The shell assumes "work" is a command
work: command not found

A user-defined variable like work is just as legitimate and usable as a system-defined variable like HOME. The only practical difference is that some Linux programs change their behavior internally based on the values of HOME, USER, and other system-defined variables. For example, a Linux program with a graphical interface might retrieve your username from the shell and display it. Such programs don’t pay attention to an invented variable like work because they weren’t programmed to do so.

Variables and Superstition

When you print the value of a variable with echo:

$ echo $HOME
/home/smith

you might think that the echo command examines the HOME variable and prints its value. That is not the case. echo knows nothing about variables. It just prints whatever arguments you hand it. What’s really happening is that the shell evaluates $HOME before running echo. From echo’s perspective, you typed:

$ echo /home/smith

This behavior is extremely important to understand, especially as we delve into more complicated commands. The shell evaluates the variables in a command—as well as patterns and other shell constructs—before executing the command.

Patterns Versus Variables

Let’s test your understanding of pattern and variable evaluation. Suppose you’re in a directory with two subdirectories, mammals and reptiles, and oddly, the mammals subdirectory contains files named lizard.txt and snake.txt:

$ ls
mammals   reptiles
$ ls mammals
lizard.txt  snake.txt

In the real world, lizards and snakes are not mammals, so the two files should be moved to the reptiles subdirectory. Here are two proposed ways to do it. One works, and one does not:

mv mammals/*.txt reptiles                     Method 1

FILES="lizard.txt snake.txt"
mv mammals/$FILES reptiles                    Method 2

Method 1 works because patterns match an entire file path. See how the directory name mammals is part of both matches for mammals/*.txt:

$ echo mammals/*.txt
mammals/lizard.txt mammals/snake.txt

So, method 1 operates as if you’d typed the following correct command:

$ mv mammals/lizard.txt mammals/snake.txt reptiles

Method 2 uses variables, which evaluate to their literal value only. They have no special handling for file paths:

$ echo mammals/$FILES
mammals/lizard.txt snake.txt

So, method 2 operates as if you’d typed the following problematic command:

$ mv mammals/lizard.txt snake.txt reptiles

This command looks for the file snake.txt in the current directory, not in the mammals subdirectory, and fails:

$ mv mammals/$FILES reptiles
/bin/mv: cannot stat 'snake.txt': No such file or directory

To make a variable work in this situation, use a for loop that prepends the directory name mammals to each filename:

FILES="lizard.txt snake.txt"
for f in $FILES; do
  mv mammals/$f reptiles
done

Cursoring Through History

To recall your previous command in a given shell, press the up arrow key. It’s that simple. Keep pressing the up arrow to recall earlier commands in reverse chronological order. Press the down arrow to head in the other direction (toward more recent commands). When you reach the desired command, press Enter to run it.

Cursoring through the command history is one of the two most common speedups that Linux users learn. (The other is pattern matching filenames with *, which you saw in Chapter 2.) Cursoring is efficient if your desired command is nearby in the history—no more than two or three commands in the past—but it’s tedious to reach commands that are further away. Whacking the up arrow 137 times gets old quickly.

The best use case for cursoring is recalling and running the immediately previous command. On many keyboards, the up arrow key is near the Enter key, so you can press the two keys in sequence with a quick flick of the fingers. On a full-sized American QWERTY keyboard, I place my right ring finger on the up arrow and my right index finger on Enter to tap both keys efficiently. (Try it.)

History Expansion

History expansion is a shell feature that accesses the command history using special expressions. The expressions begin with an exclamation point, which traditionally is pronounced “bang.” For example, two exclamation points in a row (“bang bang”) evaluates to the immediately previous command:

$ echo Efficient Linux
Efficient Linux
$ !!                            "Bang bang" = previous command
echo Efficient Linux            The shell helpfully prints the command being run
Efficient Linux

To refer to the most recent command that began with a certain string, place an exclamation point in front of that string. So, to rerun the most recent grep command, run “bang grep”:

$ !grep
grep Perl animals.txt
alpaca	Intermediate Perl	2012	Schwartz, Randal

To refer to the most recent command that contained a given string somewhere, not just at the beginning of the command, surround the string with question marks as well:¹

$ !?grep?
history | grep -w cd
 1000  cd $HOME/Music
 1092  cd ..
⋮

You can also retrieve a particular command from a shell’s history by its absolute position—the ID number to its left in the output of history. For example, the expression !1203 (“bang 1023”) means “the command at position 1023 in the history”:

$ history | grep hosts
 1203  cat /etc/hosts
$ !1203                         The command at position 1023
cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       example.oreilly.com
::1             example.oreilly.com

A negative value retrieves a command by its relative position in the history, rather than absolute position. For example, !-3 (“bang minus three”) means “the command you executed three commands ago”:

$ history
 4197  cd /tmp/junk
 4198  rm *
 4199  head -n2 /etc/hosts
 4199  cd
 4200  history
$ !-3                         The command you executed three commands ago
head -n2 /etc/hosts
127.0.0.1       localhost
127.0.1.1       example.oreilly.com

History expansion is quick and convenient, if a bit cryptic. It can be risky, however, if you provide a wrong value and execute it blindly. Look carefully at the preceding example. If you miscounted and typed !-4 instead of !-3, you’d run rm * instead of the intended head command and delete files in your home directory by mistake! To mitigate this risk, append the modifier :p to print the command from your history but not execute it:

$ !-3:p
head -n2 /etc/hosts              Printed, not executed

The shell appends the unexecuted command (head) to the history, so if it looks good, you can run it conveniently with a quick “bang bang”:

$ !-3:p
head -n2 /etc/hosts              Printed, not executed, and appended to history
$ !!                             Run the command for real
head -n2 /etc/hosts              Printed and then executed
127.0.0.1       localhost
127.0.1.1       example.oreilly.com

Some people refer to history expansion as “bang commands,” but expressions like !! and !grep are not commands. They are string expressions that you can place anywhere in a command. As a demonstration, use echo to print the value of !! on stdout without executing it, and count the number of words with wc:

$ ls -l /etc | head -n3       Run any command
total 1584
drwxr-xr-x  2 root     root       4096 Jun 16 06:14 ImageMagick-6/
drwxr-xr-x  7 root     root       4096 Mar 19  2020 NetworkManager/

$ echo "!!" | wc -w           Count the words in the previous command
echo "ls -l /etc | head -n3" | wc -w
6

This toy example demonstrates that history expansions have more uses than executing commands. You’ll see a more practical, powerful technique in the next section.

I’ve covered only a few features of command history here. For full information, run man history.

$ ls               Run any command
hello.txt
$ cd Music         Run some other command
$ !-2              Use history expansion
ls
song.mp3
$ history          View the history
 1000  ls
 1001  cd Music
 1002  ls          "ls" appears in the history, not "!-2"
 1003  history

Never Delete the Wrong File Again (Thanks to History Expansion)

Have you ever meant to delete files using a pattern, such as *.txt, but accidentally mistyped the pattern and wiped out the wrong files? Here’s an example with an accidental space character after the asterisk:

$ ls
123  a.txt   b.txt   c.txt  dont-delete-me  important-file  passwords
$ rm * .txt       DANGER!! Don't run this! Deletes the wrong files!

The most common solution to this hazard is to alias rm to run rm -i so it prompts for confirmation before each deletion:

$ alias rm='rm -i'                  Often found in a shell configuration file
$ rm *.txt
/bin/rm: remove regular file 'a.txt'? y
/bin/rm: remove regular file 'b.txt'? y
/bin/rm: remove regular file 'c.txt'? y

As a result, an extra space character needn’t be fatal, because the prompts from rm -i will warn that you’re removing the wrong files:

$ rm * .txt
/bin/rm: remove regular file '123'?      Something is wrong: kill the command

The alias solution is cumbersome, however, because most of the time you might not want or need rm to prompt you. It also doesn’t work if you’re logged into another Linux machine without your aliases. I’ll show you a better way to avoid matching the wrong filenames with a pattern. The technique has two steps and relies on history expansion:

1. Verify. Before running rm, run ls with the desired pattern to see which files match.

   $ ls *.txt
   a.txt   b.txt   c.txt

2. Delete. If the output of ls looks correct, run rm !$ to delete the same files that were matched.²

   $ rm !$
   rm *.txt

The history expansion !$ (“bang dollar”) means “the final word that you typed in the previous command.” Therefore, rm !$ here is shorthand for “delete whatever I just listed with ls,” namely, *.txt. If you accidentally add a space after the asterisk, the output of ls will make it obvious—safely—that something is wrong:

$ ls * .txt
/bin/ls: cannot access '.txt': No such file or directory
123  a.txt   b.txt   c.txt  dont-delete-me  important-file  passwords

It’s a good thing you ran ls first instead of rm! You can now modify the command to remove the extra space and proceed safely. This two-command sequence—ls followed by rm !$—is a great safety feature to incorporate into your Linux toolbox.

A related technique is peeking at a file’s contents with head before you delete it, to make sure you’re targeting the right file, and then running rm !$:

$ head myfile.txt
(first 10 lines of the file appear)
$ rm !$
rm myfile.txt

The shell also provides a history expansion !* (“bang star”), which matches all arguments you typed in the previous command, rather than just the final argument:

$ ls *.txt *.o *.log
a.txt   b.txt   c.txt   main.o   output.log   parser.o
$ rm !* 
rm *.txt *.o *.log

In practice, I use !* much less often than !$. Its asterisk carries the same risk of being interpreted as a pattern-matching character for filenames (if you mistype something), so it’s not much safer than typing a pattern like *.txt by hand.

Incremental Search of Command History

Wouldn’t it be great if you could type a few characters of a command and the rest would appear instantly, ready to run? Well, you can. This speedy feature of the shell, called incremental search, is similar to the interactive suggestions provided by web search engines. In most cases, incremental search is the easiest and fastest technique to recall commands from history, even commands you ran long ago. I highly recommend adding it to your toolbox:

1. At the shell prompt, press Ctrl-R (the R stands for reverse incremental search).

2. Start typing any part of a previous command—beginning, middle, or end.

3. With each character you type, the shell displays the most recent historical command that matches your typing so far.

4. When you see the command you want, press Enter to run it.

Suppose you typed the command cd $HOME/Finances/Bank a while ago and you want to rerun it. Press Ctrl-R at the shell prompt. The prompt changes to indicate an incremental search:

(reverse-i-search)`':

Start typing the desired command. For example, type c:

(reverse-i-search)`': c

The shell displays its most recent command that contains the string c, highlighting what you’ve typed:

(reverse-i-search)`': less /etc/hosts

Type the next letter, d:

(reverse-i-search)`': cd

The shell displays its most recent command that contains the string cd, again highlighting what you’ve typed:

(reverse-i-search)`': cd /usr/local

Continue typing the command, adding a space and a dollar sign:

(reverse-i-search)`': cd $

The command line becomes:

(reverse-i-search)`': cd $HOME/Finances/Bank

This is the command you want. Press Enter to run it, and you’re done in five quick keystrokes.

I’ve assumed here that cd $HOME/Finances/Bank was the most recent matching command in the history. What if it’s not? What if you typed a whole bunch of commands that contain the same string? If so, the preceding incremental search would have displayed a different match, such as:

(reverse-i-search)`': cd $HOME/Music

What now? You could type more characters to hone in on your desired command, but instead, press Ctrl-R a second time. This keystroke causes the shell to jump to the next matching command in the history:

(reverse-i-search)`': cd $HOME/Linux/Books

Keep pressing Ctrl-R until you reach the desired command:

(reverse-i-search)`': cd $HOME/Finances/Bank

and press Enter to run it.

Here are a few more tricks with incremental search:

• To recall the most recent string that you searched for and executed, begin by pressing Ctrl-R twice in a row.

• To stop an incremental search and continue working on the current command, press the Escape key, or Ctrl-J, or any key for command-line editing (the next topic in this chapter), such as the left or right arrow key.

• To quit an incremental search and clear the command line, press Ctrl-G or Ctrl-C.

Take the time to become expert with incremental search. You’ll soon be locating commands with incredible speed.³

Cursoring Within a Command

Simply press the left arrow and right arrow keys to move back and forth on the command line, one character at a time. Use the Backspace or Delete key to remove text, and then type any corrections you need. Table 3-1 summarizes these and other standard keystrokes for editing the command line.

Cursoring back and forth is easy but inefficient. It’s best when the changes are small and simple.

History Expansion with Carets

Suppose you’ve mistakenly run the following command by typing jg instead of jpg:

$ md5sum *.jg | cut -c1-32 | sort | uniq -c | sort -nr
md5sum: '*.jg': No such file or directory

To run the command properly, you could recall it from the command history, cursor over to the mistake and fix it, but there’s a quicker way to accomplish your goal. Just type the old (wrong) text, the new (corrected) text, and a pair of carets (^), like this:

$ ^jg^jpg

Press Enter, and the correct command will appear and run:

$ ^jg^jpg
md5sum *.jpg | cut -c1-32 | sort | uniq -c | sort -nr
⋮

The caret syntax, which is a type of history expansion, means, “In the previous command, instead of jg, substitute jpg.” Notice that the shell helpfully prints the new command before executing it, which is standard behavior for history expansion.

This technique changes only the first occurrence of the source string (jg) in the command. If your original command contained jg more than once, only the first instance would change to jpg.

Emacs- or Vim-Style Command-Line Editing

The most powerful way to edit a command line is with familiar keystrokes inspired by the text editors Emacs and Vim. If you’re already skilled with one of these editors, you can jump into this style of command-line editing right away. If not, Table 3-2 will get you started with the most common keystrokes for movement and editing. Note that the Emacs “Meta” key is usually Escape (pressed and released) or Alt (pressed and held).

The shell default is Emacs-style editing, and I recommend it as easier to learn and use. If you prefer Vim-style editing, run the following command (or add it to your $HOME/.bashrc file and source it):

$ set -o vi

To edit a command using Vim keystrokes, press the Escape key to enter command-editing mode, and then use keystrokes from the “Vim” column in Table 3-2. To switch back to Emacs-style editing, run:

$ set -o emacs

Now practice, practice, practice until the keystrokes (either Emacs’s or Vim’s) are second nature. Trust me, you’ll quickly be paid back in saved time.

For more details on Emacs-style editing, see the section “Bindable Readline Commands” in GNU’s bash manual. For Vim-style editing, see the document “Readline VI Editing Mode Cheat Sheet”.

Jump to Your Home Directory

Let’s begin with the basics. No matter where you go in the filesystem, you can return to your home directory by running cd with no arguments:

$ pwd
/etc                              Start somewhere else
$ cd                              Run cd with no arguments...
$ pwd
/home/smith                       ...and you're home again

To jump to subdirectories within your home directory from anywhere in the filesystem, refer to your home directory with a shorthand rather than an absolute path such as /home/smith. One shorthand is the shell variable HOME:

$ cd $HOME/Work

Another is a tilde:

$ cd ~/Work

Both $HOME and ~ are expressions expanded by the shell, a fact that you can verify by echoing them to stdout:

$ echo $HOME ~
/home/smith /home/smith

The tilde can also refer to another user’s home directory if you place it immediately in front of their username:

$ echo ~jones
/home/jones

Move Faster with Tab Completion

When you’re entering cd commands, save typing by pressing the Tab key to produce directory names automatically. As a demonstration, visit a directory that contains subdirectories, such as /usr:

$ cd /usr
$ ls
bin  games  include  lib  local  sbin  share  src

Suppose you want to visit the subdirectory share. Type sha and press the Tab key once:

$ cd sha<Tab>

The shell completes the directory name for you:

$ cd share/

This handy shortcut is called tab completion. It works immediately when the text that you’ve typed matches a single directory name. When the text matches multiple directory names, your shell needs more information to complete the desired name. Suppose you had typed only s and pressed Tab:

$ cd s<Tab>

The shell cannot complete the name share (yet) because other directory names begin with s too: sbin and src. Press Tab a second time and the shell prints all possible completions to guide you:

$ cd s<Tab><Tab>
sbin/  share/  src/

and waits for your next action. To resolve the ambiguity, type another character, h, and press Tab once:

$ cd sh<Tab>

The shell completes the name of the directory for you, from sh to share:

$ cd share/

In general, press Tab once to perform as much completion as possible, or press twice to print all possible completions. The more characters you type, the less ambiguity and the better the match.

Tab completion is great for speeding up navigation. Instead of typing a lengthy path like /home/smith/Projects/Web/src/include, type as little as you want and keep pressing the Tab key. You’ll get the hang of it quickly with practice.

Hop to Frequently Visited Directories Using Aliases or Variables

If you visit a faraway directory frequently, such as /home/smith/Work/⁠Projects​/Web/src/include, create an alias that performs the cd operation:

# In a shell configuration file:
alias work="cd $HOME/Work/Projects/Web/src/include"

Simply run the alias anytime to reach your destination:

$ work
$ pwd
/home/smith/Work/Projects/Web/src/include

Alternatively, create a variable to hold the directory path:

$ work=$HOME/Work/Projects/Web/src/include
$ cd $work
$ pwd
/home/smith/Work/Projects/Web/src/include
$ ls $work/css                                Use the variable in other ways
main.css  mobile.css

# Place in a shell configuration file and source it:
alias rcedit='$EDITOR $HOME/.bashrc'

If you regularly visit lots of directories with long paths, you can create aliases or variables for each of them. This approach has some disadvantages, however:

• It’s hard to remember all those aliases/variables.

• You might accidentally create an alias with the same name as an existing command, causing a conflict.

An alternative is to create a shell function like the one in Example 4-1, which I’ve named qcd (“quick cd”). This function accepts a string key as an argument, such as work or recipes, and runs cd to a selected directory path.

# Define the qcd function
qcd () {
  # Accept 1 argument that's a string key, and perform a different
  # "cd" operation for each key.
  case "$1" in
    work)
      cd $HOME/Work/Projects/Web/src/include
      ;;
    recipes)
      cd $HOME/Family/Cooking/Recipes
      ;;
    video)
      cd /data/Arts/Video/Collection
      ;;
    beatles)
      cd $HOME/Music/mp3/Artists/B/Beatles
      ;;
    *)
      # The supplied argument was not one of the supported keys
      echo "qcd: unknown key '$1'"
      return 1
      ;;
  esac
  # Helpfully print the current directory name to indicate where you are
  pwd
}
# Set up tab completion
complete -W "work recipes video beatles" qcd

Store the function in a shell configuration file such as $HOME/.bashrc (see “Environments and Initialization Files, the Short Version”), source it, and it’s ready to run. Type qcd followed by one of the supported keys to quickly visit the associated directory:

$ qcd beatles
/home/smith/Music/mp3/Artists/B/Beatles

As a bonus, the script’s final line runs the command complete, a shell builtin that sets up customized tab completion for qcd, so it completes the four supported keys. Now you don’t have to remember qcd’s arguments! Just type qcd followed by a space and press the Tab key twice, and the shell will print all the keys for your reference, and you can complete any of them in the usual way:

$ qcd <Tab><Tab>
beatles  recipes  video    work
$ qcd v<Tab><Enter>                       Completes 'v' to 'video'
/data/Arts/Video/Collection

Make a Big Filesystem Feel Smaller with CDPATH

The qcd function handles only the directories that you specify. The shell provides a more general cd-ing solution without this shortcoming, called a cd search path. This shell feature transformed how I navigate the Linux filesystem.

Suppose you have an important subdirectory that you visit often, named Photos. It’s located at /home/smith/Family/Memories/Photos. As you cruise around the filesystem, anytime you want to get to the Photos directory, you may have to type a long path, such as:

$ cd ~/Family/Memories/Photos

Wouldn’t it be great if you could shorten this path to just Photos, no matter where you are in the filesystem, and reach your subdirectory?

$ cd Photos

Normally, this command would fail:

bash: cd: Photos: No such file or directory

unless you happen to be in the correct parent directory (~/Family/Memories) or some other directory with a Photos subdirectory by coincidence. Well, with a little setup, you can instruct cd to search for your Photos subdirectory in locations other than your current directory. The search is lightning fast and looks only in parent directories that you specify. For example, you could instruct cd to search $HOME/Family/Memories in addition to the current directory. Then, when you type cd Photos from elsewhere in the filesystem, cd will succeed:

$ pwd
/etc
$ cd Photos
/home/smith/Family/Memories/Photos

A cd search path works like your command search path, $PATH, but instead of finding commands, it finds subdirectories. Configure it with the shell variable CDPATH, which has the same format as PATH: a list of directories separated by colons. If your CDPATH consists of these four directories, for example:

$HOME:$HOME/Projects:$HOME/Family/Memories:/usr/local

and you type:

$ cd Photos

then cd will check the existence of the following directories in order, until it finds one or it fails entirely:

1. Photos in the current directory

2. $HOME/Photos

3. $HOME/Projects/Photos

4. $HOME/Family/Memories/Photos

5. /usr/local/Photos

In this case, cd succeeds on its fourth try and changes directory to $HOME/Family/Memories/Photos. If two directories in $CDPATH have a subdirectory named Photos, the earlier parent wins.

$ CDPATH=/usr     Set a CDPATH
$ cd /tmp         No output: CDPATH wasn't consulted
$ cd bin          cd consults CDPATH...
/usr/bin          ...and prints the new working directory

Fill CDPATH with your most important or frequently used parent directories, and you can cd into any of their subdirectories from anywhere in the filesystem, no matter how deep they are, without typing most of the path. Trust me, this is awesome, and the following case study should prove it.

Organize Your Home Directory for Fast Navigation

Let’s use CDPATH to simplify the way you navigate your home directory. With a little configuration, you can make many directories within your home directory easily accessible with minimal typing, no matter where you are in the filesystem. This technique works best if your home directory is well organized with at least two levels of subdirectories. Figure 4-1 shows an example of a well-organized directory layout.

Figure 4-1. Two levels of subdirectories in the directory /home/smith

The trick is to set up your CDPATH to include, in order:

1. $HOME

2. Your choice of subdirectories of $HOME

3. The relative path for a parent directory, indicated by two dots (..)

By including $HOME, you can jump immediately to any of its subdirectories (Family, Finances, Linux, Music, and Work) from anywhere else in the filesystem without typing a leading path:

$ pwd
/etc                                   Begin outside your home directory
$ cd Work
/home/smith/Work
$ cd Family/School                     You jumped 1 level below $HOME
/home/smith/Family/School

By including subdirectories of $HOME in your CDPATH, you can jump into their subdirectories in one shot:

$ pwd
/etc                                   Anywhere outside your home directory
$ cd School
/home/smith/Family/School              You jumped 2 levels below $HOME

All the directories in your CDPATH so far are absolute paths in $HOME and its subdirectories. By including the relative path .. however, you empower new cd behavior in every directory. No matter where you are in the filesystem, you can jump to any sibling directory (../sibling) by name without typing the two dots, because cd will search your current parent. For example, if you’re in /usr/bin and want to move to /usr/lib, all you need is cd lib:

$ pwd
/usr/bin                              Your current directory
$ ls ..
bin   include   lib   src             Your siblings
$ cd lib
/usr/lib                              You jumped to a sibling

Or, if you’re a programmer working on code that has subdirectories src, include, and docs:

$ pwd
/usr/src/myproject
$ ls
docs   include   src

you can jump between the subdirectories concisely:

$ cd docs                            Change your current directory
$ cd include
/usr/src/myproject/include           You jumped to a sibling
$ cd src
/usr/src/myproject/src               Again

A CDPATH for the tree in Figure 4-1 might contain six items: your home directory, four of its subdirectories, and the relative path for a parent directory:

# Place in a shell configuration file and source it:
export CDPATH=$HOME:$HOME/Work:$HOME/Family:$HOME/Linux:$HOME/Music:..

After sourcing the configuration file, you can cd to a large number of important directories without typing long directory paths, just short directory names. Hooray!

This technique works best if all subdirectories beneath the CDPATH directories have unique names. If you have duplicate names, such as $HOME/Music and $HOME/Linux/Music, you might not get the behavior you want. The command cd Music will always check $HOME before $HOME/Linux and consequently will not locate $HOME/Linux/Music by search.

To check for duplicate subdirectory names in the first two levels of $HOME, try this brash one-liner. It lists all subdirectories and sub-subdirectories of $HOME, isolates the sub-subdirectory names with cut, sorts the list, and counts occurrences with uniq:

$ cd
$ ls -d */ && (ls -d */*/ | cut -d/ -f2-) | sort | uniq -c | sort -nr | less

You may recognize this duplicate-checking technique from “Detecting Duplicate Files”. If the output displays any counts greater than 1, you have duplicates. I realize this command includes a few features I haven’t covered yet. You’ll learn double ampersand (&&) in “Technique #1: Conditional Lists” and the parentheses in “Technique #10: Explicit Subshells”.

Toggle Between Two Directories with “cd -”

Suppose you’re working in a deep directory and you run cd to go somewhere else:

$ pwd
/home/smith/Finances/Bank/Checking/Statements
$ cd /etc

and then think, “No, wait, I want to go back to the Statements directory where I just was.” Don’t retype the long directory path. Just run cd with a dash as an argument:

$ cd -
/home/smith/Finances/Bank/Checking/Statements

This command returns your shell to its previous directory and helpfully prints its absolute path so you know where you are.

To jump back and forth between a pair of directories, run cd - repeatedly. This is a time-saver when you’re doing focused work in two directories in a single shell. There’s a catch, however: the shell remembers just one previous directory at a time. For example, if you are toggling between /usr/local/bin and /etc:

$ pwd
/usr/local/bin
$ cd /etc                 The shell remembers /usr/local/bin
$ cd -                    The shell remembers /etc
/usr/local/bin
$ cd -                    The shell remembers /usr/local/bin
/etc

and you run cd without arguments to jump to your home directory:

$ cd                      The shell remembers /etc

the shell has now forgotten /usr/local/bin as a previous directory:

$ cd -                    The shell remembers your home directory
/etc
$ cd -                    The shell remembers /etc
/home/smith

The next technique overcomes this limitation.

Toggle Among Many Directories with pushd and popd

The cd - command toggles between two directories, but what if you have three or more to keep track of? Suppose you’re creating a local website on your Linux computer. This task often involves four or more directories:

• The location of live, deployed web pages, such as /var/www/html

• The web-server configuration directory, often /etc/apache2

• The location of SSL certificates, often /etc/ssl/certs

• Your work directory, such as ~/Work/Projects/Web/src

Believe me, it’s tedious to keep typing:

$ cd ~/Work/Projects/Web/src
$ cd /var/www/html
$ cd /etc/apache2
$ cd ~/Work/Projects/Web/src
$ cd /etc/ssl/certs

If you have a large, windowing display, you can ease the burden by opening a separate shell window for each directory. But if you’re working in a single shell (say, over an SSH connection), take advantage of a shell feature called a directory stack. It lets you quickly travel among multiple directories with ease, using the built-in shell commands pushd, popd, and dirs. The learning curve is maybe 15 minutes, and the huge payoff in speed lasts a lifetime.²

A directory stack is a list of directories that you’ve visited in the current shell and decided to keep track of. You manipulate the stack by performing two operations called pushing and popping. Pushing a directory adds it to the beginning of the list, which is traditionally called the top of the stack. Popping removes the topmost directory from the stack.³ Initially, the stack contains only your current directory, but you can add (push) and remove (pop) directories and rapidly cd among them.

I’ll begin with the basic operations (pushing, popping, viewing) and then get to the good stuff.

$ pwd
/home/smith/Work/Projects/Web/src
$ pushd /var/www/html
/var/www/html ~/Work/Projects/Web/src
$ pushd /etc/apache2
/etc/apache2 /var/www/html ~/Work/Projects/Web/src
$ pushd /etc/ssl/certs
/etc/ssl/certs /etc/apache2 /var/www/html ~/Work/Projects/Web/src
$ pwd
/etc/ssl/certs

$ dirs
/etc/ssl/certs /etc/apache2 /var/www/html ~/Work/Projects/Web/src

$ dirs -p
/etc/ssl/certs
/etc/apache2
/var/www/html
~/Work/Projects/Web/src

$ dirs -p | nl -v0
     0  /etc/ssl/certs
     1  /etc/apache2
     2  /var/www/html
     3  ~/Work/Projects/Web/src

$ dirs -v
 0  /etc/ssl/certs
 1  /etc/apache2
 2  /var/www/html
 3  ~/Work/Projects/Web/src

# Place in a shell configuration file and source it:
alias dirs='dirs -v'

$ dirs
/etc/ssl/certs /etc/apache2 /var/www/html ~/Work/Projects/Web/src

$ popd
/etc/apache2 /var/www/html ~/Work/Projects/Web/src
$ popd
/var/www/html ~/Work/Projects/Web/src
$ popd
~/Work/Projects/Web/src
$ popd
bash: popd: directory stack empty
$ pwd
~/Work/Projects/Web/src

# Place in a shell configuration file and source it:
alias gd=pushd
alias pd=popd

$ dirs
/etc/apache2 ~/Work/Projects/Web/src /var/www/html
$ pushd
~/Work/Projects/Web/src /etc/apache2 /var/www/html
$ pushd
/etc/apache2 ~/Work/Projects/Web/src /var/www/html
$ pushd
~/Work/Projects/Web/src /etc/apache2 /var/www/html

$ dirs
~/Work/Projects/Web/src /var/www/html /etc/apache2
$ cd /etc/ssl/certs
$ dirs
/etc/ssl/certs /var/www/html /etc/apache2

$ pushd -
~/Work/Projects/Web/src /etc/ssl/certs /var/www/html /etc/apache2
$ pushd
/etc/ssl/certs ~/Work/Projects/Web/src /var/www/html /etc/apache2

# Place in a shell configuration file and source it:
alias slurp='pushd - && pushd'

$ pushd +N

$ dirs
/etc/ssl/certs ~/Work/Projects/Web/src /var/www/html /etc/apache2
$ pushd +1
~/Work/Projects/Web/src /var/www/html /etc/apache2 /etc/ssl/certs
$ pushd +2
/etc/apache2 /etc/ssl/certs ~/Work/Projects/Web/src /var/www/html

$ dirs -v
 0  /etc/apache2
 1  /etc/ssl/certs
 2  ~/Work/Projects/Web/src
 3  /var/www/html

$ dirs
/etc/apache2 /etc/ssl/certs ~/Work/Projects/Web/src /var/www/html
$ pushd -0
/var/www/html /etc/apache2 /etc/ssl/certs ~/Work/Projects/Web/src

$ popd +N

$ dirs
/var/www/html /etc/apache2 /etc/ssl/certs ~/Work/Projects/Web/src
$ popd +1
/var/www/html /etc/ssl/certs ~/Work/Projects/Web/src
$ popd +2
/var/www/html /etc/ssl/certs

The date Command

The date command prints the current date and/or time in various formats:

$ date                                Default formatting
Mon Jun 28 16:57:33 EDT 2021
$ date +%Y-%m-%d                      Year-Month-Day format
2021-06-28
$ date +%H:%M:%S                      Hour:Minute:Seconds format
16:57:33

To control the output format, provide an argument that begins with a plus sign (+) followed by any text. The text may contain special expressions that begin with a percent sign (%), such as %Y for the current four-digit year and %H for the current hour on a 24-hour clock. A full list of expressions is on the manpage for date.

$ date +"I cannot believe it's already %A!"        Day of week
I cannot believe it's already Tuesday!

The seq Command

The seq command prints a sequence of numbers in a range. Provide two arguments, the low and high values of the range, and seq prints the whole range:

$ seq 1 5                     Print all integers from 1 to 5, inclusive
1
2
3
4
5

If you provide three arguments, the first and third define the range, and the middle number is the increment:

$ seq 1 2 10                  Increment by 2 instead of 1
1
3
5
7
9

Use a negative increment such as -1 to produce a descending sequence:

$ seq 3 -1 0
3
2
1
0

or a decimal increment to produce floating-point numbers:

$ seq 1.1 0.1 2                       Increment by 0.1
1.1
1.2
1.3
⋮
2.0

By default, values are separated by a newline character, but you can change the separator with the -s option followed by any string:

$ seq -s/ 1 5                 Separate values with forward slashes
1/2/3/4/5

The option -w makes all values the same width (in characters) by adding leading zeros as needed:

$ seq -w 8 10
08
09
10

seq can produce numbers in many other formats (see the manpage), but my examples represent the most common uses.

Brace Expansion (A Shell Feature)

The shell provides its own way to print a sequence of numbers, known as brace expansion. Start with a left curly brace, add two integers separated by two dots, and end with a right curly brace:

$ echo {1..10}                        Forward from 1
1 2 3 4 5 6 7 8 9 10
$ echo {10..1}                        Backward from 10
10 9 8 7 6 5 4 3 2 1
$ echo {01..10}                       With leading zeros (equal width)
01 02 03 04 05 06 07 08 09 10

More generally, the shell expression {x..y..z} generates the values x through y, incrementing by z:

$ echo {1..1000..100}                               Count by hundreds from 1
1 101 201 301 401 501 601 701 801 901
$ echo {1000..1..100}                               Backward from 1000
1000 900 800 700 600 500 400 300 200 100
$ echo {01..1000..100}                              With leading zeros
0001 0101 0201 0301 0401 0501 0601 0701 0801 0901

$ ls
file1 file2 file4
$ ls file[2-4]           Matches existing filenames
file2 file4
$ ls file{2..4}          Evaluates to: file2 file3 file4
ls: cannot access 'file3': No such file or directory
file2  file4

Brace expansion also can produce sequences of letters, which seq cannot:

$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Brace expansion always produces output on a single line separated by space characters. Change this by piping the output to other commands, such as tr (see “The tr Command”):

$ echo {A..Z} | tr -d ' '      Delete spaces
ABCDEFGHIJKLMNOPQRSTUVWXYZ
$ echo {A..Z} | tr ' ' '\n'    Change spaces into newlines
A
B
C
⋮
Z

Create an alias that prints the nth letter of the English alphabet:

$ alias nth="echo {A..Z} | tr -d ' ' | cut -c"
$ nth 10
J

The find Command

The find command lists files in a directory recursively, descending into subdirectories and printing full paths.¹ Results are not alphabetical (pipe the output to sort if needed):

$ find /etc -print            List all of /etc recursively
/etc
/etc/issue.net
/etc/nanorc
/etc/apache2
/etc/apache2/sites-available
/etc/apache2/sites-available/default.conf
⋮

find has numerous options that you can combine. Here are a few highly useful ones. Limit the output only to files or directories with the option -type:

$ find . -type f -print               Files only
$ find . -type d -print               Directories only

Limit the output to names that match a filename pattern with the option -name. Quote or escape the pattern so the shell doesn’t evaluate it first:

$ find /etc -type f -name "*.conf" -print           Files ending with .conf
/etc/logrotate.conf
/etc/systemd/logind.conf
/etc/systemd/timesyncd.conf
⋮

Make the name-matching case insensitive with the option -iname:

$ find . -iname "*.txt" -print

find can also execute a Linux command for each file path in the output, using -exec. The syntax is a bit wonky:

1. Construct a find command and omit -print.

2. Append -exec followed by the command to execute. Use the expression {} to indicate where the file path should appear in the command.

3. End with a quoted or escaped semicolon, such as ";" or \;.

Here’s a toy example to print an @ symbol on either side of the file path:

$ find /etc -exec echo @ {} @ ";"
@ /etc @
@ /etc/issue.net @
@ /etc/nanorc @
⋮

A more practical example performs a long listing (ls -l) for all .conf files in /etc and its subdirectories:

$ find /etc -type f -name "*.conf" -exec ls -l {} ";"
-rw-r--r-- 1 root root 703  Aug 21  2017 /etc/logrotate.conf
-rw-r--r-- 1 root root 1022 Apr 20  2018 /etc/systemd/logind.conf
-rw-r--r-- 1 root root 604  Apr 20  2018 /etc/systemd/timesyncd.conf
⋮

find -exec works well for mass deletions of files throughout a directory hierarchy (but be careful!). Let’s delete files with names ending in a tilde (~) within the directory $HOME/tmp and its subdirectories. For safety, first run the command echo rm to see which files would be deleted, then remove echo to delete for real:

$ find $HOME/tmp -type f -name "*~" -exec echo rm {} ";"       echo for safety
rm /home/smith/tmp/file1~
rm /home/smith/tmp/junk/file2~
rm /home/smith/tmp/vm/vm-8.2.0b/lisp/vm-cus-load.el~
$ find $HOME/tmp -type f -name "*~" -exec rm {} ";"            Delete for real

The yes Command

The yes command prints the same string over and over until terminated:

$ yes          Repeats "y" by default
y
y
y ^C           Kill the command with Ctrl-C
$ yes woof!    Repeat any other string
woof!
woof!
woof! ^C

What’s the use of this curious behavior? yes can supply input to interactive programs so they can run unattended. For example, the program fsck, which checks a Linux filesystem for errors, may prompt the user to continue and wait for a response of y or n. The output of the yes command, when piped to fsck, responds to every prompt on your behalf, so you can walk away and let fsck run to completion.²

The main use of yes for our purposes is printing a string a specific number of times by piping yes to head (you’ll see a practical example in “Generating Test Files”):

$ yes "Efficient Linux" | head -n3            Print a string 3 times
Efficient Linux
Efficient Linux
Efficient Linux

grep: A Deeper Look

You’ve already seen grep print lines from a file that match a given string:

$ cat frost
Whose woods these are I think I know.
His house is in the village though;
He will not see me stopping here
To watch his woods fill up with snow.
This is not the end of the poem.
$ grep his frost                              Print lines containing "his"
To watch his woods fill up with snow.
This is not the end of the poem.              "This" matches "his"

grep also has some highly useful options. Use the -w option to match whole words only:

$ grep -w his frost                         Match the word "his" exactly
To watch his woods fill up with snow.

Use the -i option to ignore the case of letters:

$ grep -i his frost
His house is in the village though;             Matches "His"
To watch his woods fill up with snow.           Matches "his"
This is not the end of the poem.                "This" matches "his"

Use the -l option to print only the names of the files that contain matching lines, but not the matching lines themselves:

$ grep -l his *             Which files contain the string "his"?
frost

The real power of grep, however, appears when you move beyond matching simple strings to matching patterns, called regular expressions.³ The syntax is different from filename patterns; a partial description is in Table 5-1.

Here are some example grep commands with regular expressions. Match all lines that begin with a capital letter:

$ grep '^[A-Z]' myfile

Match all nonblank lines (i.e., match blank lines and use -v to omit them):

$ grep -v '^$' myfile

Match all lines that contain either cookie or cake:

$ grep 'cookie\|cake' myfile

Match all lines at least five characters long:

$ grep '.....' myfile

Match all lines in which a less-than symbol appears somewhere before a greater-than symbol, such as lines of HTML code:

$ grep '<.*>' page.html

Regular expressions are great, but sometimes they get in the way. Suppose you want to search for the two lines in the frost file that contain a w followed by a period. The following command produces the wrong results, because a period is a regular expression meaning “any character”:

$ grep w. frost
Whose woods these are I think I know.
He will not see me stopping here
To watch his woods fill up with snow.

To work around this problem, you can escape the special character:

$ grep 'w\.' frost
Whose woods these are I think I know.
To watch his woods fill up with snow.

but this solution becomes cumbersome if you have many special characters to escape. Fortunately, you can force grep to forget about regular expressions and search for every character literally in the input by using the -F (“fixed”) option; or, for an alternative with equivalent results, run fgrep instead of grep:

$ grep -F w. frost
Whose woods these are I think I know.
To watch his woods fill up with snow.
$ fgrep w. frost
Whose woods these are I think I know.
To watch his woods fill up with snow.

grep has many other options; I’ll present just one more that solves a common problem. Use the -f option (lowercase; don’t confuse it with -F) to match against a set of strings rather than a single string. As a practical example, let’s list all shells found in the file /etc/passwd, which I introduced in “Command #5: sort”. As you may recall, each line in /etc/passwd contains information about a user, organized as colon-separated fields. The final field on each line is the program launched when the user logs in. This program is often but not always a shell:

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash                         7th field is a shell
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin         7th field is not a shell
⋮

How can you tell if a program is a shell? Well, the file /etc/shells lists all valid login shells on a Linux system:

$ cat /etc/shells
/bin/sh
/bin/bash
/bin/csh

So, you can list all valid shells in /etc/passwd by extracting the seventh field with cut, eliminating duplicates with sort -u, and checking the results against /etc/shells with grep -f. I also add the -F option to be cautious, so all lines in /etc/shells are taken literally, even if they contain special characters:

$ cut -d: -f7 /etc/passwd | sort -u | grep -f /etc/shells -F
/bin/bash
/bin/sh

The tail Command

The tail command prints the last lines of a file—10 lines by default. It’s a partner to the head command. Suppose you have a file named alphabet containing 26 lines, one per letter:

$ cat alphabet
A is for aardvark
B is for bunny
C is for chipmunk
⋮
X is for xenorhabdus
Y is for yak
Z is for zebu

Print the last three lines with tail. The option -n sets the number of lines to be printed, just as it does for head:

$ tail -n3 alphabet
X is for xenorhabdus
Y is for yak
Z is for zebu

If you precede the number with a plus sign (+), printing begins at that line number and proceeds to the end of the file. The following command begins at the 25th line of the file:

$ tail -n+25 alphabet
Y is for yak
Z is for zebu

Combine tail and head to print any range of lines from a file. To print the fourth line alone, for example, extract the first four lines and isolate the last one:

$ head -n4 alphabet | tail -n1
D is for dingo

In general, to print lines M through N, extract the first N lines with head, then isolate the last N-M+1 lines with tail. Print lines six through eight of the alphabet file:

$ head -n8 alphabet | tail -n3
F is for falcon
G is for gorilla
H is for hawk

$ head -4 alphabet       Same as head -n4 alphabet
$ tail -3 alphabet       Same as tail -n3 alphabet
$ tail +25 alphabet      Same as tail -n+25 alphabet

The awk {print} Command

The command awk is a general-purpose text processor with hundreds of uses. Let’s preview one small feature, print, that extracts columns from a file in ways that cut cannot. Consider the system file /etc/hosts, which includes IP addresses and hostnames separated by any amount of whitespace:

$ less /etc/hosts
127.0.0.1       localhost
127.0.1.1           myhost      myhost.example.com
192.168.1.2       frodo
192.168.1.3     gollum
192.168.1.28        gandalf

Suppose you want to isolate hostnames by printing the second word on each line. The challenge is that each hostname is preceded by an arbitrary amount of whitespace. cut needs its columns either lined up neatly by column number (-c) or separated by a single consistent character (-f). You need a command to print the second word on each line, which awk provides with ease:

$ awk '{print $2}' /etc/hosts
localhost
myhost
frodo
gollum
gandalf

awk refers to any column by a dollar sign followed by the column number: for example, $7 for the seventh column. If the column number has more than one digit, surround the number with parentheses: for example, $(25). To refer to the final field, use $NF (“number of fields”). To refer to the entire line, use $0.

awk does not print whitespace between values by default. If you want whitespace, separate the values with commas:

$ echo Efficient fun Linux | awk '{print $1 $3}'           No whitespace
EfficientLinux
$ echo Efficient fun Linux | awk '{print $1, $3}'          Whitespace
Efficient Linux

awk’s print statement is great for processing the output of commands that strays outside tidy columns. An example is df, which prints the amount of free and used disk space on a Linux system:

$ df / /data
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sda1      1888543276  902295944  890244772  51% /
/dev/sda2      7441141620 1599844268 5466214400  23% /data

The column locations may vary depending on the length of the Filesystem paths, the disk sizes, and the options you pass to df, so you can’t reliably extract values with cut. With awk, however, you can easily isolate (say) the fourth value on each line, representing available disk space:

$ df / /data | awk '{print $4}'
Available
890244772
5466214400

and even remove the first line (the header) at the same time with a little awk magic, printing only line numbers greater than 1:

$ df / /data | awk 'FNR>1 {print $4}'
890244772
5466214400

If you encounter input separated by something other than space characters, awk can change its field separator to any regular expression with the -F option:

$ echo efficient:::::linux | awk -F':*' '{print $2}'       Any number of colons
linux

You’ll learn more details about awk in “awk essentials”.

The tac Command

The tac command reverses a file line by line. Its name is cat spelled backward.

$ cat poem1 poem2 poem3 | tac
Now wherefore stopp'st thou me?
'By thy long grey beard and glittering eye,
And he stoppeth one of three.
It is an ancient Mariner,

Notice I concatenated three files before reversing the text. If I instead provide multiple files to tac as arguments, it reverses the lines of each file in turn, producing different output:

$ tac poem1 poem2 poem3
And he stoppeth one of three.                    First file reversed
It is an ancient Mariner,
'By thy long grey beard and glittering eye,      Second file
Now wherefore stopp'st thou me?                  Third file

tac is great for processing data that is already in chronological order but not reversible with the sort -r command. A typical case is reversing a web-server log file to process its lines from newest to oldest:

192.168.1.34 - - [30/Nov/2021:23:37:39 -0500] "GET / HTTP/1.1" ...
192.168.1.10 - - [01/Dec/2021:00:02:11 -0500] "GET /notes.html HTTP/1.1" ...
192.168.1.8 - - [01/Dec/2021:00:04:30 -0500] "GET /stuff.html HTTP/1.1" ...
⋮

The lines are in chronological order with timestamps, but they aren’t in alphabetical or numeric order, so the sort -r command isn’t helpful. The tac command can reverse these lines without needing to consider the timestamps.

The paste Command

The paste command combines files side by side in columns separated by a single tab character. It’s a partner to the cut command, which extracts columns from a tab-separated file:

$ cat title-words1
EFFICIENT
AT
COMMAND
$ cat title-words2
linux
the
line
$ paste title-words1 title-words2
EFFICIENT	linux
AT	the
COMMAND line
$ paste title-words1 title-words2 | cut -f2        cut & paste are complementary
linux
the
line

Change the separator to another character, such as a comma, with the option -d (meaning “delimiter”):

$ paste -d, title-words1 title-words2
EFFICIENT,linux
AT,the
COMMAND,line

Transpose the output, producing pasted rows instead of pasted columns, with the -s option:

$ paste -d, -s title-words1 title-words2
EFFICIENT,AT,COMMAND
linux,the,line

paste also interleaves data from two or more files if you change the separator to a newline character (\n):

$ paste -d "\n" title-words1 title-words2
EFFICIENT
linux
AT
the
COMMAND
line

The diff Command

diff compares two files line by line and prints a terse report about their differences:

$ cat file1
Linux is all about efficiency.
I hope you will enjoy this book.
$ cat file2
MacOS is all about efficiency.
I hope you will enjoy this book.
Have a nice day.
$ diff file1 file2
1c1
< Linux is all about efficiency.
---
> MacOS is all about efficiency.
2a3
> Have a nice day.

The notation 1c1 represents a change or difference between the files. It means that line 1 in the first file differs from line 1 in the second file. This notation is followed by the relevant line from file1, a three-dash separator (---), and the relevant line from file2. The leading symbol < always indicates a line from the first file, and > indicates a line from the second file.

The notation 2a3 represents an addition. It means that file2 has a third line not present after the second line of file1. This notation is followed by the extra line from file2, “Have a nice day.”

diff output may contain other notation and can take other forms. This short explanation is enough for our main purpose, however, which is to use diff as a text processor that interleaves lines from two files. Many users don’t think of diff this way, but it’s great for forming pipelines to solve certain kinds of problems. For example, you can isolate the differing lines with grep and cut:

$ diff file1 file2 | grep '^[<>]'
< Linux is all about efficiency.
> MacOS is all about efficiency.
> Have a nice day.
$ diff file1 file2 | grep '^[<>]' | cut -c3-
Linux is all about efficiency.
MacOS is all about efficiency.
Have a nice day.

You’ll see practical examples in “Technique #4: Process Substitution” and “Checking Matched Pairs of Files”.

The tr Command

tr translates one set of characters into another. I showed you one example in Chapter 2 of translating colons into newline characters to print the shell’s PATH:

$ echo $PATH | tr : "\n"              Translate colons into newlines
/home/smith/bin
/usr/local/bin
/usr/bin
/bin
/usr/games
/usr/lib/java/bin

tr takes two sets of characters as arguments, and it translates members of the first set into the corresponding members of the second. Common uses are converting text to uppercase or lowercase:

$ echo efficient | tr a-z A-Z         Translate a into A, b into B, etc.
EFFICIENT
$ echo Efficient | tr A-Z a-z
efficient

converting spaces into newlines:

$ echo Efficient Linux | tr " " "\n"
Efficient
Linux

and deleting whitespace with the -d (delete) option:

$ echo efficient linux | tr -d ' \t'      Remove spaces and tabs
efficientlinux

The rev Command

The rev command reverses the characters of each line of input:⁴

$ echo Efficient Linux! | rev
!xuniL tneiciffE

Beyond the obvious entertainment value, rev is handy for extracting tricky information from files. Suppose you have a file of celebrity names:

$ cat celebrities
Jamie Lee Curtis
Zooey Deschanel
Zendaya Maree Stoermer Coleman
Rihanna

and you want to extract the final word on each line (Curtis, Deschanel, Coleman, Rihanna). This would be easy with cut -f if each line had the same number of fields, but the number varies. With rev, you can reverse all the lines, cut the first field, and reverse again to achieve your goal:⁵

$ rev celebrities
sitruC eeL eimaJ
lenahcseD yeooZ
nameloC remreotS eeraM ayadneZ
annahiR
$ rev celebrities | cut -d' ' -f1
sitruC
lenahcseD
nameloC
annahiR
$ rev celebrities | cut -d' ' -f1 | rev
Curtis
Deschanel
Coleman
Rihanna

The awk and sed Commands

awk and sed are general-purpose “supercommands” for processing text. They can do most everything that the other commands in this chapter do, but with more cryptic-looking syntax. As a simple example, they can print the first 10 lines of a file like head does:

$ sed 10q myfile                  Print 10 lines and quit (q)
$ awk 'FNR<=10' myfile            Print while line number is ≤ 10

They can also do things that our other commands cannot, like replace or swap strings:

$ echo image.jpg | sed 's/\.jpg/.png/'                 Replace .jpg by .png
image.png
$ echo "linux efficient" | awk '{print $2, $1}'        Swap two words
efficient linux

awk and sed are harder to learn than the other commands I’ve covered, because each of them has a miniature programming language built in. They have so many capabilities that whole books have been written on them.⁶ I highly recommend spending quality time learning both commands (or at least one of them). To begin your journey, I cover basic principles of each command and demonstrate some common uses. I also recommend several online tutorials to learn more about these powerful, crucial commands.

Don’t worry about memorizing every feature of awk or sed. Success with these commands really means:

• Understanding the kinds of transformations they make possible, so you can think, “Ah! This is a job for awk (or sed)!” and apply them in your time of need

• Learning to read their manpages and to find complete solutions on Stack Exchange and other online resources

$ awk program input-files

$ awk -f program-file1 -f program-file2 -f program-file3 input-files

pattern {action}

$ awk '{print $NF}' celebrities
Curtis
Deschanel
Coleman
Rihanna

$ echo efficient linux | awk '/efficient/'
efficient linux

python	Programming Python	2010	Lutz, Mark

Lutz, Mark (2010). "Programming Python"

$ awk -F'\t' '{print $4, "(" $3 ").", "\"" $2 "\""}' animals.txt
Lutz, Mark (2010). "Programming Python"
Barrett, Daniel (2005). "SSH, The Secure Shell"
Schwartz, Randal (2012). "Intermediate Perl"
Bell, Charles (2014). "MySQL High Availability"
Siever, Ellen (2009). "Linux in a Nutshell"
Boney, James (2005). "Cisco IOS in a Nutshell"
Roman, Steven (1999). "Writing Word Macros"

$ awk -F'\t' '/^horse/{print $4, "(" $3 ").", "\"" $2 "\""}' animals.txt
Siever, Ellen (2009). "Linux in a Nutshell"

$ awk -F'\t' '$3~/^201/{print $4, "(" $3 ").", "\"" $2 "\""}' animals.txt
Lutz, Mark (2010). "Programming Python"
Schwartz, Randal (2012). "Intermediate Perl"
Bell, Charles (2014). "MySQL High Availability"

$ awk -F'\t' \
  'BEGIN {print "Recent books:"} \
  $3~/^201/{print "-", $4, "(" $3 ").", "\"" $2 "\""} \
  END {print "For more books, search the web"}' \
  animals.txt
Recent books:
- Lutz, Mark (2010). "Programming Python"
- Schwartz, Randal (2012). "Intermediate Perl"
- Bell, Charles (2014). "MySQL High Availability"
For more books, search the web

$ seq 1 100 | awk '{s+=$1} END {print s}'
5050

$ md5sum *.jpg | cut -c1-32 | sort | uniq -c | sort -nr | grep -v "      1 "
      3 f6464ed766daca87ba407aede21c8fcc
      2 c7978522c58425f6af3f095ef1de1cd5
      2 146b163929b6533f02e91bdf21cb9563

$ md5sum *.jpg
146b163929b6533f02e91bdf21cb9563  image001.jpg
63da88b3ddde0843c94269638dfa6958  image002.jpg
146b163929b6533f02e91bdf21cb9563  image003.jpg
⋮

$ md5sum *.jpg | awk '{counts[$1]++}'

for (variable in array) do something with array[variable]

for (key in counts) print array[key]

$ md5sum *.jpg \
  | awk '{counts[$1]++} \
         END {for (key in counts) print counts[key]}'
1
2
2
⋮

$ md5sum *.jpg \
  | awk '{counts[$1]++} \
         END {for (key in counts) print counts[key] " " key}'
1 714eceeb06b43c03fe20eb96474f69b8
2 146b163929b6533f02e91bdf21cb9563
2 c7978522c58425f6af3f095ef1de1cd5
⋮

$ md5sum *.jpg \
  | awk '{counts[$1]++; names[$1]=names[$1] " " $2} \
         END {for (key in counts) print counts[key] " " key ":" names[key]}'
1 714eceeb06b43c03fe20eb96474f69b8: image011.jpg
2 146b163929b6533f02e91bdf21cb9563: image001.jpg image003.jpg
2 c7978522c58425f6af3f095ef1de1cd5: image019.jpg image020.jpg
⋮

$ md5sum *.jpg \
  | awk '{counts[$1]++; names[$1]=names[$1] " " $2} \
         END {for (key in counts) print counts[key] " " key ":" names[key]}' \
  | grep -v '^1 ' \
  | sort -nr
3 f6464ed766daca87ba407aede21c8fcc: image007.jpg image012.jpg image014.jpg
2 c7978522c58425f6af3f095ef1de1cd5: image019.jpg image020.jpg
2 146b163929b6533f02e91bdf21cb9563: image001.jpg image003.jpg

$ sed script input-files

$ sed -e script1 -e script2 -e script3 input-files

$ sed -f script-file1 -f script-file2 -f script-file3 input-files

s/regexp/replacement/

$ echo Efficient Windows | sed "s/Windows/Linux/"
Efficient Linux

$ sed 's/.* //' celebrities
Curtis
Deschanel
Coleman
Rihanna

s/one/two/	s_one_two_	s@one@two@

$ echo Efficient Stuff | sed "s/stuff/linux/"         Case sensitive; no match
Efficient Stuff
$ echo Efficient Stuff | sed "s/stuff/linux/i"        Case-insensitive match
Efficient linux

$ echo efficient stuff | sed "s/f/F/"         Replaces just the first "f"
eFficient stuff
$ echo efficient stuff | sed "s/f/F/g"        Replaces all occurrences of "f"
eFFicient stuFF

$ seq 10 14 | sed 4d                 Remove the 4th line
10
11
12
14

$ seq 101 200 | sed '/[13579]$/d'    Delete lines ending in an odd digit
102
104
106
⋮
200

$ ls
image.jpg.1  image.jpg.2  image.jpg.3

image\.jpg\.[1-3]

image\.jpg\.\([1-3]\)

$ ls | sed "s/image\.jpg\.\([1-3]\)/image\1.jpg/"
image1.jpg
image2.jpg
image3.jpg

$ ls
apple.jpg.1  banana.png.2  carrot.jpg.3

\([a-z][a-z]*\)                 \1 = Base filename of one letter or more
\([a-z][a-z][a-z]\)             \2 = File extension of three letters
\([0-9]\)                       \3 = A digit

\([a-z][a-z]*\)\.\([a-z][a-z][a-z]\)\.\([0-9]\)

$ ls | sed "s/\([a-z][a-z]*\)\.\([a-z][a-z][a-z]\)\.\([0-9]\)/\1\3.\2/"
apple1.jpg
banana2.png
carrot3.jpg

Creating Environment Variables

To turn a local variable into an environment variable, use the export command:

$ MY_VARIABLE=10                  A local variable
$ export MY_VARIABLE              Export it to become an environment variable
$ export ANOTHER_VARIABLE=20      Or, set and export in a single command

export specifies that the variable and its value will be copied from the current shell to any future children. Local variables are not copied to future children:

$ export E="I am an environment variable"     Set an environment variable
$ L="I am just a local variable"              Set a local variable
$ echo $E
I am an environment variable
$ echo $L
I am just a local variable
$ bash                                        Run a child shell
$ echo $E                                     Environment variable was copied
I am an environment variable
$ echo $L                                     Local variable was not copied
                                              Empty string is printed
$ exit                                        Exit the child shell

Remember, a child’s variables are copies. Any changes to the copy do not affect the parent shell:

$ export E="I am the original value"          Set an environment variable
$ bash                                        Run a child shell
$ echo $E
I am the original value                       Parent's value was copied
$ E="I was modified in a child"               Change the child's copy
$ echo $E
I was modified in a child
$ exit                                        Exit the child shell
$ echo $E
I am the original value                       Parent's value is unchanged

Launch a new shell anytime and change anything in its environment, and all the changes disappear when you exit the shell. This means you can experiment with shell features safely—just run a shell manually, creating a child, and terminate it when finished.

Superstition Alert: “Global” Variables

Sometimes Linux hides its inner workings too well. A great example is the behavior of environment variables. Somehow, like magic, variables like HOME and PATH each have a consistent value in all your shell instances. They seem to be “global variables” in some sense. (I’ve even seen this claim in other Linux books, not published by O’Reilly.) But an environment variable is not global. Each shell instance has its own copy. Modifying an environment variable in one shell cannot change the value in any other running shells. Modifications affect only that shell’s future children (not yet invoked).

If that’s the case, how does a variable like HOME or PATH seem to keep its value in all your shell instances? There are two avenues to make this happen, which are illustrated in Figure 6-1. In short:

Children copy from their parents.

For variables like HOME, the values are usually set and exported by your login shell. All future shells (until you log out) are children of the login shell, so they receive a copy of the variable and its value. These sorts of system-defined environment variables are so rarely modified in the real world that they seem global, but they are just ordinary variables that play by the ordinary rules. (You may even change their values in a running shell, but you might disrupt the expected behavior of that shell and other programs.)

Different instances read the same configuration files.

Local variables, which are not copied to children, can have their values set in a Linux configuration file such as $HOME/.bashrc (see more details in “Configuring Your Environment”). Each instance of the shell, on invocation, reads and executes the appropriate configuration files. As a result, these local variables appear to be copied from shell to shell. The same is true for other nonexported shell features such as aliases.

This behavior leads some users to believe that the export command creates a global variable. It does not. The command export WHATEVER simply declares that the variable WHATEVER will be copied from the current shell to any future children.

Figure 6-1. Shells may share variables and values by export or by reading the same configuration files

Rereading a Configuration File

When you change any startup or initialization file, you can force a running shell to reread it by sourcing the file, as explained in “Environments and Initialization Files, the Short Version”:

$ source ~/.bash_profile             Uses the builtin "source" command
$ . ~/.bash_profile                  Uses a dot

Traveling with Your Environment

If you use many Linux machines in multiple locations, at some point you might want to install your carefully crafted configuration files on more than one machine. Don’t copy individual files from machine to machine—that approach leads to confusion eventually. Instead, store and maintain the files in a free account on GitHub or a similar software-development service with version control. Then you can download, install, and update your configuration files conveniently and consistently on any Linux machine. If you make a mistake editing a configuration file, you can roll back to a previous version by issuing a command or two. Version control is beyond the scope of this book; see “Apply Version Control to Day-to-Day Files” to learn more about it.

If you aren’t comfortable with version control systems like Git or Subversion, store the configuration files on a simple file service like Dropbox, Google Drive, or OneDrive. Updates to your configuration files will be less convenient, but at least the files will be easily available for copying to other Linux systems.

Technique #1: Conditional Lists

Suppose you want to create a file new.txt in a directory dir. A typical sequence of commands might be:

$ cd dir            Enter the directory
$ touch new.txt     Make the file

Notice how the second command depends on the success of the first. If the directory dir doesn’t exist, there is no point in running the touch command. The shell lets you make this dependency explicit. If you place the operator && (pronounced “and”) between the two commands on a single line:

$ cd dir && touch new.txt

then the second command (touch) runs only if the first command (cd) succeeds. The preceding example is a conditional list of two commands. (To learn what it means for a command to “succeed,” see “Exit Codes Indicate Success or Failure”.)

Very likely, you run commands every day that depend on previous ones. For example, have you ever made a backup copy of a file for safekeeping, modified the original, and deleted the backup when done?

$ cp myfile.txt myfile.safe      Make a backup copy
$ nano myfile.txt                Change the original
$ rm myfile.safe                 Delete the backup

Each of these commands makes sense only if the preceding command succeeds. Therefore, this sequence is a candidate for a conditional list:

$ cp myfile.txt myfile.safe && nano myfile.txt && rm myfile.safe

As another example, if you use the version-control system Git to maintain files, you’re probably familiar with the following sequence of commands after you change some files: run git add to prepare files for a commit, then git commit, and finally git push to share your committed changes. If any of these commands failed, you wouldn’t run the rest (until you fixed the cause of the failure). Therefore, these three commands work well as a conditional list:

$ git add . && git commit -m"fixed a bug" && git push

Just as the && operator runs a second command only if the first succeeds, the related operator || (pronounced “or”) runs a second command only if the first fails. For example, the following command tries to enter dir, and if it fails to do so, it creates dir:¹

$ cd dir || mkdir dir

You’ll commonly see the || operator in scripts, causing the script to exit if an error occurs:

# If a directory can't be entered, exit with an error code of 1
cd dir || exit 1

Combine the && and || operators to set up more complicated actions for success and failure. The following command tries to enter directory dir, and if it fails, it creates the directory and enters it. If all fails, the command prints a failure message:

$ cd dir || mkdir dir && cd dir || echo "I failed"

The commands in a conditional list don’t have to be simple commands; they can also be pipelines and other combined commands.

Technique #2: Unconditional Lists

Commands in a list don’t have to depend on one another. If you separate the commands with semicolons, they simply run in order. Success or failure of a command does not affect later ones in the list.

I like unconditional lists for launching ad hoc commands after I’ve left work for the day. Here’s one that sleeps (does nothing) for two hours (7,200 seconds) and then backs up my important files:

$ sleep 7200; cp -a ~/important-files /mnt/backup_drive

Here’s a similar command that functions as a primitive reminder system, sleeping for five minutes and then sending me an email:³

$ sleep 300; echo "remember to walk the dog" | mail -s reminder $USER

Unconditional lists are a convenience feature: they produce the same results (mostly) as typing the commands individually and pressing Enter after each. The only significant difference relates to exit codes. In an unconditional list, the exit codes of the individual commands are thrown away except the last one. Only the exit code of the final command in the list is assigned to the shell variable ?:

$ mv file1 file2; mv file2 file3; mv file3 file4
$ echo $?
0                           The exit code for "mv file3 file4"

Technique #3: Command Substitution

Suppose you have a few thousand text files representing songs. Each file includes a song title, artist name, album title, and the song lyrics:

Title: Carry On Wayward Son
Artist: Kansas
Album: Leftoverture

Carry on my wayward son
There'll be peace when you are done
⋮

You’d like to organize the files into subdirectories by artist. To perform this task by hand, you could search for all song files by Kansas using grep:

$ grep -l "Artist: Kansas" *.txt
carry_on_wayward_son.txt
dust_in_the_wind.txt
belexes.txt

and then move each file to a directory kansas:

$ mkdir kansas
$ mv carry_on_wayward_son.txt kansas
$ mv dust_in_the_wind.txt kansas
$ mv belexes.txt kansas

Tedious, right? Wouldn’t be great if you could tell the shell, “Move all files that contain the string Artist: Kansas to the directory kansas.” In Linux terms, you’d like to take the list of names from the preceding grep -l command and hand it to mv. Well, you can do this easily with the help of a shell feature called command substitution:

$ mv $(grep -l "Artist: Kansas" *.txt) kansas

The syntax:

$(any command here)

executes the command inside the parentheses and replaces the command by its output. So on the preceding command line, the grep -l command is replaced by the list of filenames that it prints, as if you had typed the filenames like this:

$ mv carry_on_wayward_son.txt dust_in_the_wind.txt belexes.txt kansas

Whenever you find yourself copying the output of one command into a later command line, you can usually save time with command substitution. You can even include aliases in command substitution, because its contents are run in a subshell, which includes copies of its parent’s aliases.

Here’s another example. Suppose you have several years’ worth of bank statements downloaded in PDF format. The downloaded files have names that include the statement’s year, month, and day, such as eStmt_2021-08-26.pdf for the date August 26, 2021.⁴ You’d like to view the most recent statement in the current directory. You could do it manually: list the directory, locate the file with the most recent date (which will be the final file in the listing), and display it with a Linux PDF viewer such as okular. But why do all that manual work? Let command substitution ease your way. Create a command that prints the name of the latest PDF file in the directory:

$ ls eStmt*pdf | tail -n1

and provide it to okular using command substitution:

$ okular $(ls eStmt*pdf | tail -n1)

The ls command lists all the statement files, and tail prints only the last one, such as eStmt_2021-08-26.pdf. Command substitution places that single filename right onto the command line, as if you’d typed okular eStmt_2021-08-26.pdf.

$ echo Today is $(date +%A).
Today is Saturday.
$ echo Today is `date +%A`.
Today is Saturday.

$ echo $(date +%A) | tr a-z A-Z                    Single
SATURDAY
echo Today is $(echo $(date +%A) | tr a-z A-Z)!   Nested
Today is SATURDAY!

In scripts, a common use of command substitution is to store the output of a command in a variable:

VariableName=$(some command here)

For example, to get the filenames containing Kansas songs and store them in a variable, use command substitution like so:

$ kansasFiles=$(grep -l "Artist: Kansas" *.txt) 

The output might have multiple lines, so to preserve any newline characters, make sure you quote the value wherever you use it:

$ echo "$kansasFiles"

Technique #4: Process Substitution

Command substitution, which you just saw, replaces a command with its output in place, as a string. Process substitution also replaces a command with its output, but it treats the output as if it were stored in a file. This powerful difference may look confusing at first, so I’ll explain it step-by-step.

Suppose you’re in a directory of JPEG image files named 1.jpg through 1000.jpg, but some files are mysteriously missing and you want to identify them. Produce such a directory with the following commands:

$ mkdir /tmp/jpegs && cd /tmp/jpegs
$ touch {1..1000}.jpg
$ rm 4.jpg 981.jpg

A poor way to locate the missing files is to list the directory, sorted numerically, and look for gaps by eye:

$ ls -1 | sort -n | less
1.jpg
2.jpg
3.jpg
5.jpg            4.jpg is missing
⋮

A more robust, automated solution is to compare the existing filenames to a complete list of names from 1.jpg to 1000.jpg, using the diff command. One way to achieve this solution is with temporary files. Store the existing filenames, sorted, in one temporary file, original-list:

$ ls *.jpg | sort -n > /tmp/original-list

Then print a complete list of filenames from 1.jpg to 1000.jpg to another temporary file, full-list, by generating the integers 1 to 1000 with seq, and appending “.jpg” to each line with sed:

$ seq 1 1000 | sed 's/$/.jpg/' > /tmp/full-list

Compare the two temporary files with the diff command to discover that 4.jpg and 981.jpg are missing, then delete the temporary files:

$ diff /tmp/original-list /tmp/full-list
3a4
> 4.jpg
979a981
> 981.jpg
$ rm /tmp/original-list /tmp/full-list       Clean up afterwards

That’s a lot of steps. Wouldn’t it be grand to compare the two lists of names directly and not bother with temporary files? The challenge is that diff can’t compare two lists from stdin; it requires files as arguments.⁵ Process substitution solves the problem. It makes both lists appear to diff as files. (The sidebar “How Process Substitution Works” provides the technical details.) The syntax:

<(any command here)

runs the command in a subshell and presents its output as if it were contained in a file. For example, the following expression represents the output of ls -1 | sort -n as if it were contained in a file:

<(ls -1 | sort -n)

You can cat the file:

$ cat <(ls -1 | sort -n)
1.jpg
2.jpg
⋮

You can copy the file with cp:

$ cp <(ls -1 | sort -n) /tmp/listing
$ cat /tmp/listing
1.jpg
2.jpg
⋮

and as you’ll now see, you can diff the file against another. Begin with the two commands that generated your two temporary files:

ls *.jpg | sort -n
seq 1 1000 | sed 's/$/.jpg/'

Apply process substitution so diff can treat them as files, and you get the same output as before, but without using temporary files:

$ diff <(ls *.jpg | sort -n) <(seq 1 1000 | sed 's/$/.jpg/')
3a4
> 4.jpg
979a981
> 981.jpg

Clean up the output by grepping for lines beginning with > and stripping off the first two characters with cut, and you have your missing files report:

$ diff <(ls *.jpg | sort -n) <(seq 1 1000 | sed 's/$/.jpg/') \
    | grep '>' | cut -c3-
4.jpg
981.jpg

Process substitution transformed how I use the command line. Commands that read only from disk files suddenly could read from stdin. With practice, commands that previously seemed impossible became easy.

Technique #5: Passing a Command as an Argument to bash

bash is a normal command like any other, as explained in “Shells Are Executable Files”, so you can run it by name on the command line. By default, running bash launches an interactive shell for typing and executing commands, as you’ve seen. Alternatively, you can pass a command to bash as a string, via the -c option, and bash will run that string as a command and exit:

$ bash -c "ls -l"
-rw-r--r-- 1 smith smith 325 Jul  3 17:44 animals.txt

Why is this helpful? Because the new bash process is a child with its own environment, including a current directory, variables with values, and so on. Any changes to the child shell won’t affect your currently running shell. Here’s a bash -c command that changes directory to /tmp just long enough to delete a file, then exits:

$ pwd
/home/smith
$ touch /tmp/badfile                           Create a temporary file
$ bash -c "cd /tmp && rm badfile"
$ pwd
/home/smith                                    Current directory is unchanged

The most instructive and beautiful use of bash -c, however, arises when you run certain commands as the superuser. Specifically, the combination of sudo and input/output redirection produces an interesting (sometimes maddening) situation in which bash -c is the key to success.

Suppose you want to create a log file in the system directory /var/log, which is not writable by ordinary users. You run the following sudo command to gain superuser privileges and create the log file, but it mysteriously fails:

$ sudo echo "New log file" > /var/log/custom.log
bash: /var/log/custom.log: Permission denied

Wait a minute—sudo should give you permission to create any file anywhere. How can this command possibly fail? Why didn’t sudo even prompt you for a password? The answer is: because sudo didn’t run. You applied sudo to the echo command but not to the output redirection, which ran first and failed. In detail:

1. You pressed Enter.

2. The shell began to evaluate the whole command, including redirection (>).

3. The shell tried to create the file custom.log in a protected directory, /var/log.

4. You didn’t have permission to write to /var/log, so the shell gave up and printed the “Permission denied” message.

That’s why sudo never ran. To solve this problem, you need to tell the shell, “Run the entire command, including output redirection, as the superuser.” This is exactly the kind of situation that bash -c solves so well. Construct the command you want to run, as a string:

'echo "New log file" > /var/log/custom.log'

and pass it as an argument to sudo bash -c:

$ sudo bash -c 'echo "New log file" > /var/log/custom.log'
[sudo] password for smith: xxxxxxxx
$ cat /var/log/custom.log
New log file

This time, you’ve run bash, not just echo, as the superuser, and bash executes the entire string as a command. The redirection succeeds. Remember this technique whenever you pair sudo with redirection.

Technique #6: Piping a Command to bash

The shell reads every command that you type on stdin. That means bash the program can participate in pipelines. For example, print the string "ls -l" and pipe it to bash, and bash will treat the string as a command and run it:

$ echo "ls -l"
ls -l
$ echo "ls -l" | bash
-rw-r--r-- 1 smith smith 325 Jul  3 17:44 animals.txt

This technique is terrific when you need to run many similar commands in a row. If you can print the commands as strings, then you can pipe the strings to bash for execution. Suppose you’re in a directory with many files, and you want to organize them into subdirectories by their first character. A file named apple would be moved to subdirectory a, a file named cantaloupe would move to subdirectory c, and so on.⁶ (For simplicity, we’ll assume all the filenames begin with a lowercase letter and contain no spaces or special characters.)

First, list the files, sorted. We’ll assume all the names are at least two characters long (matching the pattern ??*) so our commands don’t collide with the subdirectories a through z:

$ ls -1 ??* 
apple
banana
cantaloupe
carrot
⋮

Create the 26 subdirectories you need via brace expansion:

$ mkdir {a..z}

Now generate the mv commands you’ll need, as strings. Start with a regular expression for sed that captures the first character of the filename as expression #1 (\1):

^\(.\)

Capture the rest of the filename as expression #2 (\2):

\(.*\)$

Connect the two regular expressions:

^\(.\)\(.*\)$

Now form an mv command with the word mv followed by a space, the full filename (\1\2), another space, and the first character (\1):

mv \1\2 \1

The complete command generator is:

$ ls -1 ??* | sed 's/^\(.\)\(.*\)$/mv \1\2 \1/'
mv apple a
mv banana b
mv cantaloupe c
mv carrot c
⋮

Its output contains exactly the mv commands you need. Read the output to convince yourself it’s correct, perhaps by piping it to less for page-by-page viewing:

$ ls -1 ??* | sed 's/^\(.\)\(.*\)$/mv \1\2\t\1/' | less

When you’re satisfied that your generated commands are correct, pipe the output to bash for execution:

$ ls -1 ??* | sed 's/^\(.\)\(.*\)$/mv \1\2\t\1/' | bash

The steps you just completed are a repeatable pattern:

1. Print a sequence of commands by manipulating strings.

2. View the results with less to check correctness.

3. Pipe the results to bash.

Technique #7: Executing a String Remotely with ssh

Disclaimer: this technique will make sense only if you’re familiar with SSH, the secure shell, for logging into remote hosts. Setting up SSH relationships between hosts is beyond the scope of this book; to learn more about it, seek out an SSH tutorial.

In addition to the usual way of logging into a remote host:

$ ssh myhost.example.com

you also can execute a single command on the remote host—by passing a string to ssh on the command line. Simply append the string to the rest of the ssh command line:

$ ssh myhost.example.com ls
remotefile1
remotefile2
remotefile3

This technique is generally quicker than logging in, running a command, and logging out. If the command includes special characters, such as redirection symbols, that need to be evaluated on the remote host, then quote or escape them. Otherwise, they’ll be evaluated by your local shell. Both of the following commands run ls remotely, but the output redirection occurs on different hosts:

$ ssh myhost.example.com ls > outfile       Creates outfile on local host
$ ssh myhost.example.com "ls > outfile"     Creates outfile on remote host

You can also pipe commands to ssh to run them on the remote host, much like you pipe them to bash to run locally:

$ echo "ls > outfile" | ssh myhost.example.com

When piping commands to ssh, the remote host might print diagnostic or other messages. These generally do not affect the remote command, and you can suppress them:

• If you see messages about pseudo-terminals or pseudo-ttys, such as “Pseudo-terminal will not be allocated because stdin is not a terminal,” run ssh with the -T option to prevent the remote SSH server from allocating a terminal:

  $ echo "ls > outfile" | ssh -T myhost.example.com

• If you see welcome messages that normally appear when you log in (“Welcome to Linux!”) or other unwanted messages, try telling ssh explicitly to run bash on the remote host, and the messages should disappear:

  $ echo "ls > outfile" | ssh myhost.example.com bash

Technique #8: Running a List of Commands with xargs

Many Linux users have never heard of the command xargs, but it’s a powerful tool for constructing and running multiple, similar commands. Learning xargs was another transformative moment in my Linux education, and I hope yours as well.

xargs accepts two inputs:

• On stdin: A list of strings separated by whitespace. An example is file paths produced by ls or find, but any strings will do. I’ll call them the input strings.

• On the command line: An incomplete command that’s missing some arguments, which I’ll call the command template.

xargs merges the input strings and the command template to produce and run new, complete commands, which I’ll call the generated commands. I’ll demonstrate this process with a toy example. Suppose you’re in a directory with three files:

$ ls -1
apple
banana
cantaloupe

Pipe the directory listing to xargs to serve as its input strings, and provide wc -l to serve as the command template, like so:

$ ls -1 | xargs wc -l
3 apple
4 banana
1 cantaloupe
8 total

As promised, xargs applied the wc -l command template to each of the input strings, counting lines in each file. To print the same three files with cat, simply change the command template to “cat”:

$ ls -1 | xargs cat

My toy examples with xargs have two shortcomings, one fatal and one practical. The fatal shortcoming is that xargs may do the wrong thing if an input string contains special characters, such as spaces. A robust solution is in the sidebar “Safety with find and xargs”.

The practical shortcoming is that you don’t need xargs here—you can accomplish the same tasks more simply with file pattern matching:

$ wc -l * 
3 apple
4 banana
1 cantaloupe
8 total

Why use xargs, then? Its power becomes apparent when the input strings are more interesting than a simple directory listing. Suppose you want to count lines in all files in a directory and all its subdirectories (recursively), but only for Python source files with names ending in .py. It’s easy to produce such a list of file paths with find:

$ find . -type f -name \*.py -print
fruits/raspberry.py
vegetables/leafy/lettuce.py
⋮

xargs can now apply the command template wc -l to each file path, producing a recursive result that would be difficult to obtain otherwise. For safety, I’ll replace the option -print with -print0, and xargs with xargs -0, for reasons explained in the sidebar “Safety with find and xargs”:

$ find . -type f -name \*.py -print0 | xargs -0 wc -l
6 ./fruits/raspberry.py
3 ./vegetables/leafy/lettuce.py
⋮

By combining find and xargs, you can empower any command to run recursively through the filesystem, affecting only files (and/or directories) that match your stated criteria. (In some cases, you can produce the same effect with find alone, using its option -exec, but xargs is often a cleaner solution.)

xargs has numerous options (see man xargs) that control how it creates and runs the generated commands. The most important ones in my view (other than -0) are -n and -I. The -n option controls how many arguments are appended by xargs onto each generated command. The default behavior is to append as many arguments as will fit within the shell’s limits:⁷

$ ls | xargs echo                     Fit as many input strings as possible:
apple banana cantaloupe carrot          echo apple banana cantaloupe carrot
$ ls | xargs -n1 echo                 One argument per echo command:
apple                                   echo apple
banana                                  echo banana
cantaloupe                              echo cantaloupe
carrot                                  echo carrot
$ ls | xargs -n2 echo                 Two arguments per echo command:
apple banana                            echo apple banana
cantaloupe carrot                       echo cantaloupe carrot
$ ls | xargs -n3 echo                 Three arguments per echo command:
apple banana cantaloupe                 echo apple banana cantaloupe
carrot                                  echo carrot

The -I option controls where the input strings appear in the generated command. By default, they’re appended to the command template, but you can make them appear elsewhere. Follow -I with any string (of your choice), and that string becomes a placeholder in the command template, indicating exactly where input strings should be inserted:

$ ls | xargs -I XYZ echo XYZ is my favorite food      Use XYZ as a placeholder
apple is my favorite food
banana is my favorite food
cantaloupe is my favorite food
carrot is my favorite food

I chose “XYZ” arbitrarily as a placeholder for input strings and positioned it immediately after echo, moving the input string to the beginning of each output line. Note that the -I option limits xargs to one input string per generated command. I recommend reading the xargs manpage thoroughly to learn what else you can control.

$ rm *.txt
bash: /bin/rm: Argument list too long

$ ls | grep '\.txt$' | xargs rm

$ find . -maxdepth 1 -name \*.txt -type f -print0 \
  | xargs -0 rm

Technique #9: Backgrounding a Command

So far, all our techniques run a command to completion while you wait, and then present the next shell prompt. But you don’t have to wait, especially for commands that take a long time. You can launch commands in a special way so they disappear from sight (sort of) yet continue to run, freeing up the current shell immediately to run further commands. This technique is called backgrounding a command or running a command in the background. In contrast, commands that occupy the shell are called foreground commands. A shell instance runs at most one foreground command at a time plus any number of background commands.

$ wc -c my_extremely_huge_file.txt &     Count characters in a huge file
[1] 74931                                Cryptic-looking response
$

59837483748 my_extremely_huge_file.txt
[1]+  Done               wc -c my_extremely_huge_file.txt

[1]+  Exit 1             wc -c my_extremely_huge_file.txt

$ command1 & command2 & command3 &   All 3 commands
[1] 57351                            in background
[2] 57352
[3] 57353
$ command4 & command5 & echo hi      All in background
[1] 57431                            but "echo"
[2] 57432
hi

$ wc -c my_extremely_huge_file.txt &
[1] 74931

$ sleep 20 &                        Run in the background
[1] 126288
$ jobs                              List this shell's jobs
[1]+  Running          sleep 20 &
$
...eventually...
[1]+  Done             sleep 20

$ sleep 20 &                        Run in the background
[1] 126362
$ fg                                Bring into the foreground
sleep 20
...eventually...
$

$ sleep 20                          Run in the foreground
^Z                                  Suspend the job
[1]+  Stopped          sleep 20
$ jobs                              List this shell's jobs
[1]+  Stopped          sleep 20
$ fg                                Bring into the foreground
sleep 20
...eventually...
[1]+  Done             sleep 20

$ sleep 20                          Run in the foreground
^Z                                  Suspend the job
[1]+  Stopped          sleep 20
$ bg                                Move to the background
[1]+ sleep 20 &
$ jobs                              List this shell's jobs
[1]+  Running          sleep 20 &
$
...eventually...
[1]+  Done             sleep 20

$ sleep 100 &                        Run 3 commands in the background
[1] 126452
$ sleep 200 &
[2] 126456
$ sleep 300 &
[3] 126460
$ jobs                               List this shell's jobs
[1]   Running          sleep 100 &
[2]-  Running          sleep 200 &
[3]+  Running          sleep 300 &
$ fg %2                              Bring job 2 into the foreground
sleep 200
^Z                                   Suspend job 2
[2]+  Stopped          sleep 200
$ jobs                               See job 2 is suspended ("stopped")
[1]   Running          sleep 100 &
[2]+  Stopped          sleep 200
[3]-  Running          sleep 300 &
$ kill %3                            Terminate job 3
[3]+  Terminated       sleep 300
$ jobs                               See job 3 is gone
[1]-  Running          sleep 100 &
[2]+  Stopped          sleep 200
$ bg %2                              Resume suspended job 2 in the background
[2]+ sleep 200 &
$ jobs                               See job 2 is running again
[1]-  Running          sleep 100 &
[2]+  Running          sleep 200 &
$

$ sort /usr/share/dict/words | head -n2 &
[1] 81089
$

$ sort /usr/share/dict/words | head -n2 &
[1] 81089
$ A
A's

[1]+  Done               sort /usr/share/dict/words | head -n2
$

$ sort /usr/share/dict/words | head -n2 > /tmp/results &
[1] 81089
$
[1]+  Done               sort /usr/share/dict/words | head -n2 > /tmp/results
$ cat /tmp/results
A
A's
$

$ cat &
[1] 82455
[1]+  Stopped            cat

$ fg
cat
Here is some input
Here is some input
⋮

$ command1 && command2 && command3 &

Technique #10: Explicit Subshells

Each time you launch a simple command, it runs in a child process, as you saw in “Parent and Child Processes”. Command substitution and process substitution create subshells. There are times, however, when it’s helpful to launch an extra subshell explicitly. To do so, simply enclose a command in parentheses and it runs in a subshell:

$ (cd /usr/local && ls)
bin   etc   games   lib   man   sbin   share
$ pwd
/home/smith                   "cd /usr/local" occurred in a subshell

When applied to a whole command, this technique isn’t super useful, except maybe to save you from running a second cd command to return to your previous directory. However, if you place parentheses around one piece of a combined command, you can perform some useful tricks. A typical example is a pipeline that changes directory in the middle of execution. Suppose you have downloaded a compressed tar file, package.tar.gz, and you want to extract the files. A tar command to extract the files is:

$ tar xvf package.tar.gz
Makefile
src/
src/defs.h
src/main.c
⋮

The extraction occurs relative to the current directory.⁸ What if you want to extract them into a different directory? You could cd to the other directory first and run tar (and then cd back), but you can also perform this task with a single command. The trick is to pipe the tarred data to a subshell that performs directory operations and runs tar as it reads from stdin:⁹

$ cat package.tar.gz | (mkdir -p /tmp/other && cd /tmp/other && tar xzvf -)

This technique also works to copy files from one directory dir1 to another existing directory dir2 using two tar processes, one writing to stdout and one reading from stdin:

$ tar czf - dir1 | (cd /tmp/dir2 && tar xvf -)

The same technique can copy files to an existing directory on another host via SSH:

$ tar czf - dir1 | ssh myhost '(cd /tmp/dir2 && tar xvf -)'

Technique #11: Process Replacement

Normally when you run a command, the shell runs it in a separate process that is destroyed when the command exits, as described in “Parent and Child Processes”. You can change this behavior with the exec command, which is a shell builtin. It replaces the running shell (a process) with another command of your choice (another process). When the new command exits, no shell prompt will follow because the original shell is gone.

To demonstrate this, run a new shell manually and change its prompt:

$ bash                   Run a child shell
$ PS1="Doomed> "         Change the new shell's prompt
Doomed> echo hello       Run any command you like
hello

Now exec a command and watch the new shell die:

Doomed> exec ls          ls replaces the child shell, runs, and exits
animals.txt
$                        A prompt from the original (parent) shell

Why would you ever run exec? One reason is to conserve resources by not launching a second process. Shell scripts sometimes make use of this optimization by running exec on the final command in the script. If the script is run many times (say, millions or billions of executions), the savings might be worth it.

exec has a second ability—it can reassign stdin, stdout, and/or stderr for the current shell. This is most practical in a shell script, such as this toy example that prints information to a file, /tmp/outfile:

#!/bin/bash
echo "My name is $USER"                                 > /tmp/outfile
echo "My current directory is $PWD"                     >> /tmp/outfile
echo "Guess how many lines are in the file /etc/hosts?" >> /tmp/outfile
wc -l /etc/hosts                                        >> /tmp/outfile
echo "Goodbye for now"                                  >> /tmp/outfile

Instead of redirecting the output of each command to /tmp/outfile individually, use exec to redirect stdout to /tmp/outfile for the entire script. Subsequent commands can simply print to stdout:

#!/bin/bash
# Redirect stdout for this script
exec > /tmp/outfile2
# All subsequent commands print to /tmp/outfile2
echo "My name is $USER"
echo "My current directory is $PWD"
echo "Guess how many lines are in the file /etc/hosts?"
wc -l /etc/hosts
echo "Goodbye for now"

Run this script and examine the file /tmp/outfile2 to see the results:

$ cat /tmp/outfile2
My name is smith
My current directory is /home/smith
Guess how many lines are in the file /etc/hosts?
122 /etc/hosts
Goodbye for now

You probably won’t use exec often, but it’s there when you need it.

Be Flexible

A key to writing brash one-liners is flexibility. You’ve learned some awesome tools by this point—a core set of Linux programs (and umpteen ways to run them) along with command history, command-line editing, and more. You can combine these tools in many ways, and a given problem usually has multiple solutions.

Even the simplest Linux tasks can be accomplished in many ways. Consider how you might list .jpg files in your current directory. I’ll bet 99.9% of Linux users would run a command like this:

$ ls *.jpg

But this is just one solution of many. For example, you could list all the files in the directory and then use grep to match only the names ending in .jpg:

$ ls | grep '\.jpg$'

Why would you choose this solution? Well, you saw an example in “Long Argument Lists”, when a directory contained so many files that they couldn’t be listed by pattern matching. The technique of grepping for a filename extension is a robust, general approach for solving all sorts of problems. What’s important here is to be flexible and understand your tools so you can apply the best one in your time of need. That is a wizard’s skill when creating brash one-liners.

All of the following commands list .jpg files in the current directory. Try to figure out how each command works:

$ echo $(ls *.jpg)
$ bash -c 'ls *.jpg'
$ cat <(ls *.jpg)
$ find . -maxdepth 1 -type f -name \*.jpg -print
$ ls > tmp && grep '\.jpg$' tmp && rm -f tmp
$ paste <(echo ls) <(echo \*.jpg) | bash
$ bash -c 'exec $(paste <(echo ls) <(echo \*.jpg))'
$ echo 'monkey *.jpg' | sed 's/monkey/ls/' | bash
$ python -c 'import os; os.system("ls *.jpg")'

Are the results identical or do some commands behave a bit differently? Can you come up with any other suitable commands?

Think About Where to Start

Every brash one-liner begins with the output of a simple command. That output might be the contents of a file, part of a file, a directory listing, a sequence of numbers or letters, a list of users, a date and time, or other data. Your first challenge, therefore, is to produce the initial data for your command.

For example, if you want to know the 17th letter of the English alphabet, then your initial data could be 26 letters produced by brace expansion:

$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Once you can produce this output, the next step is deciding how to massage it to fit your goal. Do you need to slice the output by rows or columns? Join the output with other information? Transform the output in a more complicated way? Look to the programs in Chapters 1 and 5 to do that work, like grep and sed and cut, and apply them using the techniques of Chapter 7.

For this example, you could print the 17th field with awk, or remove spaces with sed and locate the 17th character with cut:

$ echo {A..Z} | awk '{print $(17)}'
Q
$ echo {A..Z} | sed 's/ //g' | cut -c17
Q

As another example, if you want to print the months of the year, your initial data could be the numbers 1 through 12, again produced by brace expansion:

$ echo {1..12}
1 2 3 4 5 6 7 8 9 10 11 12

From there, augment the brace expansion so it forms dates for the first day of each month (from 2021-01-01 through 2021-12-01); then run date -d on each line to produce month names:

$ echo 2021-{01..12}-01 | xargs -n1 date +%B -d
January
February
March
⋮
December

Or, suppose you want to know the length of the longest filename in the current directory. Your initial data could be a directory listing:

$ ls
animals.txt  cartoon-mascots.txt  ...  zebra-stripes.txt

From there, use awk to generate commands to count characters in each filename with wc -c:

$ ls | awk '{print "echo -n", $0, "| wc -c"}'
echo -n "animals.txt" | wc -c
echo -n "cartoon-mascots.txt | wc -c"
⋮
echo -n "zebra-stripes.txt | wc -c"

(The -n option prevents echo from printing newline characters, which would throw off each count by one.) Finally, pipe the commands to bash to run them, sort the numeric results from high to low, and grab the maximum value (the first line) with head -n1:

$ ls | awk '{print "echo -n", $0, "| wc -c"}' | bash | sort -nr | head -n1
23

This last example was tricky, generating pipelines as strings and passing them to a further pipeline. Nevertheless, the general principle is the same: figure out your starting data and manipulate it to fit your needs.

Know Your Testing Tools

Building a brash one-liner may require trial and error. The following tools and techniques will help you try different solutions quickly:

Use command history and command-line editing.

Don’t retype commands while you experiment. Use techniques from Chapter 3 to recall previous commands, tweak them, and run them.

Add echo to test your expressions.

If you aren’t sure how an expression will evaluate, print it with echo beforehand to see the evaluated results on stdout.

Use ls or add echo to test destructive commands.

If your command invokes rm, mv, cp, or other commands that might overwrite or remove files, place echo in front of them to confirm which files will be affected. (So, instead of executing rm, execute echo rm.) Another safety tactic is to replace rm with ls to list files that would be removed.

Insert a tee to view intermediate results.

If you want to view the output (stdout) in the middle of a long pipeline, insert the tee command to save output to a file for examination. The following command saves the output from command3 in the file outfile, while piping that same output to command4:

$ command1 | command2 | command3 | tee outfile | command4 | command5
$ less outfile

OK, let’s build some brash one-liners!