Ubuntu 15.04 Server with systemd: Administration and Reference (2015)

Part V. Shells

Chapter 19. Shells

The shell is a command interpreter that provides a line-oriented interactive and non-interactive interface between the user and the operating system. You enter commands on a command line; they are interpreted by the shell and then sent as instructions to the operating system (the command line interface is accessible from GNOME and KDE through a Terminal windows – Applications lens /Accessories filter). You can also place commands in a script file to be consecutively executed, much like a program. This interpretive capability of the shell provides for many sophisticated features. For example, the shell has a set of file expansion characters that can generate filenames. The shell can redirect input and output, as well as run operations in the background, freeing you to perform other tasks.

Table 19-1: Linux Shells

Several different types of shells have been developed for Linux: the Bourne Again shell (BASH), the Korn shell, the TCSH shell, and the Z shell. All shells are available for your use, although the BASH shell is the default. You only need one type of shell to do your work. Ubuntu Linux includes all the major shells, although it installs and uses the BASH shell as the default. If you use the command line shell, you will be using the BASH shell unless you specify another. This chapter discusses the BASH shell, which shares many of the same features as other shells.

You can find out more about shells at their respective Web sites as listed in Table 19-1 . Also, a detailed online manual is available for each installed shell. Use the man command and the shell’s keyword to access them, bash for the BASH shell, ksh for the Korn shell, zsh for the Z shell, and tsch for the TSCH shell. For example, the command man bash will access the BASH shell online manual.

Note: You can find out more about the BASH shell at www.gnu.org/software/bash. A detailed online manual is available on your Linux system using the man command with the bash keyword.

The Command Line

The Linux command line interface consists of a single line into which you enter commands with any of their options and arguments. From GNOME or KDE, you can access the command line interface by opening a terminal window (Applications | Accessories | Terminal). Should you start Linux with the command line interface, you will be presented with a BASH shell command line when you log in.

By default, the BASH shell has a dollar sign ($) prompt, but Linux has several other types of shells, each with its own prompt (like % for the C shell). The root user will have a different prompt, the #. A shell prompt, such as the one shown here, marks the beginning of the command line:

$

You can enter a command along with options and arguments at the prompt. For example, with an -l option, the ls command will display a line of information about each file, listing such data as its size and the date and time it was last modified. In the next example, the user enters the lscommand followed by a -l option. The dash before the -l option is required. Linux uses it to distinguish an option from an argument.

$ ls -l

If you wanted only the information displayed for a particular file, you could add that file’s name as the argument, following the -l option:

$ ls -l mydata
-rw-r--r-- 1 chris weather 207 Feb 20 11:55 mydata

Tip: Some commands can be complex and take some time to execute. When you mistakenly execute the wrong command, you can interrupt and stop such commands with the interrupt key—CTRL-C.

You can enter a command on several lines by typing a backslash just before you press ENTER. The backslash “escapes” the ENTER key, effectively continuing the same command line to the next line. In the next example, the cp command is entered on three lines. The first two lines end in a backslash, effectively making all three lines one command line.

$ cp -i \
mydata \
/home/george/myproject/newdata

You can also enter several commands on the same line by separating them with a semicolon (;). In effect the semicolon operates as an execute operation. Commands will be executed in the sequence in which they are entered. The following command executes an ls command followed by a date command.

$ ls ; date

You can also conditionally run several commands on the same line with the && operator. A command is executed only if the previous command is true. This feature is useful for running several dependent scripts on the same line. In the next example, the ls command is run only if the datecommand is successfully executed.

$ date && ls

TIP: Command can also be run as arguments on a command line, using their results for other commands. To run a command within a command line, you encase the command in back quotes.

Table 19-2: Command Line Editing Operations

Command Line Editing

The BASH shell, which is your default shell, has special command line editing capabilities that you may find helpful as you learn Linux (see Table 19-2 ). You can easily modify commands you have entered before executing them, moving anywhere on the command line and inserting or deleting characters. This is particularly helpful for complex commands.

You can press CTRL-F or the RIGHT ARROW key to move forward a character, or the CTRL-B or LEFT ARROW key to move back a character. CTRL-D or DEL deletes the character the cursor is on, and CTRL-H or BACKSPACE deletes the character preceding the cursor. To add text, you use the arrow keys to move the cursor to where you want to insert text and type the new characters.

You can even cut words with the CTRL-W or ALT-D key, and then press the CTRL-Y key to paste them back in at a different position, effectively moving the words. As a rule, the CTRL version of the command operates on characters, and the ALT version works on words, such as CTRL-T to transpose characters and ALT-T to transpose words. At any time, you can press ENTER to execute the command. For example, if you make a spelling mistake when entering a command, rather than re-entering the entire command, you can use the editing operations to correct the mistake. The actual associations of keys and their tasks, along with global settings, are specified in the /etc/inputrc file.

The editing capabilities of the BASH shell command line are provided by Readline. Readline supports numerous editing operations. You can even bind a key to a selected editing operation. Readline uses the /etc/inputrc file to configure key bindings. This file is read automatically by your /etc/profile shell configuration file when you log in. Users can customize their editing commands by creating an .inputrc file in their home directory (this is a dot file). It may be best to first copy the /etc/inputrc file as your .inputrc file and then edit it. /etc/profile will first check for a local .inputrc file before accessing the /etc/inputrc file. You can find out more about Readline in the BASH shell reference manual at www.gnu.org/manual/bash.

Command and Filename Completion

The BASH command line has a built-in feature that performs command line and file name completion. Automatic completions can be effected by pressing the TAB key. If you enter an incomplete pattern as a command or filename argument, you can press the TAB key to activate the command and filename completion feature, which completes the pattern. A directory will have a forward slash (/) attached to its name. If more than one command or file has the same prefix, the shell simply beeps and waits for you to press the TAB key again. It then displays a list of possible command completions and waits for you to add enough characters to select a unique command or filename. For situations where you know multiple possibilities are likely, you can just press the ESC key instead of two TABs. In the next example, the user issues a cat command with an incomplete filename. When the user presses the TAB key, the system searches for a match and, when it finds one, fills in the filename. The user can then press ENTER to execute the command.

$ cat pre <tab>
$ cat preface

The automatic completions also work with the names of variables, users, and hosts. In this case, the partial text needs to be preceded by a special character, indicating the type of name. A listing of possible automatic completions follows:

Filenames begin with any text or /.

Shell variable text begins with a $ sign.

User name text begins with a ~ sign.

Host name text begins with a @.

Commands, aliases, and text in files begin with normal text.

Variables begin with a $ sign, so any text beginning with a dollar sign is treated as a variable to be completed. Variables are selected from previously defined variables, like system shell variables. User names begin with a tilde (~). Host names begin with a @ sign, with possible names taken from the /etc/hosts file. For example, to complete the variable HOME given just $HOM, simply press a tab key.

$ echo $HOM <tab>
$ echo $HOME

If you entered just an H, then you could press TAB twice to see all possible variables beginning with H. The command line is redisplayed, letting you complete the name.

$ echo $H <tab> <tab>
$HISTCMD $HISTFILE $HOME $HOSTTYPE HISTFILE $HISTSIZE $HISTNAME
$ echo $H

You can also specifically select the kind of text to complete, using corresponding command keys. In this case, it does not matter what kind of sign a name begins with.

Table 19-3: Command Line Text Completion Commands

For example, the pressing ALT-~ will treat the current text as a user name. Pressing ALT-@ will treat it as a host name, and ALT-$, as a variable. Pressing ALT-! will treat it as a command. To display a list of possible completions, press the CTRL-X key with the appropriate completion key, as in CTRL-X-$ to list possible variable completions. See Table 19-3 for a complete listing.

History

The BASH shell keeps a history list, of your previously entered commands. You can display each command, in turn, on your command line by pressing the UP ARROW key. Press the DOWN ARROW key to move down the list. You can modify and execute any of these previous commands when you display them on the command line.

Tip: The ability to redisplay a command is helpful when you’ve already executed a command you had entered incorrectly. In this case, you would be presented with an error message and a new, empty command line. By pressing the UP ARROW key, you can redisplay the previous command, make corrections to it, and then

History Events

In the BASH shell, the history utility keeps a record of the most recent commands you have executed. The commands are numbered starting at 1, and a limit exists to the number of commands remembered—the default is 500. The history utility is a kind of short-term memory, keeping track of the most recent commands you have executed. To see the set of your most recent commands, type history on the command line and press ENTER. A list of your most recent commands is then displayed, preceded by a number.

$ history
1 cp mydata today
2 vi mydata
3 mv mydata reports
4 cd reports
5 ls

Each of these commands is technically referred to as an event. An event describes an action that has been taken—a command that has been executed. The events are numbered according to their sequence of execution. The most recent event has the highest number. Each of these events can be identified by its number or beginning characters in the command.

The history utility lets you reference a former event, placing it on your command line so you can execute it. The easiest way to do this is to use the UP ARROW and DOWN ARROW keys to place history events on the command line, one at a time. You needn’t display the list first withhistory. Pressing the UP ARROW key once places the last history event on the command line. Pressing it again places the next history event on the command line. Pressing the DOWN ARROW key places the previous event on the command line.

You can use certain control and meta keys to perform other history operations like searching the history list. A meta key is the ALT key, and the ESC key on keyboards that have no ALT key. The ALT key is used here. Pressing ALT-< will move you to the beginning of the history list; ALT-N will search it. CTRL-S and CTRL-R will perform incremental searches, display matching commands as you type in a search string. Table 19-4 lists the different commands for referencing the history list.

Tip: If more than one history event matches what you have entered, you will hear a beep, and you can then enter more characters to help uniquely identify the event.

You can also reference and execute history events using the ! history command. The ! is followed by a reference that identifies the command. The reference can be either the number of the event or a beginning set of characters in the event. In the next example, the third command in the history list is referenced first by number and then by the beginning characters:

$ !3
mv mydata reports
$ !mv my
mv mydata reports

Table 19-4: History Commands and History Event References

You can also reference an event using an offset from the end of the list. A negative number will offset from the end of the list to that event, thereby referencing it. In the next example, the fourth command, cdmydata, is referenced using a negative offset, and then executed. Remember that you are offsetting from the end of the list—in this case, event 5—up toward the beginning of the list, event 1. An offset of 4 beginning from event 5 places you at event 2.

$ !-4
vi mydata

To reference the last event, you use a following !, as in !!. In the next example, the command !! executes the last command the user executed—in this case, ls:

$ !!
ls
mydata today reports

Filename Expansion: *, ?, [ ]

Filenames are the most common arguments used in a command. Often you will know only part of the filename, or you will want to reference several filenames that have the same extension or begin with the same characters. The shell provides a set of special characters that search out, match, and generate a list of filenames. These are the asterisk, the question mark, and brackets (*, ?, []). Given a partial filename, the shell uses these matching operators to search for files and expand to a list of filenames found. The shell replaces the partial filename argument with the expanded list of matched filenames. This list of filenames can then become the arguments for commands such as ls, which can operate on many files. Table 19-5 lists the shell’s file expansion characters.

Matching Multiple Characters

The asterisk (*) references files beginning or ending with a specific set of characters. You place the asterisk before or after a set of characters that form a pattern to be searched for in filenames.

If the asterisk is placed before the pattern, filenames that end in that pattern are searched for. If the asterisk is placed after the pattern, filenames that begin with that pattern are searched for. Any matching filename is copied into a list of filenames generated by this operation.

In the next example, all filenames beginning with the pattern “doc” are searched for and a list generated. Then all filenames ending with the pattern “day” are searched for and a list is generated. The last example shows how the * can be used in any combination of characters.

$ ls
doc1 doc2 document docs mydoc monday tuesday
$ ls doc*
doc1 doc2 document docs
$ ls *day
monday tuesday
$ ls m*d*
monday
$

Filenames often include an extension specified with a period and followed by a string denoting the file type, such as .c for C files, .cpp for C++ files, or even .jpg for JPEG image files. The extension has no special status, and is only part of the characters making up the filename. Using the asterisk makes it easy to select files with a given extension. In the next example, the asterisk is used to list only those files with a .c extension. The asterisk placed before the .c constitutes the argument for ls.

$ ls *.c
calc.c main.c

You can use * with the rm command to erase several files at once. The asterisk first selects a list of files with a given extension, or beginning or ending with a given set of characters, and then it presents this list of files to the rm command to be erased. In the next example, the rm command erases all files beginning with the pattern “doc”:

$ rm doc*

Table 19-5: Shell Symbols

Tip: Use the * file expansion character carefully and sparingly with the rm command. The combination can be dangerous. A misplaced * in an rm command without the -i option could easily erase all the files in your current directory. The -i option will first prompt the user to confirm whether the file should be deleted.

Matching Single Characters

The question mark (?) matches only a single incomplete character in filenames. Suppose you want to match the files doc1 and docA, but not the file document. Whereas the asterisk will match filenames of any length, the question mark limits the match to one extra character. The next example matches files that begin with the word “doc” followed by a single differing letter:

$ ls
doc1 docA document
$ ls doc?
doc1 docA

Matching a Range of Characters

Whereas the * and ? file expansion characters specify incomplete portions of a filename, the brackets ([]) enable you to specify a set of valid characters to search for. Any character placed within the brackets will be matched in the filename. Suppose you want to list files beginning with “doc”, but only ending in 1 or A. You are not interested in filenames ending in 2 or B, or any other character. Here is how it’s done:

$ ls
doc1 doc2 doc3 docA docB docD document
$ ls doc[1A]
doc1 docA

You can also specify a set of characters as a range, rather than listing them one by one. A dash placed between the upper and lower bounds of a range of characters selects all characters within that range. The range is usually determined by the character set in use. In an ASCII character set, the range “a-g” will select all lowercase alphabetic characters from a through g, inclusive. In the next example, files beginning with the pattern “doc” and ending in characters 1 through 3 are selected. Then, those ending in characters B through E are matched.

$ ls doc[1-3]
doc1 doc2 doc3
$ ls doc[B-E]
docB docD

You can combine the brackets with other file expansion characters to form flexible matching operators. Suppose you want to list only filenames ending in either a .c or .o extension, but no other extension. You can use a combination of the asterisk and brackets: *[co]. The asterisk matches all filenames, and the brackets match only filenames with extension .c or .o.

$ ls *.[co]
main.c main.o calc.c

Matching Shell Symbols

At times, a file expansion character is actually part of a filename. In these cases, you need to quote the character by preceding it with a backslash (\) to reference the file. In the next example, the user needs to reference a file that ends with the ? character, called answers?. The ? is, however, a file expansion character and would match any filename beginning with “answers” that has one or more characters. In this case, the user quotes the ? with a preceding backslash to reference the filename.

$ ls answers\?
answers?

Placing the filename in double quotes will also quote the character.

$ ls "answers?"
answers?

This is also true for filenames or directories that have white space characters like the space character. In this case you could either use the backslash to quote the space character in the file or directory name, or place the entire name in double quotes.

$ ls My\ Documents
My Documents
$ ls "My Documents"
My Documents

Generating Patterns

Though not a file expansion operation, {} is often useful for generating names that you can use to create or modify files and directories. The braces operation only generates a list of names. It does not match on existing filenames. Patterns are placed within the braces and separated with commas. Any pattern placed within the braces will be used to generate a version of the pattern, using either the preceding or following pattern, or both. Suppose you want to generate a list of names beginning with “doc”, but ending only in the patterns “ument”, “final”, and “draft”. Here is how it’s done:

$ echo doc{ument,final,draft}
document docfinal docdraft

Since the names generated do not have to exist, you could use the {} operation in a command to create directories, as shown here:

$ mkdir {fall,winter,spring}report
$ ls
fallreport springreport winterreport

Standard Input/Output and Redirection

The data in input and output operations are organized like a file. Data input at the keyboard is placed in a data stream arranged as a continuous set of bytes. Data output from a command or program is also placed in a data stream and arranged as a continuous set of bytes. This input data stream is referred to in Linux as the standard input, while the output data stream is called the standard output. A separate output data stream reserved solely for error messages is called the standard error.

Because the standard input and standard output have the same organization as that of a file, they can easily interact with files. Linux has a redirection capability that lets you easily move data in and out of files. You can redirect the standard output so that, instead of displaying the output on a screen, you can save it in a file. You can also redirect the standard input away from the keyboard to a file, so that input is read from a file instead of from your keyboard.

When a Linux command is executed that produces output, this output is placed in the standard output data stream. The default destination for the standard output data stream is a device—in this case, the screen. Devices, such as the keyboard and screen, are treated as files. They receive and send out streams of bytes with the same organization as that of a byte-stream file. The screen is a device that displays a continuous stream of bytes. By default, the standard output will send its data to the screen device, which will then display the data.

For example, the ls command generates a list of all filenames and outputs this list to the standard output. Next, this stream of bytes in the standard output is directed to the screen device. The list of filenames is then printed on the screen. The cat command also sends output to the standard output. The contents of a file are copied to the standard output, whose default destination is the screen. The contents of the file are then displayed on the screen.

Table 19-6: The Shell Operations

Redirecting the Standard Output: > and >>

Suppose that instead of displaying a list of files on the screen, you would like to save this list in a file. In other words, you would like to direct the standard output to a file rather than the screen. To do this, you place the output redirection operator, the greater-than sign (>), followed by the name of a file on the command line after the Linux command. Table 19-6 lists the different ways you can use the redirection operators. In the next example, the output of the ls command is redirected from the screen device to a file:

$ ls -l *.c > programlist

The redirection operation creates the new destination file. If the file already exists, it will be overwritten with the data in the standard output. You can set the noclobber feature to prevent overwriting an existing file with the redirection operation. In this case, the redirection operation on an existing file will fail. You can overcome the noclobber feature by placing an exclamation point after the redirection operator. You can place the noclobber command in a shell configuration file to make it an automatic default operation. The next example sets the noclobber feature for the BASH shell and then forces the overwriting of the oldarticle file if it already exists:

$ set -o noclobber
$ cat myarticle >! oldarticle

Although the redirection operator and the filename are placed after the command, the redirection operation is not executed after the command. In fact, it is executed before the command. The redirection operation creates the file and sets up the redirection before it receives any data from the standard output. If the file already exists, it will be destroyed and replaced by a file of the same name. In effect, the command generating the output is executed only after the redirected file has been created.

In the next example, the output of the ls command is redirected from the screen device to a file. First the ls command lists files, and in the next command, ls redirects its file list to the listf file. Then the cat command displays the list of files saved in listf. Notice the list of files in listfincludes the listf filename. The list of filenames generated by the ls command includes the name of the file created by the redirection operation—in this case, listf. The listf file is first created by the redirection operation, and then the ls command lists it along with other files. This file list output by ls is then redirected to the listf file, instead of being printed on the screen.

$ ls
mydata intro preface
$ ls > listf
$ cat listf
mydata intro listf preface

Tip: Errors occur when you try to use the same filename for both an input file for the command and the redirected destination file. In this case, because the redirection operation is executed first, the input file, because it exists, is destroyed and replaced by a file of the same name. When the command is executed, it finds an input file that is empty.

You can also append the standard output to an existing file using the >> redirection operator. Instead of overwriting the file, the data in the standard output is added at the end of the file. In the next example, the myarticle and oldarticle files are appended to the allarticles file. Theallarticles file will then contain the contents of both myarticle and oldarticle.

$ cat myarticle >> allarticles
$ cat oldarticle >> allarticles

The Standard Input

Many Linux commands can receive data from the standard input. The standard input itself receives data from a device or a file. The default device for the standard input is the keyboard. Characters typed on the keyboard are placed in the standard input, which is then directed to the Linux command. Just as with the standard output, you can also redirect the standard input, receiving input from a file rather than the keyboard. The operator for redirecting the standard input is the less-than sign (<). In the next example, the standard input is redirected to receive input from themyarticle file, rather than the keyboard device (use CTRL-D to end the typed input). The contents of myarticle are read into the standard input by the redirection operation. Then the cat command reads the standard input and displays the contents of myarticle.

$ cat < myarticle
hello Christopher
How are you today
$

You can combine the redirection operations for both standard input and standard output. In the next example, the cat command has no filename arguments. Without filename arguments, the cat command receives input from the standard input and sends output to the standard output. However, the standard input has been redirected to receive its data from a file, while the standard output has been redirected to place its data in a file.

$ cat < myarticle > newarticle

Redirecting the Standard Error: >&, 2>, |&

When you execute commands, it is possible for an error to occur. You may give the wrong number of arguments or some kind of system error could take place. When an error occurs, the system will issue an error message. Usually such error messages are displayed on the screen along with the standard output. Error messages are placed in another standard byte stream called the standard error. In the next example, the cat command is given as its argument the name of a file that does not exist, myintro. In this case, the cat command will simply issue an error. Redirection operators are listed in Table 19-6 .

$ cat myintro
cat : myintro not found

Because error messages are in a separate data stream from the standard output, this means that if you have redirected the standard output to a file, error messages will still appear on the screen for you to see. Though the standard output may be redirected to a file, the standard error is still directed to the screen. In the next example, the standard output of the cat command is redirected to the file mydata. The standard error, containing the error messages, is still directed toward the screen

$ cat myintro > mydata
cat : myintro not found

Like the standard output, you can also redirect the standard error. This means that you can save your error messages in a file for future reference. This is helpful if you need to save a record of the error messages. Like the standard output, the standard error's default destination is the display. Using special redirection operators, you can redirect the standard error to any file or device that you choose. If you redirect the standard error, the error messages will not be displayed on the screen. You can examine them later by viewing the contents of the file in which you saved them.

All the standard byte streams can be referenced in redirection operations with numbers. The numbers 0, 1, and 2 reference the standard input, standard output, and standard error respectively. By default an output redirection, >, operates on the standard output, 1. You can modify the output redirection to operate on the standard error by preceding the output redirection operator with the number 2, 2>. In the next example, the cat command again will generate an error. The error message is redirected to the standard byte stream represented by number 2, the standard error.

$ cat nodata 2> myerrors
$ cat myerrors
cat : nodata not found

You can also append the standard error to a file by using the number 2 and the redirection append operator, >>. In the next example, the user appends the standard error to the myerrors file, which then functions as a log of errors.

$ cat nodata 2>> myerrors
$ cat compls 2>> myerrors
$ cat myerrors
cat : nodata not found
cat : compls not found
$

To redirect both the standard output as well as the standard error, you would need a separate redirection operation and file for each. In the next example, the standard output is redirected to the file mydata, and the standard error is redirected to myerrors. If nodata were to exist, thenmydata would hold a copy of its contents.

$ cat nodata 1> mydata 2> myerrors
cat myerrors
cat : nodata not found

If, however, you want to save a record of your errors in the same file as that used for the redirected standard output, you need to redirect the standard error into the standard output. You can reference a standard byte stream by preceding its number with an ampersand. &1 references the standard output. You can use such a reference in a redirection operation to make a standard byte stream a destination file. The redirection operation 2>&1 redirects the standard error into the standard output. In effect, the standard output becomes the destination file for the standard error. Conversely the redirection operation 1>&2 would redirect the standard input into the standard error.

Pipes: |

You may encounter situations in which you need to send data from one command to another. In other words, you may want to send the standard output of a command to another command, rather than to a destination file. Suppose you want to send a list of your filenames to the printer to be printed. You need two commands to do this: the ls command to generate a list of filenames and the lpr command to send the list to the printer. In effect, you need to take the output of the ls command and use it as input for the lpr command. You can think of the data as flowing from one command to another. To form such a connection in Linux, you use what is called a pipe. The pipe operator (|, the vertical bar character) placed between two commands forms a connection between them. The standard output of one command becomes the standard input for the other. The pipe operation receives output from the command placed before the pipe and sends this data as input to the command placed after the pipe. As shown in the next example, you can connect the ls command and the lpr command with a pipe. The list of filenames output by the ls command is piped into the lpr command.

$ ls | lpr

You can combine the pipe operation with other shell features, such as file expansion characters, to perform specialized operations. The next example prints only files with a .c extension. The ls command is used with the asterisk and “.c” to generate a list of filenames with the .c extension. Then this list is piped to the lpr command.

$ ls *.c | lpr

In the preceding example, a list of filenames was used as input, but what is important to note is that pipes operate on the standard output of a command, whatever that might be. The contents of whole files, or even several files, can be piped from one command to another. In the next example, the cat command reads and outputs the contents of the mydata file, which are then piped to the lpr command:

$ cat mydata | lpr

Linux has many commands that generate modified output. For example, the sort command takes the contents of a file and generates a version with each line sorted in alphabetic order. The sort command works best with files that are lists of items. Commands such as sort that output a modified version of its input are referred to as filters. Filters are often used with pipes. In the next example, a sorted version of mylist is generated and piped into the more command for display on the screen. The original file, mylist, has not been changed and is not sorted. Only the output ofsort in the standard output is sorted.

$ sort mylist | more

The standard input piped into a command can be more carefully controlled with the standard input argument (-). When you use the dash as an argument for a command, it represents the standard input.