Command-Line Master Class Part 1 - System Administration - Ubuntu Unleashed 2017 Edition (2017)

Ubuntu Unleashed 2017 Edition (2017)

Part III: System Administration

Chapter 11. Command-Line Master Class Part 1


In This Chapter

Image Why Use the Command Line?

Image Using Basic Commands

Image References


Some Linux users like to focus on the graphical environments that are available—they rush to tell new users that the command line isn’t vital when using Linux. Although there are some amazing graphical user interface (GUI)desktops and this statement is mostly true, avoiding the command line limits your options and makes some tasks more difficult. The command-line interface is where the greatest power and flexibility are found, and those who actively avoid learning how to use it are also actively limiting their abilities and options. You learned the basics earlier in Chapter 10, “Command-Line Beginner’s Class.” In this chapter, we dig in deeper.

It is with some trepidation that this two-chapter set retains its classic title “Command-Line Master Class.” Entire books have been published covering the depth and breadth of the command line. To believe that two short chapters make any reader a true master is foolish. Our greatest hope is to give enough information to enable any reader to perform all basic and vital tasks from the command line while inspiring readers to go on a quest to discover all the beauty and grandeur that we do not have space to cover here. Please keep this in mind as we continue.

In his book The Art of Unix Programming, Eric Raymond wrote a short story that perfectly illustrates the power of the command line versus the GUI. It’s reprinted here with permission, for your reading pleasure:

One evening, Master Foo and Nubi attended a gathering of programmers who had met to learn from each other. One of the programmers asked Nubi to what school he and his master belonged. Upon being told they were followers of the Great Way of Unix, the programmer grew scornful.

“The command-line tools of Unix are crude and backward,” he scoffed.

“Modern, properly designed operating systems do everything through a graphical user interface.”

Master Foo said nothing, but pointed at the moon. A nearby dog began to bark at the master’s hand.

“I don’t understand you!” said the programmer.

Master Foo remained silent, and pointed at an image of the Buddha. Then he pointed at a window. “What are you trying to tell me?” asked the programmer.

Master Foo pointed at the programmer’s head. Then he pointed at a rock.

“Why can’t you make yourself clear?” demanded the programmer.

Master Foo frowned thoughtfully, tapped the programmer twice on the nose, and dropped him in a nearby trash can.

As the programmer was attempting to extricate himself from the garbage, the dog wandered over and piddled on him.

At that moment, the programmer achieved enlightenment.

Whimsical as the story is, it does illustrate that there are some things that the GUI just does not do well. Enter the command line: It is a powerful and flexible operating environment and—if you practice—can actually be quite fun, too!

In this chapter, you learn more commands to help you master the command line so that you can perform common tasks through it.

Why Use the Command Line?

Moving from the GUI to the command line is a conscious choice for most people, although it is increasingly rare that it is an absolute choice accompanied by a complete abandoning of GUIs.

Reasons for using the command line include the following:

Image You want to chain two or more commands together.

Image You want to use a command or parameter available only on the shell.

Image You are working on a text-only system.

Image You have used it for a long time and feel comfortable there.

Image You want to automate a task.

Chaining two or more commands together, or piping, is what gives the shell its real power. Hundreds of commands are available, and by combining them in different ways you get tons of new options. Some of the shell commands are available through the GUI, but these commands usually have only a small subset of their parameters available, which limits what you can do with them.

Working from a text-only system is useful both for working locally with a broken GUI and for connecting to a remote, text-only system. If your Linux server is experiencing problems, the last thing you want to do is load it down with a GUI connection; working in text mode is faster and more efficient.

Many people use the shell simply because it is familiar to them. Some people even use the shell to start GUI applications just because it saves them taking their hands off the keyboard for a moment. This is not a bad thing; it provides fluency and ease with the system and is a perfectly valid way of working. Working from the command line is faster. The mouse is slow, and taking your fingers away from the keyboard makes your work even slower. Anyone looking to achieve the Zen-like power user state hinted at by Eric Raymond will understand this after making the effort to learn.

Knowing how to work in the shell is also essential if you wish to automate tasks on your system without use of the GUI. Whether you want to add a periodic task to cron or update a configuration management system, you will need to know what text commands to give in order to run programs from the command line.

Using Basic Commands

It is impossible to know how many commands the average command-line citizen uses, but if we had to guess, we would place it at about 25. Some of these were introduced in Chapter 10, “Command-Line Beginner’s Class,” but are covered here with greater depth. Others will be new to the reader. Still others are only mentioned in this list to give the reader ideas for further study. Here are some commands that every command-line user will want to learn:

Image cat—Prints the contents of a file

Image cd—Changes directories

Image chmod—Changes file access permissions

Image cp—Copies files

Image du—Prints disk usage

Image emacs—Edits text files

Image find—Finds files by searching

Image grep—Searches for a string in input

Image less—Filters for paging through output

Image ln—Creates links between files

Image locate—Finds files from an index

Image ls—Lists files in the current directory

Image make—Compiles and installs programs

Image man—Displays manual pages for reading

Image mkdir—Makes directories

Image mv—Moves files

Image nano—Edits text files

Image rm—Deletes files and directories

Image sort—Takes a text file as input and outputs the contents of the file in the order you specify

Image ssh—Connects to other machines using a secure shell connection

Image tail—Prints the last lines of a file

Image vim—Edits text files

Image which—Prints the location of a command

Many other commands also are available to you that get used fairly often—cut, diff, gzip, history, ping, su, tar, uptime, who, and so on—but if you can understand the ones listed here, you have sufficient skill to concoct your own command combinations.

Note that we say understand the commands, not know all their possible parameters and usages. This is because several of the commands, although commonly used, are used only in any complex manner by people with specific needs. make is a good example of this: Unless you plan to become a programmer, you need not worry about this beyond just typing make and make install now and then. If you want to learn more, see Chapter 38, “Using Programming Tools for Ubuntu.”

Similarly, emacs, nano and vim are text editors that have a text-based interface all their own and are covered later in this chapter, and SSH is covered in detail in Chapter 19, “Remote Access with SSH, Telnet, and VNC.”

What remains is a set of commands, each of which has many parameters you can use to customize what it actually does. Again, we can eliminate much of this because many of the parameters are esoteric and rarely used, and the few times in your Linux life that you need them, you can just read the manual page.

We go over these commands one by one, explaining the most common ways to use them.

Printing the Contents of a File with cat

Many of Ubuntu’s shell commands manipulate text strings, so if you want to be able to feed them the contents of files, you need to be able to output those files as text. Enter the cat command, which prints the contents of any files you pass to it.

Its most basic use is like this:

Click here to view code image

matthew@seymour:~$ cat myfile.txt

That prints the contents of myfile.txt. For this usage there are two extra parameters that are often used: -n numbers the lines in the output, and -s (“squeeze”) prints a maximum of one blank line at a time. That is, if your file has 1 line of text, 10 blank lines, 1 line of text, 10 blank lines, and so on, -s shows the first line of text, a single blank line, the next line of text, a single blank line, and so forth. When you combine -s and -n, cat numbers only the lines that are printed—the 10 blank lines shown as one will count as 1 line for numbering.

This command prints information about your CPU, stripping out multiple blank lines and numbering the output:

Click here to view code image

matthew@seymour:~$ cat -sn /proc/cpuinfo

You can also use cat to print the contents of several files at once, like this:

Click here to view code image

matthew@seymour:~$ cat -s myfile.txt myotherfile.txt

In that command, cat merges myfile.txt and myotherfile.txt on the output, stripping out multiple blank lines. The important thing is that cat does not distinguish between the files in the output; there are no filenames printed and no extra breaks between the two. This allows you to treat the 2 as 1 or, by adding more files to the command line, to treat 20 files as 1.

Changing Directories with cd

Changing directories is surely something that has no options, right? Not so. cd is actually more flexible than most people realize. Unlike most of the other commands here, cd is not a command in itself—it is built in to bash (or whichever shell interpreter you are using), but it is still used like a command.

The most basic usage of cd is this:

matthew@seymour:~$ cd somedir

That looks in the current directory for the somedir subdirectory and then moves you into it. You can also specify an exact location for a directory, like this:

Click here to view code image

matthew@seymour:~$ cd /home/matthew/stuff/somedir

The first part of cd’s magic lies in the characters (- and ~, a dash and a tilde). The first means “switch to my previous directory,” and the second means “switch to my home directory.” This conversation with cd shows this in action:

Click here to view code image

matthew@seymour:~$ cd /usr/local
matthew@seymour/usr/local$ cd bin
matthew@seymour/usr/local/bin$ cd -
/usr/local
matthew@seymour/usr/local$ cd ~
matthew@seymour:~$

In the first line, we change to /usr/local and get no output from the command. In the second line, we change to bin, which is a subdirectory of /usr/local. Next, cd - is used to change back to the previous directory. This time bash prints the name of the previous directory so we know where we are. Finally, cd ~ is used to change back to our /home directory, although if you want to save an extra few keystrokes, just typing cd by itself is equivalent to cd ~.

The second part of cd’s magic is its capability to look for directories in predefined locations. When you specify an absolute path to a directory (that is, one starting with a /), cd always switches to that exact location. However, if you specify a relative subdirectory—for example, cd subdir—you can tell cd where you would like that to be relative to. This is accomplished with the CDPATH environment variable. If this variable is not set, cd always uses the current directory as the base; however, you can set it to any number of other directories.

This next example shows a test of this. It starts in /home/matthew/empty, an empty directory, and the lines are numbered for later reference:

Click here to view code image

1 matthew@seymour:~/empty$ pwd
2 /home/matthew/empty
3 matthew@seymour:~/empty$ ls
4 matthew@seymour:~/empty$ mkdir local
5 matthew@seymour:~/empty$ ls
6 local
7 matthew@seymour:~/empty$ cd local
8 matthew@seymour:~/empty/local$ cd ..
9 matthew@seymour:~/empty$ export CDPATH=/usr
10 matthew@seymour:~/empty$ cd local
11 /usr/local
12 matthew@seymour:/usr/local$ cd -
13 /home/matthew/empty
14 matthew@seymour:~/empty$ export CDPATH=.:/usr
15 matthew@seymour:~/empty$ cd local
16 /home/matthew/empty/local
17 matthew@seymour:~/empty/local$

Lines 1-3 show that we are in /home/matthew/empty and that it is indeed empty; ls had no output. Lines 4-6 show the local subdirectory being made, so that /home/matthew/empty/local exists. Lines 7 and 8 show you can cd into /home/matthew/empty/local and back out again.

In line 9, CDPATH is set to /usr. This was chosen because Ubuntu has the directory /usr/local, which means our current directory (/home/matthew/empty) and our CDPATH directory (/usr) both have a local subdirectory. In line 10, while in the /home/matthew/empty directory, we use cd local. This time, bash switches us to /usr/local and even prints the new directory to ensure we know what it has done.

Lines 12 and 13 move us back to the previous directory, /home/matthew/empty. In line 14, CDPATH is set to be .:/usr. The : is the directory separator, so this means bash should look first in the current directory, ., and then in the /usr directory. In line 15 cd local is issued again, this time moving to /home/matthew/empty/local. Note that bash has still printed the new directory; it does that whenever it looks up a directory in CDPATH.

Changing File Access Permissions with chmod

Your use of chmod can be greatly extended through one simple parameter: -c. This instructs chmod to print a list of all the changes it made as part of its operation, which means we can capture the output and use it for other purposes. For example:

Click here to view code image

matthew@seymour:~$ chmod -c 600 *
mode of '1.txt' changed to 0600 (rw------)
mode of '2.txt' changed to 0600 (rw------)
mode of '3.txt' changed to 0600 (rw------)
matthew@seymour:~$ chmod -c 600 *
matthew@seymour:~$

There the chmod command is issued with -c, and you can see it has output the result of the operation: Three files were changed to rw------ (read and write by user only). However, when the command is issued again, no output is returned. This is because -c prints only the changes that it made. Files that already match the permissions you are setting are left unchanged and therefore are not printed.

There are two other parameters of interest: --reference and -R. The first allows you to specify a file to use as a template for permissions rather than specifying permissions yourself. For example, if you want all files in the current directory to have the same permissions as the file /home/matthew/myfile.txt, you use this:

Click here to view code image

chmod --reference /home/matthew/myfile.txt *

You can use -R to enable recursive operation, which means you can use it to chmod a directory and it will change the permissions of that directory as well as all files and subdirectories under that directory. You could use chmod -R 600 /home to change every file and directory under /home to become read/write to their owner(s).

Copying Files with cp

Like mv, which is covered later, cp is a command that is easily used and mastered. However, two marvelous parameters rarely see much use (which is a shame) despite their power. These are --parents and -u. The first copies the full path of the file into the new directory; the second copies only if the source file is newer than the destination.

Using --parents requires a little explanation, so here is an example. You have a file,/home/matthew/desktop/documents/work/notes.txt, and want to copy it to your /home/matthew/backup folder. You could do a normal cp, but that would give you /home/matthew/backup/notes.txt, so how would you know where that came from later? If you use --parents, the file is copied to /home/matthew/backup/desktop/ documents/work/notes.txt.

The -u parameter is perfect for synchronizing two directories because it allows you to run a command like cp -Ru myfiles myotherfiles and have cp recopy only files that have changed. The -R parameter means recursive and enables you to copy directory contents.

Printing Disk Usage with du

The du command prints the size of each file and directory that is inside the current directory. Its most basic usage is as easy as it gets:

matthew@seymour:~$ du

That outputs a long list of directories and how much space their files take up. You can modify that with the -a parameter, which instructs du to print the size of individual files as well as directories. Another useful parameter is -h, which makes du use human-readable sizes like 18M (18MB) rather than 17532 (the same number in bytes, unrounded). The final useful basic option is -c, which prints the total size of files.

So, using du we can get a printout of the size of each file in our /home directory, in human-readable format, and with a summary at the end, like this:

Click here to view code image

matthew@seymour:~$ du -ahc /home/matthew

Two advanced parameters deal with filenames you want excluded from your count. The first is --exclude, which enables you to specify a pattern that should be used to exclude files. This pattern is a standard shell file matching pattern as opposed to a regular expression, which means you can use ? to match a single character or * to match zero or many characters. You can specify multiple —exclude parameters to exclude several patterns. For example:

Click here to view code image

matthew@seymour:~$ du --exclude="*.xml" --exclude="*.xsl"

However, typing numerous --exclude parameters repeatedly is a waste of time, so you can use -X to specify a file that has the list of patterns you want excluded. The file should look like this:

*.xml
*.xsl

That is, each pattern you want excluded should be on a line by itself. If that file were called xml_exclude.txt, we could use it in place of the previous example, like this:

Click here to view code image

matthew@seymour:~$ du -X xml_exclude.txt

You can make your exclusion file as long as you need, or you can just specify multiple -X parameters.


Tip

Running du in a directory where several files are hard-linked to the same inode counts the size of the file only once. If you want to count each hard link separately for some reason, use the -l parameter (lowercase L).


Using echo

You can do many things with echo, especially with redirection (see Chapter 12, “Command-Line Master Class Part 2,” for more about redirecting output). In its simplest use, echo sends whatever you tell it to send to standard output. If you want to repeat text on the screen (useful in a shell script, for example), just enter the text string in single quotation marks (’), and the output appears below it, like this:

Click here to view code image

matthew@seymour:~$ echo 'I have the power!'
I have the power!

If you want to know the value of a system variable, such as, say, TERM, enter the variable name to ouput the value, like this:

matthew@seymour:~$ echo $TERM
xterm

You can redirect the output of echo into a text file, as when I want to add a new directory to my PATH, like this:

Click here to view code image

matthew@seymour:~$ echo 'export PATH=$PATH:/usr/local/bin' >> ~/.bashrc

You can change or set a kernel setting (1=on, 0=off) in /proc, like this:

Click here to view code image

matthew@seymour:~$ sudo sh -c 'echo "1"
>/proc/sys/location/of/setting'

Note that you can read the setting of a kernel value in /proc using cat.

Finding Files by Searching with find

The find command is one of the darkest and least understood areas of Linux, but it is also one of the most powerful. Admittedly, the find command does not help itself by using X-style parameters. The UNIX standard is -c, -s, and so on, whereas the GNU standard is --dosomething, --mooby, and so forth. X-style parameters merge the two by having words preceded by only one dash.

However, the biggest problem with find is that it has more options than most people can remember; it truly is capable of doing most things you could want. The most basic usage is this:

Click here to view code image

matthew@seymour:~$ find -name "*.txt"

That option searches the current directory and all subdirectories for files that end in .txt. The previous search finds files ending in .txt but not .TXT, .Txt, or other case variations. To search without case sensitivity, use -iname rather than -name. You can optionally specify where the search should start before the -name parameter, like this:

Click here to view code image

matthew@seymour:~$ find /home -name "*.txt"

Another useful test is -size, which you use to specify how big the files should be to match. You can specify your size in kilobytes and optionally use + or - to specify greater than or less than. For example:

Click here to view code image

matthew@seymour:~$ find /home -name "*.txt" -size 100k
matthew@seymour:~$ find /home -name "*.txt" -size +100k
matthew@seymour:~$ find /home -name "*.txt" -size -100k

The first example brings up files of exactly 100KB, the second only files larger than 100KB, and the last only files under 100KB.

Moving on, the -user option enables you to specify the user who owns the files you are looking for. So, to search for all files in /home that end with .txt, that are under 100KB, and that are owned by user matthew, you use this:

Click here to view code image

matthew@seymour:~$ find /home -name "*.txt" -size -100k -user matthew

You can flip any of the conditions by specifying -not before them. For example, you can add a -not before -user matthew to find matching files owned by everyone but matthew:

Click here to view code image

matthew@seymour:~$ find /home -name "*.txt" -size -100k -not -user matthew

You can add as many -not parameters as you need, even using -not -not to cancel each other out. (Yes, that is pointless.) Keep in mind, though, that -not -size -100k is essentially equivalent to -size +100k, with the exception that the former will match files of exactly 100KB whereas the latter will not.

You can use -perm to specify which permissions a file should have for it to be matched. This is tricky, so read carefully. The permissions are specified in the same way as with the chmod command: u for user, g for group, o for others, r for read, w for write, and x for execute. However, before you give the permissions, you need to specify a plus, a minus, or a blank space. If you specify neither a plus nor a minus, the files must exactly match the mode you give. If you specify -, the files must match all the modes you specify. If you specify +, the files must match any the modes you specify. Confused yet?

The confusion can be cleared up with some examples. This next command finds all files that have permission o=r (readable for other users). Notice that if you remove the -name parameter, it is equivalent to * because all filenames are matched:

Click here to view code image

matthew@seymour:~$ find /home -perm -o=r

Any files that have o=r set are returned from that query. Those files also might have u=rw and other permissions, but as long as they have o=r, they will match. This next query matches all files that have o=rw set:

Click here to view code image

matthew@seymour:~$ find /home -perm -o=rw

However, that query does not match files that are o=r or o=w. To be matched, a file must be readable and writeable by other users. If you want to match readable or writeable (or both), you need to use +, like this:

Click here to view code image

matthew@seymour:~$ find /home -perm +o=rw

Similarly, this next query matches only files that are readable by user, group, and others:

Click here to view code image

matthew@seymour:~$ find /home -perm -ugo=r

Whereas this query matches files as long as they are readable by the user, or by the group, or by others, or by any combination of the three:

Click here to view code image

matthew@seymour:~$ find /home -perm +ugo=r

If you use neither + or -, you are specifying the exact permissions to search for. For example, the next query searches for files that are readable by user, group, and others but not writeable or executable by anyone:

Click here to view code image

matthew@seymour:~$ find /home -perm ugo=r

You can be as specific as you need to be with the permissions. For example, this query finds all files that are readable for the user, group, and others and writeable by the user:

Click here to view code image

matthew@seymour:~$ find /home -perm ugo=r,u=w

To find files that are not readable by others, use the -not condition, like this:

Click here to view code image

matthew@seymour:~$ find /home -not -perm +o=r

Now, on to the most advanced aspect of the find command: the -exec parameter. This enables you to execute an external program each time a match is made, passing in the name of the matched file wherever you want it. This has very specific syntax: Your command and its parameters should follow immediately after -exec, terminated by \;. You can insert the filename match at any point using {} (an opening and a closing brace side by side).

So, you can match all text files on the entire system (that is, searching recursively from / rather than from /home as in our previous examples) over 10KB, owned by matthew, that are not readable by other users, and then use chmod to enable reading, like this:

Click here to view code image

matthew@seymour:~$ find / -name "*.txt" -size +10k -user matthew -not -perm +o=r -exec chmod o+r {} \;

When you type your own -exec parameters, be sure to include a space before \;. Otherwise, you might see an error such as missing argument to '-exec'.

Do you see now why some people think the find command is scary? Many people learn just enough about find to be able to use it in a very basic way, but hopefully you will see how much it can do if you give it a chance.

Searches for a String in Input with grep

The grep command, like find, is an incredibly powerful search tool in the right hands. Unlike find, though, grep processes any text, whether in files, or just in standard input.

The basic usage of grep is this:

Click here to view code image

matthew@seymour:~$ grep "some text" *

That searches all files in the current directory (but not subdirectories) for the string some text and prints matching lines along with the name of the file. To enable recursive searching in subdirectories, use the -r parameter, as follows:

Click here to view code image

matthew@seymour:~$ grep -r "some text" *

Each time a string is matched within a file, the filename and the match are printed. If a file contains multiple matches, each of the matches is printed. You can alter this behavior with the -l parameter (lowercase L), which forces grep to print the name of each file that contains at least one match, without printing the matching text. If a file contains more than one match, it is still printed only once. Alternatively, the -c parameter prints each filename that was searched and includes the number of matches at the end, even if there were no matches.

You have a lot of control when specifying the pattern to search for. You can, as we did previously, specify a simple string like some text, or you can invert that search by specifying the -v parameter. For example, the following returns all the lines of the file myfile.txt that do not contain the word hello:

Click here to view code image

matthew@seymour:~$ grep -v "hello" myfile.txt

You can also use regular expressions for your search term. For example, you can search myfile.txt for all references to cat, sat, or mat with this command:

Click here to view code image

matthew@seymour:~$ grep "[cms]at" myfile.txt

Adding the -i parameter to that removes case sensitivity, matching Cat, CAT, MaT, and so on:

Click here to view code image

matthew@seymour:~$ grep -i [cms]at myfile.txt

You can also control the output to some extent with the -n and —color parameters. The first tells grep to print the line number for each match, which is where it appears in the source file. The —color parameter tells grep to color the search terms in the output, which helps them stand out when among all the other text on the line. You choose which color you want using the GREP_COLOR environment variable: export GREP_COLOR=36 gives you cyan, and export GREP_COLOR=32 gives you lime green.

This next example uses these two parameters to number and color all matches to the previous command:

Click here to view code image

matthew@seymour:~$ grep -in —color [cms]at myfile.txt

Later you learn how important grep is for piping with other commands.

Paging Through Output with less

The less command enables you to view large amounts of text in a more convenient way than the cat command. For example, your /etc/passwd file is probably more than a screen long, so if you run cat /etc/passwd, you are not able to see what the lines at the top were. Using less /etc/passwd enables you to use the cursor keys to scroll up and down the output freely. Type q to quit and return to the shell.

On the surface, less sounds like an easy command; however, it has the infamy of being one of the few Linux commands that have a parameter for every letter of the alphabet. That is, -a does something, -b does something else, -c, -d, -e ... -x, -y, -z; they all do things, with some letters even differentiating between upper- and lowercase. Furthermore, these parameters are only used when invoking less. After you are viewing your text, even more commands are available. Make no mistake, less is a complex beast to master.

Input to less can be divided into two categories: what you type before running less and what you type while running it. The former category is easy, so we start there.

We have already discussed how many parameters less can take, but they can be distilled down to three that are very useful: -M, -N, and +. Adding -M (this is different from -m) enables verbose prompting in less. Instead of just printing a colon and a flashing cursor, less prints the filename, the line numbers being shown, the total number of lines, and the percentage of how far you are through the file. Adding -N (again, this is different from -n) enables line numbering.

The last option, +, enables you to pass a command to less for it to execute as it starts. To use this, you first need to know the commands available to you in less, which means it’s time to move to the second category of lessinput: what you type while less is running.

The basic navigation keys are the up, down, left, and right cursors; Home and End (for navigating to the start and end of a file); and Page Up and Page Down. Beyond that, the most common command is /, which initiates a text search. You type what you want to search for and press Enter to have less find the first match and highlight all subsequent matches. Type / again and press Enter to have less jump to the next match. The inverse of that is ?, which searches backward for text. Type ?, enter a search string, and press Enter to go to the first previous match of that string, or just use ? and press Enter to go to the next match preceding the current position. You can use / and ?interchangeably by searching for something with / and then using ? to go backward in the same search.

Searching with / and ? is commonly used with the + command-line parameter from earlier, which passes less a command for execution after the file has loaded. For example, you can tell less to load a file and place the cursor at the first match for the search hello, like this:

Click here to view code image

matthew@seymour:~$ less +/hello myfile.txt

Or, to place the cursor at the last match for the search hello:

Click here to view code image

matthew@seymour:~$ less +?hello myfile.txt

Beyond the cursor keys, the controls primarily involve typing a number and then pressing a key. For example, to go to line 50 you type 50g, or to go to the 75 percent point of the file you type 75p. You can also place invisible mark points through the file by pressing m and then typing a single letter. Later, while in the same less session, you can press ' (a single quote) and then type the letter, and it moves you back to the same position. You can set up to 52 marks, named a-z and A-Z.

One clever feature of less is that you can, at any time, press v to have your file open inside your text editor. This defaults to vim, but you can change that by setting the EDITOR environment variable to something else.

If you have made it this far, you can already use less better than most users. You can, at this point, justifiably skip to the next section and consider yourself proficient with less. However, if you want to be a less guru, there are two more things to learn: how to view multiple files simultaneously and how to run shell commands.

Like most other file-based commands in Linux, less can take several files as its parameters. For example:

Click here to view code image

matthew@seymour:~$ less -MN 1.txt 2.txt 3.txt

That loads all three files into less, starting at 1.txt. When viewing several files, less usually tells you which file you are in, as well as numbering them: 1.txt (file 1 of 3) should be at the bottom of the screen. However, certain things make that go away, so you should use -M anyway.

You can navigate between files by typing a colon and then pressing n to go to the next file or press p to go to the previous file; these are referred to from now on as :n and :p. You can open another file for viewing by typing :e and providing a filename. This can be any file you have permission to read, including files outside the local directory. Use Tab to complete filenames. Files you open in this way are inserted one place after your current position, so if you are viewing file 1 of 4 and open a new file, the new file is numbered 2 of 5 and is opened for viewing right away. To close a file and remove it from the list, use :d.

Viewing multiple files simultaneously has implications for searching. By default, less searches within only one file, but it is easy to search within all files. When you type / or ? to search, follow it with a *. You should see EOF-ignore followed by a search prompt. You can now type a search and it searches in the current file; if nothing is found, it looks in subsequent files until it finds a match. You can repeat searches by pressing Esc and then either n or N. The lowercase option repeats the search forward across files, and the uppercase repeats it backward.

The last thing you need to know is that you can get to a shell from less and execute commands. The simplest way to do this is just to type ! and press Enter. This launches a shell and leaves you free to type all the commands you want, as per normal. Type exit to return to less. You can also type specific commands by entering them after the exclamation mark, using the special character % for the current filename. For example, du -h % prints the size of the current file. Finally, you can use !! to repeat the previous command.

Creating Links Between Files with ln

Linux allows you to create links between files that look and work like normal files for the most part. Moreover, it allows you to make two types of links, known as hard links and symbolic links (symlinks). The difference between the two is crucial, although it might not be obvious at first.

Each filename on your system points to what is known as an inode, which is the absolute location of a file. Linux allows you to point more than one filename to a given inode, and the result is a hard link—two filenames pointing to the same file. Each of these files shares the same contents and attributes. So, if you edit one, the other changes because they are both the same file.

On the other hand, a symlink—sometimes called a soft link—is a redirect to the real file. When a program tries to read from a symlink, it automatically is redirected to what the symlink is pointing at. The fact that symlinks are really just dumb pointers has two advantages: You can link to something that does not exist (and create it later if you want), and you can link to directories.

Both types of links have their uses. Creating a hard link is a great way to back up a file on the same disk. For example, if you delete the file in one location, it still exists untouched in the other location. Symlinks are popular because they allow a file to appear to be in a different location; you could store your website in /var/www/live and an under-construction holding page in /var/www/construction. Then you could have Apache point to a symlink /var/www/html that is redirected to either the live or construction directory, depending on what you need.


Tip

The shred command overwrites a file’s contents with random data, allowing for safe deletion. Because this directly affects a file’s contents, rather than just a filename, this means that all filenames hard linked to an inode are affected.


Both types of link are created using the ln command. By default, it creates hard links, but you can create symlinks by passing it the -s parameter. The syntax is ln [-s] something somewhere, for example:

Click here to view code image

matthew@seymour:~$ ln -s myfile.txt mylink

That creates the symlink mylink that points to myfile.txt. You don’t see it here, but the file I created is 341 bytes. This is important later. Remove the -s to create a hard link. You can verify that your link has been created by running ls -l. Your symlink should look something like this:

Click here to view code image

lrwxrwxrwx 1 matthew matthew 5 Feb 19 12:39 mylink -> myfile.txt

Note how the file properties start with l (lowercase L) for link and how ls -l also prints where the link is going. Symlinks are always very small in size; the previous link is 5 bytes. If you created a hard link, it should look like this:

Click here to view code image

-rw-rw-r 2 matthew matthew 341 Feb 19 12:39 mylink

This time the file has normal attributes, but the second number is 2 rather than 1. That number is how many hard links point to this file, which is why it is 2 now. The file size is also the same as that of the previous filename because it is the file, as opposed to just being a pointer.

Symlinks are used extensively in Linux. Programs that have been superseded, such as sh, now point to their replacements (in this case bash), and library versioning is accomplished through symlinks. For example, applications that link against zlib load /usr/lib/libz.so. Internally, however, that is just a symlink that points to the actual zlib library:/usr/lib/libz.so.1.2.1.2. This enables multiple versions of libraries to be installed without application developers needing to worry about these specific details.

Finding Files from an Index with locate

When you use the find command, it searches recursively through each directory each time you request a file. This is slow, as you can imagine. Fortunately, Ubuntu ships with a cron job that creates an index of all the files on your system every night. Searching this index is extremely fast, which means that if the file you are looking for has been around since the last index, this is the preferable way of searching.

To look for a file in your index, use the command locate followed by the names of the files you want to find, like this:

Click here to view code image

matthew@seymour:~$ locate myfile.txt

On a relatively modern computer (say, with at least one processor that runs at 1.5GHz or higher), locate should be able to return all the matching files in less than one second. The trade-off for this speed is lack of flexibility. You can search for matching filenames, but, unlike find, you cannot search for sizes, owners, access permissions, or other attributes. The one thing you can change is case sensitivity; use the -i parameter to do a search that is not case sensitive.

Although Ubuntu rebuilds the filename index nightly, you can force a rebuild whenever you want by running the command updatedb with sudo. This usually takes a few minutes, but when it’s done the new database is immediately available.

Listing Files in the Current Directory with ls

The ls command, like ln, is one of those you expect to be very straightforward. It lists files, but how many options can it possibly have? In true Linux style, the answer is many, although again you need only know a few to wield great power!

The basic usage is simply ls, which lists the files and directories in the current location. You can filter that using normal wildcards, so all these are valid:

Click here to view code image

matthew@seymour:~$ ls *
matthew@seymour:~$ ls *.txt
matthew@seymour:~$ ls my*ls *.txt *.xml

Any directories that match these filters are recursed into one level. That is, if you run ls my* and you have the files myfile1.txt and myfile2.txt and a directory mystuff, the matching files are printed first. Then lsprints the contents of the mystuff directory.

The most popular parameters for customizing the output of ls are as follows:

Image -a—Includes hidden files

Image -h—Uses human-readable sizes

Image -l (lowercase L)—Enables long listing

Image -r—Reverse order

Image -R—Recursively lists directories

Image -s—Shows sizes

Image --sort—Sorts the listing

All files that start with a period are hidden in Linux, so that includes the .gnome directory in your /home directory, as well as .bash_history and the . and .. implicit directories that signify the current directory and the parent. By default, ls does not show these files, but if you run ls -a, they are shown. You can also use ls -A to show all the hidden files except . and ....

The -h parameter needs to be combined with the -s parameter, like this:

matthew@seymour:~$ ls -sh *.txt

That outputs the size of each matching file in a human-readable format, such as 108KB or 4.5MB.

Using the -l parameter enables you to get much more information about your files. Instead of just providing the names of the files, you get output like this:

Click here to view code image

drwxrwxr-x 24 matthew matthew 4096 Dec 24 21:33 arch
-rw-r--r-- 1 matthew matthew 18691 Dec 24 21:34 COPYING
-rw-r--r-- 1 matthew matthew 88167 Dec 24 21:35 CREDITS
drwxrwxr-x 2 matthew matthew 4096 Dec 24 21:35 crypto

That shows four matches and prints a lot of information about each of them. The first row shows the arch directory; you can tell it is a directory because its file attributes starts with a d. The rwxrwxr-x following that shows the access permissions, and this has special meanings because it is a directory. Read access for a directory enables users to see the directory contents, write access enables you to create files and subdirectories, and execute access enables you to cd into the directory. If a user has execute access but not read access, the user can cd into the directory but not list files.

Moving on, the next number on the line is 24, which also has a special meaning for directories: It is the number of subdirectories (including . and ..). After that is matthew matthew, which is the name of the user owner and the group owner for the directory. Next is the size and modification time, and finally the directory name itself.

The next line shows the file COPYING, and most of the numbers have the same meaning, with the exception of the 1 immediately after the access permissions. For directories, this is the number of subdirectories, but for files this is the number of hard links to this file. A 1 in this column means this is the only filename pointing to this inode, so if you delete it, it is gone.

Ubuntu comes configured with a shortcut command for ls -l: ll.

The --sort parameter enables you to reorder the output from the default alphabetic sorting. You can sort by various things, although the most popular are extension (alphabetically), size (largest first), and time (newest first). To flip the sorting (making size sort by smallest first), use the -r parameter also. So, the following command lists all .ogg files, sorted smallest to largest:

Click here to view code image

matthew@seymour:~$ ls --sort size -r *.ogg

Finally, the -R parameter recurses through subdirectories. For example, typing ls /etc lists all the files and subdirectories in /etc, but ls -R /etc lists all the files in and subdirectories in /etc, all the files and subdirectories in /etc/acpi, all the files and subdirectories in /etc/acpi/actions, and so on until every subdirectory has been listed.

Listing System Information with lsblk, lshw, lsmod, and lspci

These commands are not really related to ls, but they work in a similar way. Here, the focus is on listing information about your system rather than the contents of a directory.

To list the storage, or block, devices that are attached to your system, use:

Click here to view code image

matthew@seymour:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
Imagesda1 8:1 0 1K 0 part
Imagesda2 8:2 0 453.7G 0 part /
Imagesda5 8:5 0 12.1G 0 part [SWAP]
sdb 8:16 0 1.4T 0 disk
Imagesdb1 8:17 0 1.4T 0 part
sr0 11:0 1 1024M 0 rom

The next command must be run as root for a full listing. Note that the output may be quite long, so this command may be most useful if you pipe it into grep and search for a specific bit of text, as described in Chapter 12, “Command-Line Master Class Part 2.” To list the hardware detected in your system, use:

matthew@seymour:~$ sudo lshw

To list the status of modules in the Linux kernel, use this, which takes the contents of /proc/modules and formats it nicely:

matthew@seymour:~$ lsmod

To list the PCI devices attached to your system, use:

matthew@seymour:~$ lspci

Reading Manual Pages with man

Time for a much-needed mental break: The man command is easy to use. Most people use only the topic argument, like this: man gcc. However, two other commands work closely with man that are very useful: whatis and apropos.

The whatis command returns a one-line description of another command, which is the same text you get at the top of that command’s man page. For example:

Click here to view code image

matthew@seymour:~$ whatis ls
ls (1) - list directory contents

The output explains what ls does but also provides the man page section number for the command so you can query it further.

The apropos command takes a search string as its parameter and returns all man pages that match the search. For example, apropos mixer returns this list:

Click here to view code image

alsamixer (1) - soundcard mixer for ALSA soundcard driver
mixer (1) - command-line mixer for ALSA soundcard driver
aumix (1) - adjust audio mixer

So, use apropos to help you find commands and use whatis to tell you what a command does.

One neat trick is that many of the tips and tricks we learned for less also work when viewing man pages (for example, using / and ? to search). This is one of the beautiful things about UNIX systems: Gaining knowledge in one area often benefits you in other areas.

You can use a number of other commands to search the file system, including the following:

Image whereis command—Returns the location of the command (for example, /bin, /sbin, or /usr/bin/command) and its man page, which is an entry for the command included as a file already on your computer in a standardized manual format.

Image whatis command—Returns a one-line synopsis from the command’s man page.

Image type name—Returns how a name would be interpreted if used as a command. This generally shows options or the location of the binary that will be used. For example, type ls returns ls is aliased to 'ls -color=auto'.

Making Directories with mkdir

Making directories is as easy as it sounds, although there is one parameter you should be aware of: -p. If you are in /home/matthew and you want to create the directory /home/matthew/audio/sound, you get an error like this:

Click here to view code image

mkdir: cannot create directory 'audio/sound': No such file or directory

At first glance, this seems wrong; mkdir cannot create the directory because it does not exist. What it actually means is that it cannot create the directory sound because the directory audio does not exist. This is where the -pparameter comes in: If you try to make a directory within another directory that does not exist, like the previous, -p creates the parent directories, too. So:

Click here to view code image

matthew@seymour:~$ mkdir -p audio/sound

That first creates the audio directory and then creates the sound directory inside it.

Moving Files with mv

This command is one of the easiest around. There are two helpful parameters to mv: -f, which overwrites files without asking; and -u, which moves the source file only if it is newer than the destination file. That’s it. You can use absolute paths to indicate the destination directory (starting from /) or relative paths from the current directory (starting without a slash). This is generally entered in the source directory, but it doesn’t have to be; you can use an absolute path to indicate the source directory, too.

Click here to view code image

matthew@seymour:~$ mv filename /newdirectory/newfilename

Renaming Files with rename

We often use mv to rename a single file. This would be tedious, however, for renaming multiple files. For this, we use rename. The basic syntax is simple:

Click here to view code image

matthew@seymour:~$ rename 's/filename/newfilename/'

The part in the single quotes is a Perl expression, meaning the command is far more powerful than this example suggests. Let’s say we have a directory filled with files that end with a .htm extension, and we want to rename them all to .html:

Click here to view code image

matthew@seymour:~$ rename 's/\.htm/\.html/' *.htm

Notice here that the . characters must be preceded with a \ to let the command know they are part of the text being searched out and replaced rather than part of the command. This is called “escaping” the characters. Next, notice the replace part of the command is followed by a wildcard character (*) and the remainder of the filename to search for in the directory. Anything matching the combination of filename.htm will be renamed to filename.html.

rename is incredibly powerful. See the man page for more.

Deleting Files and Directories with rm

The rm command has only one parameter of interest: —preserve-root. By now, you should know that issuing rm -rf / with sudo destroys your Linux installation because -r means recursive and -f means force (do not prompt for confirmation before deleting). It is possible for a clumsy person to issue this command by accident—not by typing the command on purpose, but by putting a space in the wrong place. For example:

Click here to view code image

matthew@seymour:~$ rm -rf /home/matthew

That command deletes the /home directory of the user matthew. This is not an uncommon command; after you have removed a user and backed up the user’s data, you will probably want to issue something similar. However, if you add an accidental space between the / and the h in home, you get this:

Click here to view code image

matthew@seymour:~$ rm -rf / home/matthew

This time the command means “delete everything recursively from / and then delete home/matthew”—quite a different result! You can stop this from happening by using the --preserve-root parameter, which stops you from catastrophe with this message:

Click here to view code image

rm: it is dangerous to operate recursively on '/'
rm: use --no-preserve-root to override this failsafe.

However, no one wants to keep typing --preserve-root each time when running rm, so you could add this line to the .bashrc file in your /home directory:

alias rm='rm --preserve-root'

That alias automatically adds --preserve-root to all calls to rm in future bash sessions. Ubuntu (and Debian and perhaps other distributions now) have done this by default, not by adding an alias in bashrc, but by making --preserve-root the default for the command.

Sorting the Contents of a File with sort

You have a text file. You want to sort its contents alphabetically. That is easy. Let’s assume our test file is filled with one letter on each line, upper or lower case.

Click here to view code image

matthew@seymour:~$ sort testfile.txt
a
A
b
B

That is useful. You can also sort in reverse order:

Click here to view code image

matthew@seymour:~$ sort -r testfile.txt

This will output the same contents, but this time from Z to a.

You can use sort with files of numbers, but you don’t want a normal sort. That will give you output like this.

Click here to view code image

matthew@seymour:~$ sort numberfile.txt
1
14
5
58
6

The command performs a rather useless alphabetic style sorting. Fortunately, there is a switch we can use.

Click here to view code image

matthew@seymour:~$ sort -n numberfile.txt
1
5
6
14
58

There are tons of neat tricks you can do with sort and many more parameters explained in the man file. For files with a number of columns, such as a directory listing maybe, you can pick which column you use to sort by using the -k switch and the number of the column, such as with this example of a directory of old Conky files. We will start with an ls command for comparison.

Click here to view code image

matthew@seymour:~/conky$ ls -la
total 60
drwxr-xr-x 2 matt matt 4096 Dec 25 2012 .
drwxr-xr-x 91 matt matt 4096 Jul 28 18:42 ..
-rwxr-xr-x 1 matt matt 5526 Dec 25 2012 conkyrc_main
-rwxr-xr-x 1 matt matt 5502 Dec 25 2012 conkyrc_main~
-rwxr-xr-x 1 matt matt 5387 Apr 16 2008 conkyrc_main (old)
-rwxr-xr-x 1 matt matt 1326 Mar 15 2008 conkyrc_weather
-rwxr-xr-x 1 matt matt 2549 Oct 23 2009 sample_conky.conf
-rwxr-xr-x 1 matt matt 128 Apr 8 2008 start_conky (copy).sh
-rwxr-xr-x 1 matt matt 139 Dec 25 2012 start_conky.sh
-rwxr-xr-x 1 matt matt 140 Dec 25 2012 start_conky.sh~
-rwxr-xr-x 1 matt matt 1503 Sep 30 2007 weather.sh
-rwxr-xr-x 1 matt matt 2379 Sep 30 2007 weather.xslt
And here is the same listing, this time sorted by file size.


matthew@seymour:~$ ls -la | sort -n -k5
total 60
-rwxr-xr-x 1 matt matt 128 Apr 8 2008 start_conky (copy).sh
-rwxr-xr-x 1 matt matt 139 Dec 25 2012 start_conky.sh
-rwxr-xr-x 1 matt matt 140 Dec 25 2012 start_conky.sh~
-rwxr-xr-x 1 matt matt 1326 Mar 15 2008 conkyrc_weather
-rwxr-xr-x 1 matt matt 1503 Sep 30 2007 weather.sh
-rwxr-xr-x 1 matt matt 2379 Sep 30 2007 weather.xslt
-rwxr-xr-x 1 matt matt 2549 Oct 23 2009 sample_conky.conf
drwxr-xr-x 2 matt matt 4096 Dec 25 2012 .
drwxr-xr-x 91 matt matt 4096 Jul 28 18:42 ..
-rwxr-xr-x 1 matt matt 5387 Apr 16 2008 conkyrc_main (old)
-rwxr-xr-x 1 matt matt 5502 Dec 25 2012 conkyrc_main~
-rwxr-xr-x 1 matt matt 5526 Dec 25 2012 conkyrc_main

Printing the Last Lines of a File with tail

If you want to watch a log file as it is written to, or want to monitor a user’s actions as they are occurring, you need to be able to track log files as they change. In these situations, you need the tail command, which prints the last few lines of a file and updates as new lines are added. This command tells tail to print the last few lines of /var/log/apache2/access.log, the Apache hit log:

Click here to view code image

matthew@seymour:~$ tail /var/log/apache2/access.log

To get tail to remain running and update as the file changes, add the -f parameter (follow):

Click here to view code image

matthew@seymour:~$ tail -f /var/log/apache2/access.log

You can tie the lifespan of a tail follow to the existence of a process by specifying the --pid parameter. When you do this, tail continues to follow the file you asked for until it sees that the process identified by process ID (PID)is no longer running, at which point it stops tailing.

If you specify multiple files on the command line, tail follows both, printing file headers whenever the input source changes. Press Ctrl+C to terminate tail when in follow mode.

Printing the Location of a Command with which

The purpose of which is to tell you the exact command that would be executed if you typed it. For example, which mkdir returns /bin/mkdir, telling you that running the command mkdir runs /bin/mkdir.

Download Files with wget

Let’s say you see a website with useful content that you need to download to your server, say http://releases.ubuntu.com/, because you want to make a copy of the ISO of the current release of Ubuntu available to students that work in your computer lab. This is called mirroring and is a commonly accepted practice. Or, maybe you want to download the latest Minecraft server file to your server and don’t want to download it from the website to your local machine first, before you upload it to your server. You can do this with wget, which exists to download files using HTTP, HTTPS, and FTP, like this:

Click here to view code image

matthew@seymour:~$ wget http://releases.ubuntu.com/16.04/ubuntu-16.04-desktop- amd64.iso

This downloads the linked file directly to the directory in which you issue the command.

What if you wanted to copy all of the content files from your existing web server onto a new server? You can use the -m or the --mirror flag to do this. This example downloads all of the contents of the directory you specify, assuming you have access, to your current directory:

Click here to view code image

matthew@seymour:~$ wget http://youroldserver.com/website/files


Tip

You can use wget with any standard URL syntax, including specifying ports and usernames and passwords, but you should realize that the username and password information will be transmitted in plain text; therefore, this is not considered a secure method of data transfer. For that, use scp., which is covered in Chapter 19, “Remote Access with SSH, Telnet, and VNC.”


References

Image www.gnu.org/—The website of the GNU project, it contains manuals and downloads for lots of command-line software.

Image http://shop.oreilly.com/product/9780596154493.do—A wide selection of Linux commands and explanations of what they do taken from O’Reilly’s excellent book Linux in a Nutshell, which is a dictionary of Linux commands.

Image www.linuxcommand.org/—Describes itself as “your one-stop command-line shop!” It contains a wealth of useful information about the console.

Image Understanding the way UNIX works at the nuts and bolts level can be both challenging and rewarding, and there are several good books that will help guide you on your way. Perhaps the best is The Art of UNIX Programming by Eric Raymond (Addison-Wesley, ISBN: 0-13-142901-9), which focuses on the philosophy behind UNIX and manages to mix in much about the command line.