Take Control of the Mac Command Line with Terminal (2015)
Learn Advanced Techniques
Now that you know the basics of working with the command line, I want to show you a few techniques that build on your knowledge and enable you to perform more advanced tasks.
First I tell you how to Pipe and Redirect Data—two powerful (and related) techniques you can apply to many different commands in order to combine them in useful ways and do more with your data. Next, you’ll Get a Grip on grep, a tool that helps you locate files containing specified patterns of characters. Finally, I explain the basics of how you can Add Logic to Shell Scripts, making them much more useful than simple sequences of commands.
As you can imagine, these are but a few of many advanced techniques for using the command line, but I’ve found them to be consistently helpful, and I hope you will too.
Pipe and Redirect Data
Most of the time when you enter commands on the command line, the output—a list of files, the date, the contents of a log, or whatever—is shown directly on the screen. But that isn’t always what you want.
For example, maybe the output of some command is a list of hundreds or thousands of files, but that’s more information than you need; you want to filter the list to show only files that meet certain criteria. Or, maybe having that list in a Terminal window isn’t useful to you, but if it were in a BBEdit document, it would be. In cases like these, you can use either of two commands to take a command’s output and do something other than display it on the screen.
Pipe (|)
The pipe operator, which is the | symbol that you get when you type Shift-\, sends the output of a command to another program. To use it, you type the first command, then a space, the | character, another space, and the name of the second program. Like so:
program | other-program
We saw the pipe earlier, in Ps, and there are also a few instances of this in Command-Line Recipes, but let me give you some further examples to illustrate how this works and what you might do with it.
If I used the ls /Library/Preferences command to show me everything in my /Library/Preferences folder, that would be a pretty long list. But suppose I remembered that most of the items in that folder started with com.apple and I wanted to see just the last, say, ten items because that would filter out most of the Apple stuff. And then I remember that the Tail command does exactly that. Ordinarily, tail expects you to give it a file as an argument. But instead, I could give it a file listing as an argument, using the pipe operator, like so:
ls /Library/Preferences | tail
And that does what I expect—it shows just the last ten items from that directory. If I wanted to show the last 15, I could instead enter:
ls /Library/Preferences | tail -n 15
Most flags and arguments work as usual with piped commands. The exception, of course, is that commands expecting a file as an argument normally put the file after the command, but when you use a pipe, the order is reversed.
How about another example? If I used the locate command to find all the files containing Apple in the name—again, an awkwardly large number—they’d all scroll by at a dizzying speed. If instead I wanted to be able to page through them one screenful at a time—hey, just like you can do with less (see View a Text File)—I can just pipe the output of locate into less, like so:
locate Apple | less
Or perhaps I’d like to get the path of the current directory and put it on my Mac OS X clipboard. With a pipe and the pbcopy (pasteboard copy) command, it’s easy:
pwd | pbcopy
The same idea works for other commands. Need to copy a list of every GIF image in a directory? Entering ls *.gif | pbcopy will do the trick.
These examples are all fairly simple, but the concept can be extended in all kinds of ways. If a command can accept a file as an argument, it can probably be used on the right side of a pipe.
And, in case you were wondering, yes, you can chain pipes! That is, send the output of one program to a second, and the output of the second to a third (and so on). So, if I want my clipboard to contain a list of the last ten files in my /Library/Preferences directory (without displaying them on the screen), I could combine a couple of earlier example like so:
ls /Library/Preferences | tail | pbcopy
This is a technique that rewards experimentation, so see what other interesting combinations you can come up with.
Redirect (>)
Whereas the pipe sends the output of a program to another program, the redirect (>) operator sends the output of a program to a file (without displaying it on screen). For example, maybe I want to put the list of all the files in /Library/Preferences into a text file to study later. I could do it like this:
ls /Library/Preferences > ~/Desktop/prefs.txt
That creates a file on my Desktop called prefs.txt which contains the output of the previous command, which then lists everything in that directory.
You can use redirect with nearly any (non-interactive) program that displays its output on screen. But be careful with commands that produce continuous output; the file will keep growing indefinitely. For example, you wouldn’t want to use top > file.txt because the top command produces a dynamic output. Instead, you might try ps -ax > file.txt for a static snapshot of all running processes.
At this point, perhaps your gears are turning and you want to know whether you can combine piping and redirecting in a single line. Why, yes, you can! If I wanted to put the last ten items from a directory into a text file in my home folder, I could do it like this:
ls /Library/Preferences | tail > ~/files.txt
Get a Grip on grep
Depending on who you ask, the command grep stands for either “globally search a regular expression and print” or “global regular expression parser.” In any case, grep is a pattern-matching tool that can make use of a sequence of characters known as a regular expression (sometimes abbreviated to regex or regexp) in order to locate files by their content. If you know what you’re looking for inside a file but not the file’s name or location, this is the command you want.
We’ll get back to regular expressions in a moment. First, let’s look at a very basic use of grep that uses a plain text search string.
Earlier, in Find a File, I showed how to use the find command to find a file by name. It’s also possible to find a file with find based on the file’s content, but an easier way is to use the grep command. Enter the following, replacing your text with what you want to find:
grep -R "your text" .
For example, to find all files within the current directory and its subdirectories whose contents (not necessarily filenames) include the word Apple, I’d use:
grep -R "Apple" .
The -R flag means “recursive”—that is, look in all the subdirectories. Also notice the period (.) at the end. That signifies “this directory.” So the combination of -R and the period mean “search recursively from this directory down.” To search just the directory you’re in, you can leave out -R, but then you’ll also need to replace the period with an asterisk (*), to mean “any file”—without that, grep will give an error because you’ve told it to search a directory, but it searches only files.
If I wanted to search recursively from the parent directory of the one I’m in, I’d do this:
grep -R "Apple" ..
Those are the same two dots (..) we used with cd (see Move Up or Down). And if I wanted to search a specific directory, I’d fill in its path:
grep -R "Apple" /Library/Preferences
I suggest resisting the temptation to put / (a whole disk) as the search target, because the search would be enormously time-consuming.
Note: By default, grep finds partial-word matches; the string "bar" matches both baroque and lumbar.
That’s the simple way to use grep, and it’s pretty useful. But what if you’re not looking for a specific string, like Apple, but rather a pattern, such as a phone number, a URL, or any line that starts with the word butter? That’s where regular expressions come in.
A regular expression is basically a pattern of regular characters and metacharacters (such as wildcards, parentheses, and other special symbols that tell grep to look for particular characters or patterns of characters). With practice, you should be able to create a regular expression that represents almost any text you can describe in words. Here are some simple metacharacters to get you started:
· Any character: . (period)
· One or more times: +
· Anything in a particular set of characters: [ ] (for example, [abcde] for any of the letters a, b, c, d, or e; or [1-5] for any digit 1 through 5)
· Start of a line: ^
So, putting various combinations of these together, we can look for:
· Any seven-digit phone number:
[0-9][0-9][0-9]\-[0-9][0-9][0-9][0-9]
Bracketed sets of characters can include ranges, like [0-9], [A-Z], or [a-z]. Because some characters, like -, have special meanings in regex, you put a backslash (\) before them to indicate that you’re looking for the literal character here.
· Any number of digits followed by a hyphen and any number of additional digits:
[0-9]+\-[0-9]+
· Any instance of the word butter at the beginning of a line:
^butter
· Any number with three or more digits at the beginning of a line:
^[0-9][0-9][0-9]+
· Any line that starts with a digit or an uppercase letter:
^[0-9A-Z].+
You can combine ranges of characters in a set. And .+ means “any character, one or more times.”
That’s just the beginning. There are metacharacters to represent all kinds of things. A few more examples:
· Anything not in this set of characters: [^ ]
· A space: \s
· A tab: \t
· End of a line: $
· A return character: \n
You can also group elements in parentheses ( ), use the pipe | character to indicate “or,” and much more. (There are many different versions of regular expressions, with variation in which metacharacters they support.)
Now let’s go back to searching for files by content, because that’s what kicked off this topic. Let’s say I’m looking for any file that contains the word Apple as the next-to-last word of a line. I start with the regular expression:
Apple\s[A-Za-z]+$
That reads “the word Apple, followed by a space, followed by any string of one or more uppercase or lowercase letters, at the end of a line.” Now I feed that to grep, like this:
grep -RIE "Apple\s[A-Za-z]+$" .
Notice the two new flags: -E means “treat this as a regular expression, not plain text,” and -I means “ignore binary files” (since I know I’m searching only for matching text files, this makes the command run much faster by skipping things like image, audio, and video files, but also ignores things like Microsoft Word documents and PDFs).
Needless to say, you can also combine grep with other commands using piping and redirecting (as discussed in the previous topic). For example, to list all the files in a directory but show only those containing the word Apple, you might try:
ls /Library/Preferences | grep -ERI "Apple"
All this is still the tip of the iceberg. Regular expressions are useful not just in grep but in Perl scripts and in Mac OS X apps such as Nisus Writer Pro and BBEdit. And grep can do far more than what I’ve described here.
Note: To learn more about grep specifically, read Kirk McElhearn’s Macworld article Find anything with grep, and to learn more about regular expressions more generally, read Jason Snell’s article Transform HTML with Regular Expressions.
Add Logic to Shell Scripts
When I showed you how to Create Your Own Shell Script, the examples I gave were simple sequences of commands: do this, then this, then this; and you’re done. But sometimes you’ll need scripts to be more flexible. They might need to accept input, make decisions, perform calculations, and employ other sorts of logic.
If you’ve done any type of programming or scripting, you’ve certainly encountered concepts like variables, conditionals, and loops. You can use all these things in bash, too, although you’ll need to learn bash’s idiosyncratic way of dealing with them. Alternatively, if these concepts are brand new to you, shell scripting is one of the easiest ways to learn by experimentation.
My intention here is not to teach you programming or provide extensive tutorials, but only to provide a few simple examples to get you started, along with some pointers to places where you can learn more.
Variables and Input
In bash, variables are about as simple as they get in any programming language. You can pick almost any word you like to serve as a variable, and you give it a value by typing = and a number or string (any sequence of characters). For example, if I want a variable called city, I can create it and give it the value 12345 like so:
city=12345
Or, if I want it to have the value New York, I do it this way:
city="New York"
I put New York in quotation marks because it has a space in it. If the string didn’t have a space, I could have left out the quotation marks, but using them with strings is a good habit to get into, because multi-word strings are pretty common. Other than that detail, you don’t need to do anything special to tell bash whether a variable is an integer or a string.
Note: You’ll notice that there are no spaces around the = sign. This is crucial: if you used spaces (as in city = 12345), bash would mistakenly think that the variable name is the name of a command, and the script wouldn’t work.
Later on, if I want to do something with my variable, such as display it on the screen, use it in a computation, or compare it to another value, I put a dollar sign ($) in front of it. For example, this (rather pointless) script assigns a value to a variable and then displays it:
#!/bin/bash
city="New York"
echo $city
I’d like to show you three additional tricks with variables, two of which involve getting input from the user.
Turn a Command Line Argument into a Variable
We’ve seen a lot of commands that take arguments. For example, the command nano file.txt opens the file file.txt in the nano editor, and ls /Library lists the contents of the /Library directory. You can do the same thing with your own scripts: add one or more arguments after the script’s name to provide more information to the script about what you want it to do. Best of all, it requires almost no effort.
When you enter a script name followed by a space and one or more terms, each term is automatically assigned to variables called $1, $2, $3, and so on in the order the terms were typed. For example, suppose we created this script and named it test.sh:
#!/bin/bash
echo "The first three arguments you entered were $1, $2, and $3."
Now run the script like so:
./test.sh Alice Bob Carol
The output will be:
The first three arguments you entered were Alice, Bob, and Carol.
If you entered more than three arguments, the rest will be ignored (although you could add $4 to the script easily enough), and if you entered fewer, the response would have some blanks, as in:
The first three arguments you entered were Alice, , and .
(And yes, you could add logic to the script to eliminate those blanks, but I’m trying to keep things simple for now.)
Note that anything you can type on the command line can be a variable, including pathnames and filenames.
Tip: If your script needs to know its own name for any reason, that’s also stored in a variable automatically: $0.
Get Interactive User Input
You can also have the script ask you a question while it’s running and turn your response into a variable. You do that with the command read followed by the name of the variable you want the response to be stored in:
#!/bin/bash
echo "What do you have to say for yourself?"
read reply
echo "Oh yeah? Well, $reply to you too!"
A script can carry on an extensive conversation with the user, if need be, and each response can influence what happens later in the script.
Put the Output of a Command into a Variable
The last variable trick I want to mention is useful when your script needs to run a command and then do something with that command’s output. For example, if you use the date command to find the date, you may want to put the date in a variable so that you can later use it as part of a filename. Or if your script uses the pwd command to find the path of the current directory, you might want to use that information later on when saving a file.
To do this, surround the command in question (including any flags or arguments) in parentheses, with a dollar sign $ before them, as in:
today=$(date)
or
directory=$(pwd)
Then, later on you could refer to $today or $directory, respectively, to retrieve the contents of those variables.
Flow Control
Scripts frequently make decisions based on user input or information they encounter as they run. For example, let’s say you have a script that renames the files in a directory, but you want to rename them one way if they’re text files, a different way if they’re PDFs, and a third way if they’re PNG graphics. Or suppose you want to ask the user for a number and perform one action if the number is less than or equal to 5, but a different action if the number is higher.
In cases like these, you need to use conditional statements like if, then, and else. These are sometimes called flow control statements, because they determine the path the script takes.
The bash shell has a weird way of structuring if/then statements. Here’s the basic structure:
if [ condition to test ]
then
action to take
fi
I want to point out a few key items here:
· The condition in the first line (a mathematical or logical test that yields a true/false result) must be surrounded by spaces inside the brackets. (Remember that bash forbids spaces around = in variable assignments; here, they’re mandatory.)
· After the if line containing the condition, you need the word then—either on a line by itself (as above), or on the same line after a semicolon (as I’ll show in the next example).
· By convention, most people indent the command(s) that follow then by a few spaces or a tab, but that’s just to make your script easier to read. You can leave them out if you prefer.
· Every if statement must end with fi (that’s if backward), which is equivalent to end or endif in other languages.
Here’s a complete script that shows how if works:
#!/bin/bash
echo "Pick a number."
read reply
if [ $reply -le 5 ]; then
echo "$reply is less than or equal to 5"
fi
(We’ll get to that funny -le thing in a minute.)
But wait… what if the number is greater than 5? Then you need to expand the if statement to include else (what to do if the condition presented is false).
You do it like so:
#!/bin/bash
echo "Pick a number."
read reply
if [ $reply -le 5 ]; then
echo "$reply is less than or equal to 5"
else
echo "$reply is greater than 5"
fi
You can check for two or more conditions, too. For example, do one thing if the number is less than 5, a second thing if the number is exactly 5, and a third thing if the number is greater than 5.
To do this, you’ll add elif (else if), along with another then, like this:
#!/bin/bash
echo "Pick a number."
read reply
if [ $reply -lt 5 ]; then
echo "$reply is less than 5"
elif [ $reply -eq 5 ]; then
echo "$reply is exactly 5"
else
echo "$reply is 5 or greater"
fi
Well, what about that funny -le in the first example, or the -lt in the last one? Those mean less than or equal and less than, respectively. Wacky, I know, but bash doesn’t use symbols like ≤ or <=, relying instead on abbreviations for the most part. Here’s a longer list of operators you might need to know:
· Is greater than: -gt
· Is less than: -lt
· Is equal to (for integers): -eq
· Is equal to (for strings): ==
· Is not equal to: !=
· Is greater than or equal to: -ge
· Is less than or equal to: -le
· Contains a string (not integer or empty): -n
· Is empty: -z
· Logical AND: &&
· Logical OR: ||
· Logical NOT: !
Be especially careful with those “is equal to” operators, because if you use the wrong one for the type of thing you’re comparing, you’ll get the wrong results (or an error message). For example, if $this is a number, you might have if [ $this -eq 5 ], but if $this is a string, you would need to use if [ $this == "Joe" ].
Loops
If you need to perform an operation on every file in a directory, every line in a file, or every whatever of a something, you need a loop. As in most programming languages, bash offers several loop varieties. Here’s how they look.
While Loops
If you need to repeat an action as long as some condition is true (or while it’s true) but then stop when it becomes false, you want a while loop. The structure is as follows:
while [ condition to test ]
do
stuff to do
done
For example, this while loop displays the numbers from 1 to 10:
#!/bin/bash
count=1
while [ $count -le 10 ]
do
echo "$count"
((count++))
done
Note: Like if/then statements, the do can go on its own line, or on the same line as while, separated with a semicolon.
We start by saying that the $count variable is 1, and each time through the loop we display its current value and then add 1. That’s what the ((count++)) line does—the double parentheses mean “this is a mathematical operation” and the ++ means “add 1.”
For Loops
A for loop starts with a list, series, or range of items (numbers, files, etc.) and performs one or more actions once for each of those items. Its basic structure is:
for variable in list
do
stuff to do
done
As an example, here’s a simple script that displays five consecutive messages, each with the number of the current iteration:
#!/bin/bash
for i in 1 2 3 4 5
do
echo "This is iteration number $i"
done
You can also represent a range using brackets, as in {1..5} (notice that there are just two periods in between the numbers, not three). And the items don’t have to be numbers—they can be anything. For example:
#!/bin/bash
for i in Red Orange Yellow Green Blue
do
echo "$i is a lovely color."
done
If an item in a range includes a space, you must escape the space by putting a backslash before it:
for i in New\ York Seattle San\ Francisco
Math
When it comes to math, bash is at about first-grade level. It can add, subtract, multiply, divide, and compare integers (whole numbers)… and that’s about it. You can use external calculators (such as bc) in your scripts to perform more advanced calculations, but bash itself keeps it basic.
As we saw in While Loops, you can tell bash that you want it to calculate something by surrounding it with double parentheses:
((7*5+3))
But if you want to do anything with that result, such as assign it to a variable, you’ll need to add a $ to the beginning, which in bash is known as arithmetic expansion:
number=$((7*5+3))
A different way to achieve the same result is to use the let command, which also requires quotation marks around the entire expression, including the variable, like this:
let "number=7*5+3"
Note: Although bash sometimes requires spaces and sometimes forbids them, they’re optional in mathematical expressions. So, number=$((7*5+3)) and number=$(( 7 * 5 + 3 )) both work.
Learn More about Shell Scripting
You can find oodles of sites on the Web dedicated to teaching bash—from beginner to advanced levels. Here are some examples:
· Apple’s Shell Scripting Primer
· A quick guide to writing scripts using the bash shell by Donovan Rebbechi
· Bash Guide for Beginners by Machtelt Garrels
· Bash Scripting Tutorial at LinuxConfig
· Bash Tutorial (PDF) by Erik Hjelmås
Using Terminal in Recovery Mode
If your Mac has disk problems, a damaged copy of OS X, or other issues that keep it from booting properly, you might use Recovery mode to run Disk Utility or perform other maintenance. While in Recovery mode, you can also use Terminal, which can come in handy for running certain commands—for instance, finding files on your Mac and copying them onto a flash drive.
In particular, you’ll need Terminal to reset a forgotten administrator password. Here’s how you do it:
1. Restart your Mac and immediately hold down Command-R. When the gray Apple logo appears, you can release the keys. In a moment or two, Recovery mode’s OS X Utilities window appears.
2. Choose Tools > Terminal. A Terminal window opens.
3. Type resetpassword and press Return. You may need to wait a moment or two, but a new window called Reset Password opens. If it’s behind the Terminal window, click it to bring it to the front.
4. Select your startup volume. From the Select the User Account pop-up menu, choose your username.
5. Enter and confirm a new password. Click Save, and then click OK to confirm the password reset.
6. Choose Reset Password > Quit Reset Password; then choose Terminal > Quit Terminal. Finally choose OS X Utilities > Quit OS X Utilities and click Restart.
One point to be aware of is that in Recovery mode, the bash shell offers only a subset of its regular commands. To see what commands are available, enter ls /bin /sbin /usr/bin /usr/sbin.