Pro Bash Programming: Scripting the GNU/Linux Shell, Second Edition (2015)

CHAPTER 12. Runtime Configuration

When I download my e-mail from three or four different POP3 servers, I don’t use a different script for each one. When I open a terminal to ssh to a remote computer (half a dozen of them) with a different background color for each, I use the same script for every connection. To upload files to my web sites (I look after six sites), I use the same script for all of them.

You can configure a script’s behavior in several ways when you run it. This chapter looks at seven methods: initialized variables, command-line options and arguments, menus, Q&A dialogue, configuration files, multiple names for one script, and environment variables. These methods are not mutually exclusive; in fact, they are often combined. A command-line option could tell the script to use a different configuration file or present the user with a menu.

Defining Variables

If the runtime requirements for a script will rarely change, hard-coded variables may be all the configuration you need (Listing 12-1). You can set them when the script is installed. When a change is needed, the parameters can quickly be changed with a text editor.

Listing 12-1. Example of Initialized Default Variables

## File locations
dict=/usr/share/dict
wordfile=$dict/singlewords
compoundfile=$dict/Compounds

## Default is not to show compound words
compounds=no

If the variables need changing often, one or more of the other methods can be added.

Command-Line Options and Arguments

The most common method for changing runtime behavior uses command-line options. As shown in Listing 12-2, all the values defined earlier can be modified at the command line.

Listing 12-2. Parse Command-Line Options

while getopts d:w:f:c var
do
case "$var" in
c) compounds=1 ;;
d) dict=$OPTARG ;;
w) wordfile=$OPTARG ;;
f) compoundfile=$OPTARG ;;
esac
done

Menus

For a user unfamiliar with a piece of software, a menu is a good way to allow runtime changes. In the menu example shown in Listing 12-3, the selections are numbered from 1 to 4, and q exits the menu.

Listing 12-3. Set Parameters via Menu

while : ## loop until user presses 'q'
do
## print menu
printf "\n\n%s\n" "$bar"
printf " Dictionary parameters\n"
printf "%s\n\n" "$bar"
printf " 1. Directory containing dictionary: %s\n" "$dict"
printf " 2. File containing word list: %s\n" "$wordfile"
printf " 3. File containing compound words and phrases: %s\n" "$compoundfile"
printf " 4. Include compound words and phrases in results? %s\n" "$compounds"
printf " q. %s\n" "Exit menu"
printf "\n%s\n\n" "$bar"

## get user response
read -sn1 -p "Select (1,2,3,4,q): " input
echo

## interpret user response
case $input in
1) read -ep "Enter dictionary directory: " dict ;;
2) read -ep "Enter word-list file: " wordfile ;;
3) read -ep "Enter compound-word file: " compoundfile ;;
4) [ "$compounds" = y ] && compounds=n || compounds=y ;;
q) break ;;
*) printf "\n\aInvalid selection: %c\n" "$input" >&2
sleep 2
;;
esac
done

Q&A Dialogue

A question-and-answer function cycles through all the parameters, prompting the user to enter a value for each one (Listing 12-4). This can get tedious for the user, and it is probably best used when there are no defaults, when there are very few parameters to enter, or when values need to be entered for a new configuration file.

Listing 12-4. Set Variables by Question and Answer

read -ep "Directory containing dictionary: " dict
read -ep "File containing word list: " wordfile
read -ep "File containing compound words and phrases: " compoundfile
read -sn1 -p "Include compound words and phrases in results (y/n)? " compounds
echo
read -ep "Save parameters (y/n)? " save
case $save in
y|Y) read -ep "Enter path to configuration file: " configfile
{
printf '%-30s ## %s"\n' \
"dict=$dict" "Directory containing dictionary" \
"wordfile=$wordfile" "File containing word list" \
"compoundfile=$compoundfile" "File containing compound words and phrases" \
"Compounds" "$Compounds" "Include compound words and phrases in results?"
} > "${configfile:-/dev/tty}"
esac

Configuration Files

Configuration files can use any format, but it’s easiest to make them shell scripts that can be sourced. The example file shown in Listing 12-5 can be sourced, but it can also provide more information.

Listing 12-5. Configuration File, words.cfg

dict=/usr/share/dict ## directory containing dictionary files
wordfile=singlewords ## file containing word list
compoundfile=Compounds ## file containing compound words and phrases
compounds=no ## include compound words and phrases in results?

The words.cfg file can be sourced with either of these two commands:

. words.cfg
source words.cfg

Rather than sourcing the file, it can be parsed in various ways (Listing 12-6). In bash-4.x, you can read the file into an array and extract the variables and comments using parameter expansion, the expansion being applied to each element of the array.

Listing 12-6. Parsing Configuration File

IFS=$'\n'
file=words.cfg
settings=( $( < "$file") ) ## store file in array, 1 line per element
eval "${settings[@]%%#*}" ## extract and execute the assignments
comments=( "${settings[@]#*## }" ) ## store comments in array

The comments array contains just the comments, and the assignments can be extracted from settings with "${settings[@]%%#*}":

$ printf "%s\n" "${comments[@]}"
directory containing dictionary files
file containing word list
file containing compound words and phrases
include compound words and phrases in results?

You can also read the file in a loop to set the variables and provide information about the variables it contains by displaying the comments (Listing 12-7).

Listing 12-7. Parsing Configuration File with Comments

while read assignment x comment
do
if [ -n "$assignment" ]
then
printf "%20s: %s\n" "${assignment#*=}" "$comment"
eval "$assignment"
fi
done < "$file"

The following is the result:

/usr/share/dict: directory containing dictionary files
singlewords: file containing word list
Compounds: file containing compound words and phrases
n: include compound words and phrases in results?

Configuration files can be made as complex as you like, but parsing them then falls more properly under the category of data processing, which is the subject of Chapter 13.

Scripts with Several Names

By storing the same file under different names, you can avoid command-line options and menus. The script in Listing 12-8 opens a terminal and connects to different remote computers using a secure shell. The terminal’s colors, the mac to log on to, and the name of the remote user are all determined by the name of the script.

Listing 12-8. bashful, Connect to Remote Computer via ssh

scriptname=${0##*/}

## default colours
bg=#ffffcc ## default background: pale yellow
fg=#000000 ## default foreground: black

user=bashful ## default user name
term=xterm ## default terminal emulator (I prefer rxvt)

case $scriptname in
sleepy)
bg=#ffffff
user=sleepy
host=sleepy.example.com
;;
sneezy)
fg=#aa0000
bg=#ffeeee
host=sneezy.example.org
;;
grumpy)
fg=#006600
bg=#eeffee
term=rxvt
host=cfajohnson.example.com
;;
dopey)
host=127.0.0.1
;;
*) echo "$scriptname: Unknown name" >&2
exit 1
;;
esac

"$term" -fg "$fg" -bg "$bg" -e ssh -l "$user" "$host"

To create the multiple names for the same file, create links with ln (Listing 12-9).

Listing 12-9. Make Multiple Links to bashful Script

cd "$HOME/bin" &&
for name in sleepy sneezy grumpy dopey
do
ln -s bashful "$name" ## you can leave out the -s option if you like
done

Environment Variables

You can also pass settings to a program using variables. These can be either exported or defined on the same line as the command. In the latter case, the variable is defined for that command only.

You alter the behavior of the program by checking for the value of a variable or even just for its existence. I use this technique most often to adjust the output of a script using verbose. This would be a typical line in a script:

[ ${verbose:-0} -gt 0 ] && printf "%s\n" "Finished parsing options"

The script would be called with the following:

verbose=1 myscriptname

You can see an example in the following script below.

All Together Now

The following is the program I use to update all my web sites. It finds new or modified files in a directory hierarchy, stores them in a tarball, and uploads them to a web site on a (usually) remote computer. I have shell access on all the sites I use, so I can use a secure shell, ssh, to transfer the files and unpack them with tar on the site:

ssh -p "$port" -l "$user" "$host" \
"cd \"$dest\" || exit;tar -xpzf -" < "$tarfile" &&
touch "$syncfile"

All of my sites use authentication keys (created with ssh-keygen) so that no password is required and the script can be run as a cron job.

This program uses all the techniques mentioned earlier except for multiple names. It’s more than you would usually use in a single program, but it’s a good illustration.

The user can select whether to use command-line options, a menu, a Q&A dialogue, or a configuration file to adjust the settings, or the user can even use the defaults. Command-line options are available for all settings:

· -c configfile: Reads settings from configfile

· -h host: Specifies the URL or IP address of remote computer

· -p port: Specifies the SSH port to use

· -d dest: Specifies the destination directory on the remote host

· -u user: Specifies the user’s login name on remote computer

· -a archivedir: Specifies the local directory to store archive files

· -f syncfile: Specifies the file whose timestamp is the cutoff point

· And there are three further options that control the script itself:

· -t: Tests only, displays final settings, does not archive or upload

· -m: Presents user with the menu

· -q: Uses Q&A dialogue

The script is examined in the following sections in detail, section by section.

Note This is a book on Pro Bash Scripts and hence the approach using scripting. Writing a script may not necessarily be the best solution.

There are a couple of other options not necessarily Bash scripting based that are created solely to achieve administration outcomes. There is a perl script wrapper called Cluster SSH (open source) that allows you to send a command to multiple servers at the same time and is GUI based. There is another called Puppet, which is quite popular.

Script Information

Note that parameter expansion is used to pull the script name from $0, not the external command, basename (Listing 12-10).

Listing 12-10. upload, Archive and Upload Files to Remote Computer

scriptname=${0##*/}
description="Archive new or modified files and upload to web site"
author="Chris F.A. Johnson"
version=1.0

Default Configuration

Besides setting the variables, an array containing the names of the variables and their descriptions are created (Listing 12-11). This is used by the menu and qa (question and answer) functions for labels and prompts.

Listing 12-11. Default Values and settings Array

## archive and upload settings
host=127.0.0.1 ## Remote host (URL or IP address)
port=22 ## SSH port
dest=work/upload ## Destination directory
user=jayant ## Login name on remote system
source=$HOME/public_html/oz-apps.com ## Local directory to upload
archivedir=$HOME/work/webarchives ## Directory to store archive files
syncfile=.sync ## File to touch with time of last upload

## array containing variables and their descriptions
varinfo=( "" ## Empty element to emulate 1-based array
"host:Remote host (URL or IP address)"
"port:SSH port"
"dest:Destination directory"
"user:Login name on remote system"
"source:Local directory to upload"
"archivedir:Directory to store archive files"
"syncfile:File to touch with time of last upload"
)

## These may be changed by command-line options
menu=0 ## do not print a menu
qa=0 ## do not use question and answer
test=0 ## 0 = upload for real; 1 = don't archive/upload, show settings
configfile= ## if defined, the file will be sourced
configdir=$HOME/.config ## default location for configuration files
sleepytime=2 ## delay in seconds after printing messages

## Bar to print across top and bottom of menu (and possibly elsewhere)
bar=================================================================
bar=$bar$bar$bar$bar ## make long enough for any terminal window
menuwidth=${COLUMNS:-80}

Screen Variables

These variables use the ISO-6429 standard, which is now all but universal in terminals and terminal emulators (Listing 12-12). This is discussed in detail in Chapter 14. When printed to the terminal, these escape sequences perform the actions indicated in the comments.

Listing 12-12. Define Screen Manipulation Variables

topleft='\e[0;0H' ## Move cursor to top left corner of screen
clearEOS='\e[J' ## Clear from cursor position to end of screen
clearEOL='\e[K' ## Clear from cursor position to end of line

Function Definitions

There are five functions, two of which, menu and qa, allow the user to change the settings. With readline able to accept the user’s input, the -i option to read is used if the shell version is bash-4.x or greater. If the test option is used, the print_config function outputs the settings in a format that is suitable for a configuration file, complete with comments.

Function: die

The program exits via the die function when a command fails (Listing 12-13).

Listing 12-13. Define die Function

die() #@ Print error message and exit with error code
{ #@ USAGE: die [errno [message]]

error=${1:-1} ## exits with 1 if error number not given
shift
[ -n "$*" ] &&
printf "%s%s: %s\n" "$scriptname" ${version:+" ($version)"} "$*" >&2
exit "$error"
}

Function: menu

The menu function uses its command-line arguments to populate the menu (Listing 12-14). Each argument contains a variable name and a description of the variable separated by a colon.

THE UPLOAD SETTINGS MENU

================================================================================

UPLOAD SETTINGS

================================================================================

1: Remote host (URL or IP address) (127.0.0.1)
2: ssh port (22)
3: Destination directory (work/upload)
4: Login name on remote system (jayant)
5: Local directory to upload (/home/jayant/public_html/oz-apps.com)
6: Directory to store archive files (/home/jayant/work/webarchives)
7: File to touch with time of last upload (.sync)
q: Quit menu, start uploading
0: Exit upload

================================================================================

Select 1..7 or 'q/0'

The function enters an infinite loop, from which the user exits by selecting q or 0. Within the loop, menu clears the screen and then cycles through each argument, storing it in item. It extracts the variable name and description using parameter expansion:

var=${item%%:*}
description=${item#*:}

The value of each var is obtained through indirect expansion, ${!var}, and is included in the menu labels. The field width for the menu number is ${#max}, that is, the length of the highest item number.

Listing 12-14. Define menu Function

menu() #@ Print menu, and change settings according to user input
{
local max=$#
local menutitle="UPLOAD SETTINGS"
local readopt

if [ $max -lt 10 ]
then ## if fewer than ten items,
readopt=-sn1 ## allow single key entry
else
readopt=
fi

printf "$topleft$clearEOS" ## Move to top left and clear screen

while : ## infinite loop
do

#########################################################
## display menu
##
printf "$topleft" ## Move cursor to top left corner of screen

## print menu title between horizontal bars the width of the screen
printf "\n%s\n" "${bar:0:$menuwidth}"
printf " %s\n" "$menutitle"
printf "%s\n\n" "${bar:0:$menuwidth}"

menunum=1

## loop through the positional parameters
for item
do
var=${item%%:*} ## variable name
description=${item#*:} ## variable description

## print item number, description and value
printf " %${#max}d: %s (%s)$clearEOL\n" \
"$menunum" "$description" "${!var}"

menunum=$(( $menunum + 1 ))
done

## … and menu adds its own items
printf " %${##}s\n" "q: Quit menu, start uploading" \
"0: Exit $scriptname"

printf "\n${bar:0:$menuwidth}\n" ## closing bar

printf "$clearEOS\n" ## Clear to end of screen
##
#########################################################

#########################################################
## User selection and parameter input
##

read -p " Select 1..$max or 'q' " $readopt x
echo

[ "$x" = q ] && break ## User selected Quit
[ "$x" = 0 ] && exit ## User selected Exit

case $x in
*[!0-9]* | "")
## contains non digit or is empty
printf "\a %s - Invalid entry\n" "$x" >&2
sleep "$sleepytime"
;;
*) if [ $x -gt $max ]
then
printf "\a %s - Invalid entry\n" "$x" >&2
sleep "$sleepytime"
continue
fi

var=${!x%%:*}
description=${!x#*:}

## prompt user for new value
printf " %s$clearEOL\n" "$description"
readline value " >> " "${!var}"

## if user did not enter anything, keep old value
if [ -n "$value" ]
then
eval "$var=\$value"
else
printf "\a Not changed\n" >&2
sleep "$sleepytime"
fi
;;
esac
##
#########################################################

done
}

Function: qa

The qa function takes the same arguments as menu, but instead of putting them into a menu, it prompts the user for a new value for each variable (Listing 12-15). When it has run through all the command-line arguments, which it splits up in the same manner as menu, it calls the menufunction for verification and editing of the values. Also like menu, it uses readline to get the input and keeps the old value if nothing is entered.

Listing 12-15. Define qa Function

qa() #@ Question and answer dialog for variable entry
{
local item var description

printf "\n %s - %s\n" "$scriptname" "$description"
printf " by %s, copyright %d\n" "$author" "$copyright"
echo
if [ ${BASH_VERSINFO[0]} -ge 4 ]
then
printf " %s\n" "You may edit existing value using the arrow keys."
else
printf " %s\n" "Press the up arrow to bring existing value" \
"to the cursor for editing with the arrow keys"
fi
echo

for item
do
## split $item into variable name and description
var=${item%%:*}
description=${item#*:}
printf "\n %s\n" "$description"
readline value " >> " "${!var}"
[ -n "$value" ] && eval "$var=\$value"
done

menu "$@"
}

The dialogue looks like this:

$ upload -qt

upload - Archive new or modified files and upload to web site
by Chris F.A. Johnson, copyright 2009

You may edit existing value using the arrow keys.

Remote host (URL or IP address)
>> oz-apps.com

SSH port
>> 99

Destination directory
>> public_html

Login name on remote system
>> jayant

Local directory to upload
>> /home/jayant/public_html/oz-apps.com

Directory to store archive files
>> /home/jayant/work/webarchives

File to touch with time of last upload
>> .sync

Function: print_config

The print_config function prints all the variables listed in the varinfo array to the standard output in a format suitable for a configuration file, as described earlier in this chapter. Although probably not necessary in this program, it encloses the assignment value in double quotes and escapes double quotes in the value using bash’s search-and-replace parameter expansion:

$ var=location
$ val='some"where'
$ printf "%s\n" "$var=\"${val//\"/\\\"}\""
location="some\"where"

See the options-parsing section in Listing 12-16 for an example of the output of print_config.

Listing 12-16. Define print_config Function

print_config() #@ Print values in a format suitable for a configuration file
{
local item var description

[ -t 1 ] && echo ## print blank line if output is to a terminal

for item in "${varinfo[@]}"
do
var=${item%%:*}
description=${item#*:}
printf "%-35s ## %s\n" "$var=\"\${!var//\"/\\\"}\"" "$description"
done

[ -t 1 ] && echo ## print blank line if output is to a terminal
}

Function: readline

If you are using bash-4.x or later, the readline function will place a value before the cursor for you to edit (Listing 12-17). With an earlier version of bash, it puts the value into the history so that you can bring it up with the up-arrow (or Ctrl+P) and then edit it.

Listing 12-17. Define readline Function

readline() #@ get line from user with editing of current value
{ #@ USAGE var [prompt] [default]
local var=${1?} prompt=${2:- >>> } default=$3

if [ ${BASH_VERSINFO[0]} -ge 4 ]
then
read -ep "$prompt" ${default:+-i "$default"} "$var"
else
history -s "$default"
read -ep "$prompt" "$var"
fi
}

Parse Command-Line Options

You can set the seven configuration variables with the a, d, f, h, p, s, and u options. In addition, you can specify a configuration file with the c option. A test run, which prints the configuration information but doesn’t attempt to create a tarball or upload any files, can be triggered with the toption. The m and q options offer the user a menu and a question-and-answer dialogue, respectively.

If a host is given as an option, a config file name is built using a standard formula. If the file exists, it is assigned to the configfile variable so that the parameters will be loaded from it. Usually this is all that would be needed to add to the command line for this purpose (Listing 12-18).

Listing 12-18. Parse Command-Line Options

while getopts c:h:p:d:u:a:s:f:mqt var
do
case "$var" in
c) configfile=$OPTARG ;;
h) host=$OPTARG
hostconfig=$configdir/$scriptname.$host.cfg
[ -f "$hostconfig" ] &&
configfile=$hostconfig
;;
p) port=$OPTARG ;;
s) source=$OPTARG ;;
d) dest=$OPTARG ;;
u) user=$OPTARG ;;
a) archivedir=$OPTARG ;;
f) syncfile=$OPTARG ;;

t) test=1 ;; ## show configuration, but do not archive or upload

m) menu=1 ;;
q) qa=1 ;;
esac
done
shift $(( $OPTIND - 1 ))

Using options and redirection, this program can create new configuration files. Here, parameters are given on the command line, and defaults are used for those not given.

$ upload -t -h www.example.com -p 666 -u paradigm -d public_html \
-s $HOME/public_html/www.example.com > www.example.com.cfg
$ cat www.example.com.cfg
host="www.example.com" ## Remote host (URL or IP address)
port="666" ## SSH port
dest="public_html" ## Destination directory
user="paradigm" ## Login name on remote system
source="/home/jayant/public_html/www.example.com" ## Local directory to upload
archivedir="/home/jayant/work/webarchives" ## Directory to store archive files
syncfile=".sync" ## File to touch with time of last upload

Bits and Pieces

Listing 12-19 below shows the rest of the script.

Listing 12-19. The Rest of the Script

## If a configuration file is defined, try to load it
if [ -n "$configfile" ]
then
if [ -f "$configfile" ]
then
## exit if problem with config file
. "$configfile" || die 1 Configuration error
else
## Exit if configuration file is not found.
die 2 "Configuration file, $configfile, not found"
fi
fi

## Execute menu or qa if defined
if [ $menu -eq 1 ]
then
menu "${varinfo[@]}"
elif [ $qa -eq 1 ]
then
qa "${varinfo[@]}"
fi

## Create datestamped filename for tarball
tarfile=$archivedir/$host.$(date +%Y-%m-%dT%H:%M:%S.tgz)

if [ $test -eq 0 ]
then
cd "$source" || die 4
fi

## verbose must be set (or not) in the environment or on the command line
if [ ${verbose:-0} -gt 0 ]
then
printf "\nArchiving and uploading new files in directory: %s\n\n" "$PWD"
opt=v
else
opt=
fi

## IFS=$'\n' # uncomment this line if you have spaces in filenames (shame on you!)

if [ ${test:-0} -eq 0 ]
then
remote_command="cd \"$dest\" || exit;tar -xpzf -"

## Archive files newer than $syncfile
tar cz${opt}f "$tarfile" $( find . -type f -newer "$syncfile") &&

## Execute tar on remote computer with input from $tarfile
ssh -p "$port" -l "$user" "$host" "$remote_command" < "$tarfile" &&

## if ssh is successful
touch "$syncfile"

else ## test mode
print_config
fi

Summary

This chapter demonstrated seven methods of altering the runtime behavior of a script. If changes will be rare, variables defined in the script may be adequate. When that isn’t enough, command-line options (parsed with getopts) are often enough.

You can use a menu or question-and-answer dialogue both for runtime configuration and for creating configuration files that can be sourced on demand. Using differently named files for the same script can save typing. In some cases, setting a variable in the shell’s environment is enough.

Exercises

1. Add code to the upload script that checks that all variables have been set to legitimate values (e.g., that port is an integer).

2. Write a usage or help function, and add it to the upload script.

3. Add an option to the upload script to save the configuration if it has been saved.

4. Write a script that creates a configuration file in the same form as words.cfg, prompting the user for the information to put in it.