Ubuntu Hacks: Tips & Tools for Exploring, Using, and Tuning Linux (2009)

Chapter 8. Administration

Hacks 74-86

Yes, Ubuntu runs pretty smoothly out of the box, and if you've got a system that one person uses, you might not have to do much in the way of system configuration. However, the time may come when you need to add new users or provide tech support to friends and family you've turned onto Ubuntu. If so, then you'll find hacks in this chapter to help you out.

There are plenty of other cases where you may need to wear a system administrator's hat, and there are hacks in this chapter to help with those, too. Perhaps you need to manage and mount external drives, or mount directories from remote servers. And once in a while, things will go wrong: in many cases, you can peek into the system logs to figure out what happened, but sometimes you'll need to resort to a rescue disc.

And there's one system administration task no one can avoid: backups. You'll find hacks in this chapter for making traditional backups and keeping files between two or more computers in sync.

Hack 74. Edit Configuration Files

When you need to edit a configuration file from the command line in a pinch, use these tips for the ins and (especially) outs of vim.

If one thing is for sure about Linux, it's that it has no shortage of text editors. This reflects the largely text-based nature of Linux, from the command line, to the source code, to the configuration files that programs refer to. When you are in a desktop environment, you can use one of many graphical tools to edit text files; however, in an emergency situation, or when logged into a machine remotely, you may need to fall back on a command-line text editor. Under Ubuntu, the default text editor is vim, and this hack provides you with the basic information you need to make changes to configuration files using vim.

The vi editor has a rather mixed reputation. Some people love how quickly you can edit files with it, and others hate its steep learning curve. The vim editor (short for "Vi IMproved") takes vi and adds not only a number of powerful improvements, but also a number of user-friendly changes such as a complete, integrated help environment. Ubuntu includes vim by default, and even if you run vi, you are actually running vim in a vi-like emulation mode.

So, to edit a file, run vim on the command line followed by the filename as an argument. For instance, to edit the filesystem table (/etc/fstab), type:

$ sudo vim /etc/fstab

Unlike most other editors you may be used to, vim operates via a number of modes. Different keys perform different functions depending on which mode you are in. (This can be rather confusing the first time you use vim, since you will start typing only to see all sorts of strange and undesirable behavior.) For starters, there are two main modes you will be operating in: Normal mode (for navigating the file) and Insert mode (for editing text).

Navigate Through the File

Normal mode is the main mode you default to when you start vim. In this mode, you use a number of different keys to move through the file. Table 8-1 lists some of the main navigation keys and their functions.

Edit a Line

Once you have moved to where you wish to edit, you have a number of different options to edit a line. If you simply need to delete some text, the x key will delete the character under the current cursor. To add or change text, you will need to switch to Insert mode. Once in Insert mode, vim behaves a lot like other text editors; when you type a particular key, it will appear in the file. There are a number of ways to get into Insert mode, depending on where you want to start editing, but the simplest is to hit i, which will let you edit at the current position of the cursor.

Once in Insert mode, you can make any changes you need to a particular file. You can hit the Backspace or Delete key to delete characters and can still use the arrow keys to move around in the file. Once you are finished, hit the Esc key to exit Insert mode.

Copy and Paste

Sometimes when editing a file it is useful to copy and paste one line or a section of a line to another line. There are a number of ways to do so, some faster than others, but for new vim users, one of the easiest ways is through vim's Visual mode. To select a portion of text, type Esc until you are in Normal mode, move to the beginning of the text you want to copy, and then hit the v key to switch to Visual mode. Now move the cursor as normal, and you will notice that characters will be selected as it moves. When you have selected all of the characters you want, hit the y key to copy (or yank) the text. Then move to where you want to paste it and hit p.

Undo

Once nice feature that vim has that vi doesn't is unlimited undos. If you make a mistake and want to undo it, hit Esc until you are in Normal mode, and then simply type the u key to step back through each edit. If you find you need to redo a change, type Ctrl-R.

Save and Exit

After you have made your changes and are ready to save, you need to switch to Command mode. Command mode lets you run special vim commands, including the commands you need to save changes and exit. To save (or write) your changes, type :w and press Enter. When you are ready to quit, type :q and press Enter. You can also combine these two commands into a single command, and type :wq and press Enter.

Search and Replace

A common need when editing files is to substitute one word for another throughout the file. To do so in vim, type:

:s/word/replacement/g

to change all instances of the word on a particular line. To change all instances of the word within a file, type:

:%s/word/replacement/g

If you want to give confirmation before making each change, add a c to the end of the command:

:%s/word/replacement/gc

and vim will ask you before it makes any changes.

One particularly handy use of the search-and-replace feature when editing configuration files is for commenting or uncommenting multiple lines in the file. To do so, enter Visual mode, select each line you want to comment, and then type:

:s/^/#/

to comment the lines out with the # character. Replace # with whatever character that file uses for comments. If you instead want to uncomment a number of lines, select them in Visual mode and then type:

:s/^#//

or replace # with the character your file uses for comments.

Help

If you find you are lost, or you just want some help with vim's options, type:

:help

to enter vim's help screen. You can also follow :help with a particular keyword, so if you want to find out what the w key does, for instance, type:

:help w

Where to Learn More

This hack is by no means an exhaustive tutorial on how to use vim. vim has a number of advanced and powerful features that, once learned, allow you to make rapid changes to files. If you want to know more about vim's features and how to use them, type:

$ vimtutor

at the command line to try vim's built-in interactive tutorial program.

Hack 75. Manage Users and Groups

Use GUI and command-line tools to add, remove, and modify users and groups on the system.

So you have used your Ubuntu desktop for some time, and it has started to attract attention from the other members of your household. They all want in on the action, but you may not necessarily want them to be able to edit your files. The solution is to create new accounts for each person that wants to access your system. That way, everyone will have their own login, password, and home directory to store files. Under Ubuntu, you have a number of options to edit users and groups, both with GUI and command-line tools.

Use the User Administration Tool

Ubuntu provides a GUI tool that makes user and group management relatively easy. Click SystemAdministrationUsers and Groups to start the user administration tool (Figure 8-1). The interface is split into two tabs, the first for users and the second for groups, and has buttons on the right side to add, edit, and delete users and groups. By default, only the users and groups you most likely want to edit are displayed, but you can toggle the "Show all users and groups" checkbox at the bottom of the window to see everything.

Figure 8-1. Ubuntu user administration tool

To add a new user, click the Add User button. A new window will appear with a number of fields to fill out (Figure 8-2). The Username and Password fields are the most crucial to fill out. You can choose a password or you can have Ubuntu generate a random password for you. If you are a more advanced administrator, click on the Advanced tab to change the default group, shell, home directory, and user ID. Ubuntu also provides three user profiles to choose from. The Default profile has no extra privileges; the Desktop profile will give the user access to audio, video, and other peripheral devices; and the Administrator profile will allow the user to become root (despite its name, Default is not the default choice; Desktop is). All of these settings, including user access, can be changed later on; just select a particular user and click Properties. To remove a user, select it and click Delete. Note that none of these changes will be applied until you click OK and close the program.

Figure 8-2. User Account Editor configuration window

The Groups tab lets you configure group membership on the system. Group membership dictates who can sudo to root (members of the admin group) and who can administer printers, scanners, and other devices. To see the members of a particular group, select it and then click Properties (Figure 8-3). You can select new users to add to the group from the list on the left side of the window. To create a new group, click the Add Group button and fill out the fields in the window that appears.

Figure 8-3. Ubuntu group account editor

Manage Users and Groups from the Command Line

You can also add, change, and remove users and groups with command-line tools. To add a new user to the system, type:

$ sudo adduser

username

You will be prompted for the user's real name, along with other optional information, and, finally, you will be asked to enter the user's password. The adduser script will automatically create the user's home directory and a group for the user. You can change the user ID, default home directory, and shell with the --uid, --home, and --shell options, respectively. Later on, to tweak the user's settings, open the /etc/passwd file with your preferred text editor as root (read "Edit Configuration Files" [Hack #74] for a primer on editing text files from the command line).

To delete a user, use the deluser command:

$ sudo deluser

username

By default, this will remove only the user from the system, not any of the user's files. To remove the user's home directory and mail spool, add the --remove-home option. To remove all files on the system owned by the user no matter where they are, use the --remove-all-files option.

There are similar command-line commands to add and remove groups from the system. To add a group to the system, use the addgroup command:

$ sudo addgroup

groupname

You can use the --gid option to specify which group ID number to use. All system groups are found in the /etc/group file, so you can edit this file as root if you want to tweak group membership or other settings. If you want to add a current user to a current group, instead of editing /etc/group, you can type:

$ sudo addgroup

username

groupname

To remove a group from the system, use delgroup:

$ sudo

delgroup

groupname

Hack 76. Mount Any Filesystem

Tweak /etc/fstab to control which filesystems are mounted at boot.

By default, Ubuntu will automatically detect and configure mount points for any partitions it finds when Ubuntu is installed. However, if you add a new disk to the system, or you want to automatically mount an NFS or SMB share at boot time, you must resort to the tried-and-true Linux method: editing /etc/fstab.

The /etc/fstab file (short for filesystem table) keeps track of filesystems that you want to mount in static locations. Here is a standard Ubuntu fstab file, which provides a good example for what each field in the file stands for:

# /etc/fstab: static file system information.

#

# <file system> <mount point> <type> <options> <dump> <pass>

proc /proc proc defaults 0 0

/dev/hda1 / ext3 defaults,errors=remount-ro 0 1

/dev/hda5 none swap sw 0 0

/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0

/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0

Here you see a list of partitions Ubuntu will mount, where it will mount them, what type of filesystem they use, any special options that might need to be passed to the filesystem, whether the partitions should be included in partitions the dump utility backs up, and what order filesystems are checked at reboot time. It's a lot of information, to be sure, but when you want to add a new filesystem to the mix, there are only a few fields that are crucial.

Add a Local Partition

One of the simpler partitions to add to the fstab file is a local partition. Suppose, for instance, that we added a second IDE hard drive to the system (hdb), and it had a Windows FAT32 partition as the first partition (/dev/hdb1) and a Linux EXT3 partition as its second partition (/dev/hdb2). First, we need to create a place these filesystems will mount, so we create two new directories under /mnt called windows1 and linux1:

$ sudo mkdir /mnt/windows1 /mnt/linux1

Now we can add two lines to our /etc/fstab file, one for each of the new partitions:

/dev/hdb1 /mnt/windows1 vfat defaults 0 0

/dev/hdb2 /mnt/linux1 ext3 defaults 0 2

Notice the differences between these two lines. Besides the fact that the first column in each line lists a different partition and the second column has a different mount point, notice that in the third column we specified vfat for the filesystem type for the Windows partition and ext3 for Linux. The value that goes here varies depending on what type of partition you are mounting. Table 8-2 displays some of the common partition types and what value to put in /etc/fstab.

In the fourth column, both fields list defaults, which mounts the filesystem with default settings. Different filesystems support different mount options, such as whether to mount the filesystem as read-only or read/write, the sorts of ownership that files in a partition might have, and so forth. We can get more information about special options that a filesystem supports from the mount manpage:

$ man 8 mount

In the fifth column, we left the values at 0 because we are not planning on using the dump utility to back up these partitions. This is a safe value for most partitions you might want to mount.

In the final column, notice we set the Windows partition to 0 and the Linux partition to 2. We did so because we don't want Linux to automatically run fsck on our Windows partition, yet we do want the Linux partition to be among the drives that are checked. We set the Linux partition's value to 2, because the standard is to set the root filesystem (/) to 1 so that it is checked first, then set all other Linux filesystems to 2.

Mount the Filesystems

Now that these two filesystems are configured, we can use the mount command to mount them:

$ sudo mount /mnt/windows1

$ sudo mount /mnt/linux1

While we could have used the mount command to mount these filesystems manually, without bothering to add any entries to fstab, when we set up the filesystems in fstab, we don't have to list a lot of optionsjust the device or mount point we want to mount. After mounting our devices, we can type either mount or df to see all of our mounted filesystems.

Mount Network Filesystems

Network filesystems use slightly different syntax than ordinary partitions, so they get a separate section here. Specifically, the syntax you use to describe the filesystem (the first column) is different. For this example, we have an NFS share on host file1 at /mnt/shares, and a SMB file share on host file2 called data. We want to mount the NFS share at /mnt/share1 and the SMB share at /mnt/data1. To mount both of these partitions at boot time, we would add the following lines to our /etc/fstab:

file1:/mnt/shares /mnt/share1 nfs defaults 0 0

//file2/data /mnt/data1 smb defaults 0 0

Notice the difference in syntax. NFS shares follow the hostname:/path/to/share syntax, while SMB shares follow the //hostname/share syntax. Other than that, the remaining fields are similar to the first examples, and we could pass special NFS or SMB options in the options field if we needed to.

To mount these shares, we would make sure that the /mnt/share1 and /mnt/data1 directories existed, and then we would type:

$ sudo mount /mnt/share1

$ sudo mount /mnt/data1

As with local partitions, we can use the mount or df commands to check the status of these mount points.

For more information on the syntax for the /etc/fstab file, and what sorts of mount options you can use for particular filesystems as well as what filesystems Linux can mount, see the fstab(5) and mount(8) manpages, respectively.

Hack 77. Control Startup Services

Use GUI and command-line tools to start, stop, disable, and enable services.

As Ubuntu boots, you might notice text scrolling by, detailing all of the different things Ubuntu is doing. Among these things are a number of services that Ubuntu enables at boot time, such as the cron scheduling service, the system logger, and the graphical login manager. If you have installed other services on your system, such as a web server, those services will also be enabled at boot. Sometimes, though, you may want to either stop or temporarily disable these services, and Ubuntu provides a number of ways to do this, both graphically and through the command line. This hack shows some of the more common ways to control startup services.

Services Administration Tool

Ubuntu provides a graphical tool to manage services that start up at boot time. Click SystemAdministrationServices, or type:

$ sudo services-admin

to start the Services Administration Tool. Figure 8-4 shows how simple the program is: you have a list of services with a checkbox next to each of them. To disable a service, just deselect its checkbox and click OK. Currently, this application supports changing only whether a service starts at boot, so to manually start, stop, or restart a service, you will need to refer to the command-line method.

Figure 8-4. Ubuntu Services Administration Tool

Command-Line Method

Before learning how to start, stop, and disable services from the command line, it's important to understand Ubuntu's startup process and how Ubuntu determines which programs to run when it starts. For most Linux distributions (including Ubuntu), System V init scripts govern which programs start at boot and which programs don't. All System V init scripts that could potentially be run at boot are typically located in the /etc/init.d/ directory for Ubuntu. Not every script in /etc/init.d is executed at boot, however. Linux organizes which scripts to run for different circumstances into runlevels; most Linux systems have seven runlevels, ranging from 0 to 6. Think of a runlevel as a checklist of programs for Ubuntu to start before it presents a login.

A few of these runlevels are set aside for special states in a Linux system:

Runlevel 0

Halts the system.

Runlevel 1

Sets up single-user mode.

Runlevels 2-5

Set up different multiuser modes. Although, typically, only one or two of these are used by a distribution.

Runlevel 6

Reboots the system.

Each runlevel has a directory that stores symlinks to certain init scripts in /etc/init.d, which are started when that runlevel is selected and stopped when it is exited. Ubuntu puts these symlinks under /etc/rc<runlevel>.dfor example, all runlevel 2 scripts are located in /etc/rc2.d/.

If you look in one of these runlevel directories, you'll notice that many of the symlinks to scripts in /etc/init.d have odd names that begin with an S, K, or D; then a number; and finally the name of the script. Ubuntu defaults to runlevel 2, so here is a sample /etc/rc2.d directory:

greenfly@ubuntu:~$ ls -l /etc/rc2.d/

total 0

lrwxrwxrwx 1 root root 20 2006-01-21 14:48 K77ntp-server -> ../init.d/ntp-serverlrwxrwxrwx 1 root root 17 2006-01-07 08:46 S05vbesave -> ../init.d/vbesave

lrwxrwxrwx 1 root root 15 2006-01-07 08:19 S10acpid -> ../init.d/acpid

lrwxrwxrwx 1 root root 18 2006-01-07 08:17 S10sysklogd -> ../init.d/sysklogd

lrwxrwxrwx 1 root root 15 2006-01-07 08:17 S11klogd -> ../init.d/klogd

lrwxrwxrwx 1 root root 14 2006-01-07 08:47 S12dbus -> ../init.d/dbus

lrwxrwxrwx 1 root root 13 2006-01-07 08:50 S13gdm -> ../init.d/gdm

lrwxrwxrwx 1 root root 13 2006-01-07 08:46 S14ppp -> ../init.d/ppp

lrwxrwxrwx 1 root root 16 2006-01-07 08:48 S19cupsys -> ../init.d/cupsys

lrwxrwxrwx 1 root root 15 2006-01-07 08:52 S19hplip -> ../init.d/hplip

lrwxrwxrwx 1 root root 14 2006-01-07 08:47 S20apmd -> ../init.d/apmd

lrwxrwxrwx 1 root root 22 2006-01-07 08:48 S20hotkey-setup -> ../init.d/hotkey-setup

lrwxrwxrwx 1 root root 21 2006-01-07 08:46 S20laptop-mode -> ../init.d/laptop-mode

lrwxrwxrwx 1 root root 17 2006-01-07 08:16 S20makedev -> ../init.d/makedev

lrwxrwxrwx 1 root root 23 2006-01-07 08:18 S20nvidia-kernel -> ../init.d/nvidia-kernel

lrwxrwxrwx 1 root root 19 2006-01-07 08:48 S20powernowd -> ../init.d/powernowd

lrwxrwxrwx 1 root root 15 2006-01-07 08:46 S20rsync -> ../init.d/rsync

lrwxrwxrwx 1 root root 21 2006-01-07 08:47 S25bluez-utils -> ../init.d/bluez-utils

lrwxrwxrwx 1 root root 15 2006-01-07 08:17 S25mdadm -> ../init.d/mdadm

lrwxrwxrwx 1 root root 17 2006-01-07 08:46 S89anacron -> ../init.d/anacron

lrwxrwxrwx 1 root root 13 2006-01-07 08:46 S89atd -> ../init.d/atd

lrwxrwxrwx 1 root root 14 2006-01-07 08:46 S89cron -> ../init.d/cron

lrwxrwxrwx 1 root root 17 2006-01-07 08:18 S98usplash -> ../init.d/usplash

lrwxrwxrwx 1 root root 22 2006-01-07 08:46 S99acpi-support -> ../init.d/acpi-support

lrwxrwxrwx 1 root root 19 2006-01-07 08:16 S99rmnologin -> ../init.d/rmnologin

lrwxrwxrwx 1 root root 23 2006-01-07 08:16 S99stop-bootlogd -> ../init.d/stop-bootlogd

As you can see, this directory is full of symlinks that point to a script in the init.d directory. The letter at the beginning of each filename tells init when to execute this script. If the script begins with an S, then init starts the script when it goes through the runlevel. If the script begins with a K, then init stops (or kills) the script when it changes to a different runlevel. If the script begins with a D, then that script is disabled for the time being, and init ignores it. init runs the scripts in numerical order, so the numbers in each script let you know in which order they are to be run. This is useful to ensure that dependent services start after the service they are dependent on.

When Linux boots and starts the init process, it reads its configuration from /etc/inittab, which configures each available runlevel, the default runlevel to use, and some other settings. Next, init loads any system scripts from a special system runlevel directory at /etc/rcS.d. These scripts load daemons and services that are vital to the boot process. Lastly, init runs any startup scripts for the default runlevel in alphabetical order.

Change the Runlevel

You can change the runlevel yourself on the command line with the init command. To switch to single-user mode from the command line, type:

$ sudo init 1

This command runs all of the shutdown scripts for your current runlevel and then any startup scripts for single-user mode. To change back to the default multiuser runlevel for Ubuntu, type:

$ sudo init 2

Manually Start and Stop Services

You can start and stop scripts manually by running the script with the start or stop argument. For example, to stop the CUPS service from the command line, type:

$ sudo /etc/init.d/cupsys stop

To start the service back up, type:

$ sudo /etc/init.d/cupsys start

Most scripts also support a restart argument that will run stop, then start for you. Most init scripts are also configured to output the list of arguments they support when you execute them without any options:

greenfly@ubuntu:~$ sudo /etc/init.d/cupsys

Password:

Usage: /etc/init.d/cupsys {start|stop|restart|force-reload|status}

Disable Scripts from Starting

To disable a script, you must know your default runlevel. On Ubuntu, the default runlevel is usually set to 2, but you may want to double-check your default runlevel before you start disabling services. All runlevels are defined in /etc/inittab, so to check the default runlevel, type:

greenfly@ubuntu:~$ grep initdefault /etc/inittab

id:2:initdefault:

As you can see, in this example, the default runlevel is in fact 2. Now change to the directory containing all of the scripts for that runlevel (/etc/rc2.d) and find the script you want to disable. To disable a service, just rename its script by changing the S to a D. For example, to disable the cupsys service, type:

greenfly@ubuntu:~$ cd /etc/rc2.d

greenfly@ubuntu:/etc/rc2.d$ sudo mv S19cupsys D19cupsys

To enable it again, rename it back by changing the D to an S:

greenfly@ubuntu:~$ cd /etc/rc2.d

greenfly@ubuntu:/etc/rc2.d$ sudo mv D19cupsys S19cupsys

You'll still need to stop the service as shown earlier if you want to shut it down right away, but renaming it will control whether it's started the next time you reboot (or change runlevels).

Hack 78. Build Kernels the Ubuntu Way

Special tools take the pain out of compiling and installing custom kernels.

Debian boasts some very useful tools to make the job of building and installing kernels much easier, and of course those same tools are available in Ubuntu as well. They streamline the process of compiling a custom kernel and building a .deb package around it, allowing you to install a new kernel the same way you would any other package. This makes it possible to build a kernel on one machine and then just install the package on other machines, without having to recompile or track down all the loose endsgreat if you need to upgrade a number of similar machines!

To build a custom kernel "the Ubuntu way," you will need to get hold of the kernel source and a variety of tools to configure, compile, and package it.

Get the Source

You can get the official kernel source yourself directly from http://ftp.kernel.org/pub/linux/kernel/, or a mirror if you like, but of course there are also packages of the kernel source. A quick:

$ apt-cache search linux-source

will get you a list of Linux kernel source packages all ready to go. Picking one as an example, you could just run:

$ sudo apt-get install linux-source-2.6.15

to download the source to your system. You'll then end up with an archive sitting in /usr/src, which is where most kernel work is done. Now you can extract the source package:

$ cd /usr/src

$ sudo tar -xjf linux-source-2.6.15.tar.bz2

That will leave you with a /usr/src/linux-source-2.6.15 directory, decompressed and ready to configure. First, though, create a symlink to it called linux, like this:

$ sudo ln -sf linux-source-2.6.15 linux

The f option forces creation of the new symlink even if there's already an existing link to an old kernel source. Now you can easily get to your kernel source tree just by typing:

$ cd /usr/src/linux

Tools You Will Need

To install the main tools, you will need to run:

$ sudo apt-get install kernel-package libncurses5-dev

This will also cause your computer to pull down a big list of supporting packages that are dependencies.

Configuration Methods

There are a number of ways to configure a kernel prior to compiling it, and all but the first are invoked as arguments to the make command. Make sure you have a shell open in the kernel source directory (which should be /usr/src/linux if you followed our directions earlier) before you try any of these. In increasing order of sophistication, they are:

editing .config

The actual configuration is saved in the source directory as a text file called .config. If you really want to (or if you are trying to find an option some README has listed by name, such as CONFIG_PACKET) you can edit this file directly using a text editor such as vim or Anjuta. This should need to be done very rarely, though.

make config

The most basic approach, this will just ask you a whole heap of questions, one after another. Personally, we hate configuring the kernel this way, because it's sequential and you have to go through a heap of stuff you probably couldn't care less about. Only bother with this as a last resort, such as if other options aren't available to you for some reason.

make menuconfig

This is the most common way to do the configuration. It displays a nice, keyboard-driven menu that you can navigate using arrows, Enter, and the spacebar. However, one catch that can trip you up is that you need to have the development libraries for Ncurses installed, not just Ncurses itself. That's why we had you install the libncurses5-dev package a few paragraphs ago!

make xconfig

The preferred method for machines with X, this option is very similar to menuconfig, except it's all done in a nice point-and-click GUI with mouse navigation. The possible gotcha to getting this working is you'll need tk8.3. If you get errors saying it can't find "wish," run:

$ sudo apt-get install tk8.3

and all should be well.

Configuring the Kernel

Whichever configuration method you choose, the next thing to do is to examine the settings for the different kernel options and make changes as required.

There are a huge number of kernel options, so to help you find things, they are grouped in a logical way in different major and minor sections. The different configuration methods have their own ways to represent this, but they should all be fairly straightforward to follow.

We won't go into details of how to do the kernel configuration here, because what you'll need to configure will vary dramatically, and there are plenty of tutorials on the Net, including the Kernel-HowTo, that explain this in detail. Basically, though, you need to work through each menu in turn to find options and modules you want to enable, and set them to either off(not available), on (compiled straight into the kernel), or module (compiled separately so it can be loaded when needed).

Once you've finished going through the options, quit and save to have your choices written out to a configuration file to be used by the compiler.

One little trick to note is that the actual configuration details are stored in a file called .config in the kernel source directory. Once you've gone to all the trouble of setting up a kernel the way you want it, moving to a new kernel can be painful if you have to go through the configuration process from scratch, and you'd be almost certain to miss something crucialwe know we usually do! To make things easier for yourself, you can copy the .config file into the new kernel source directory to have all your options carried over automatically. Then you can run the configuration again just to check things over, save, and go on as before. Provided you're not changing to a totally new kernel type, this trick can save you a lot of time. Beware if you're moving to a totally different kernel, though, such as from 2.4.x to 2.6.x, because many of the options will be totally different and while the old config will still work, it may not have some options activated that appear only in the new kernel. You can also run make oldconfig after copying in the old .config file, which will then prompt you only for options that are new or have changed since you configured your previous kernel.

Compiling and Packaging

This is the point when your kernel is actually compiled and placed in a package. Normally, that would be a lot of work, but thanks to the tools provided by kernel-package, it's now one of the easiest steps.

If you have read a traditional Kernel HOWTO, you've probably seen a sequence of commands such as make dep && make clean && make bzImage used to build the dependencies and then the kernel itself. You don't need to do any of that; instead, just type:

$ sudo make-kpkg kernel-image

while in the /usr/src/linux directory, and those steps will all be taken care of. The kernel-packaging tools first compile your kernel and modules according to the configuration you just generated and then build a Debian package called kernel-image-<version_architecture>.deb outside the current source directoryi.e., in /usr/src.

This stage can take a while depending on your machine speed and which modules you selected. Expect anything from 10 minutes to a half hour, during which time you'll see a very long stream of debugging information that probably won't be of any interest to you at all, unless something goes seriously wrong.

Install Your Kernel Package

You're almost there! What you have now is a kernel, custom-compiled according to your requirements and set up as a Debian package ready to install.

Installing your new kernel is now just a matter of using dpkg to install the package as you would any other Debian package:

$ sudo dpkg -i

kernel-image-2.6.15_10.00.Custom_i386.deb

(replacing kernel-image-2.6.15_10.00.Custom_i386.deb with whatever your package is called). It may ask you if you want to create a boot floppy using that image; the dozens of machines we manage don't have a single floppy drive among them, so we always say no, but if you want to, you can say yes to have it create one for you.

dpkg will also take care of updating your bootloader configuration so your new kernel will be available next time you boot. Both GRUB and LILO are managed automatically, but if you use a different bootloader, you may need to update it yourself at this point.

What happens when you install the package is that dpkg puts your new kernel image in the /boot directory where kernels are normally stored, creates a symlink from /vmlinuz to your new kernel (you can verify this by typing ls -l /vmlinuz), modifies your /boot/grub/menu.lst or /etc/lilo.conf as appropriate so your bootloader can find your new kernel, and moves and renames the previous kernel so it's available as an emergency fallback in case your new kernel borks.

Note that at this point, you will not actually be running the new kernel; it's just set up, ready to go for next time you reboot. Installing a new kernel is one of the very few things that actually requires you to reboot a Linux system.

Rebooting and Testing

Time for the big test! If you are running X, exit the session and reboot. If not, just type sudo reboot. When GRUB loads, you will have the option of pressing Esc to see a list of available kernels, including your new one. Select it and hit Enter.

Once your machine has booted, you can use uname to check which kernel you are running:

$ uname -a

For more information about which modules were loaded and whether the new kernel correctly detected your hardware, you can look through the dmesg log:

$ dmesg | less

If everything worked as expected, congratulations! You've just compiled and installed a custom kernel "the [Debian|Ubuntu] way."

Installing on Other Machines

This is where the convenience of building kernels as packages becomes most obvious. If you want to install your custom kernel on other machines, the process is very simple: just copy the .deb package you created to the target machine and install it using dpkg exactly as before. Simple!

There's no need to compile the kernel on each machine, copy the source code to them, or even have a compiler installed on them. All the hard work was done once on one machine and doesn't need to be repeated.

If you run a server farm or computer lab and have a lot of local machines to install your kernel on, the best way to do so is probably to put the kernel package on a local package repository and have all your computers fetch it over the network. You can learn more about how to do so in "Create Your Own Package Repository" [Hack #65].

Install Multiple Copies of One Kernel Version

Often you'll want to install multiple copies of the same kernel version while doing testing, but that can cause problems: modules are installed in a directory based on the kernel version number, so if you want to try out different configuration options, you'll run into problems. Luckily, the kernel package tools allow you to work around that limitation by appending an arbitrary version string to the regular version string. This means you can install multiple builds of the same kernel version and keep everything neatly organized on disk, as well as select which one you want to use at startup.

To modify the version string, just pass the --append-to-version option when creating the package, like so:

$ fakeroot make-kpkg --append_to_version=-jon17

kernel-package

This would generate a kernel package something like kernel-image-2.6.15-jon17_10.00.Custom_i386.deb.

Hack 79. Back Up Your System

Use Ubuntu's Simple Backup utility to easily set up standalone or recurring backups of your important files.

Backups always seem to be the thing we think about after it's too late. Just about every computer user we know has been bitten at least once by failing to make a backup of an important file or directory. Yet what should you back up? How often? Out of the hundreds of backup tools out there, which should you use? Ubuntu has answered all of these questions with the aptly named Simple Backup tool. This tool allows you to schedule automatic recurring backups of predefined important files on the system, or, if you want more control, it allows you to set fine-grained backup options that better suit your needs. In this hack, we discuss some of the main options in the Simple Backup tool and how to set up a backup-and-restore solution for your computer.

Get Simple Backup

The first step is to install Simple Backup. Its package is called sbackup and is available from the universe repository, so if you haven't yet enabled universe, you will need to do so. If you need more instructions on how to enable repositories, refer to "Modify the List of Package Repositories" [Hack #60] or the individual hacks corresponding to your preferred package-management tool [Hacks #54, #55, and #56].

Once the program is installed, two new menu entries will appear under SystemAdministration: one called Simple Backup Config and a second called Simple Backup Restore. Select Simple Backup Config to see the fairly straightforward main window shown in Figure 8-5.

Figure 8-5. The main Backup Properties window

Configure Simple Backup

The General tab displays three main options for you to choose from:

Use recommended backup settings

If you are new to Ubuntu or aren't quite sure what you should back up, but you do want to set up recurring backups, select this option. It will automatically set up daily incremental and weekly full backups of important directories on your system and store them under /var/backup. Note, however, to prevent the backup from filling up your disk, this option excludes files larger than 100 MB as well as a number of multimedia files, so if you want to back up either of those, move to the next options.

Use custom backup settings

If you have specific backup needs, or the default choices from the recommended backup don't suit you, choose this option to access the fine-grained configuration options in the following tabs (more on them shortly).

Manual backups only

If you don't want to set up a recurring backup schedule, but you do want to be able to manually back up some files from time to time, select this option and read on for more information on how to configure what to back up in the following tabs.

Use recommended backup settings

If you are new to Ubuntu, don't know what exactly to back up, or don't want to fuss with a configuration, check "Use recommended backup settings" in the General tab and then click Save. Ubuntu will schedule a nightly incremental backup of important files on the system, along with a weekly full backup. If you later find you need to restore a file, see the "Restore from Backup" section later in this hack.

Use custom backup settings

There are a number of reasons why the default backup settings might not suit you. For instance:

· You may have a number of important files that are larger than 100 MB, and you need to be able to back those up.

· You want to keep a backup of your multimedia files.

· You want to save the backups somewhere other than /var/backup (either another directory on the system or possibly a directory on a remote system).

· You want to include or exclude other files or directories from the backup.

· You want to schedule your backups to run at different times or different frequencies from the default.

If you want to enable custom recurring backups, select "Use custom backup settings." If you want to configure a custom manual backup instead, select "Manual backups only." Apart from the Time tab being disabled in the manual backup option, the configuration ability will be identical.

Configuring Custom Backup Settings

Here are some settings you can modify to customize the backup of your system.

Configure files to include

By default, Ubuntu includes the major important directories you might want to back up on a system, such as /etc, /var, /home, and /usr/local. It's possible you might have an extra directory or file that you want included in addition to these (possibly an extra mount point under /mnt). If so, click the Include tab to see the directories and files currently included in the backup (Figure 8-6).

Figure 8-6. Define which directories to include in the backup

Configure files to exclude

There are a number of different files that Simple Backup excludes by default. Click the Exclude tab to configure these exclusion options. Figure 8-7 shows an example of some multimedia file types that are excluded by default. If you do want to back up your MP3s for instance, select "mp3" from the list of excluded files and click Remove. If you want to add a particular file type to the exclusion list, click the Add button and choose a predefined file type from the drop-down menu, or enter the file's extension and then click OK. This particular exclusion list depends on actual filename extensions, so it will work only on files with the correct extensions in their filenames.

Figure 8-7. Define which file types to exclude

A number of different exclusion properties are allowed via the tabs along the left side of the window. In addition to file types, click Paths to exclude by individual files or directory paths. If you are familiar with defining patterns using regular expressions, click the Regex tab and you will be able to define an exact pattern to match files to exclude. The final tab in this window, "Max size," is important because it will greatly influence the size of your backups. If you choose the custom configuration, it will default to excluding files larger than 10 MB. Depending on the size of the files you want to back up, you may want to raise or completely disable this option, but note that in doing so you risk filling up your backup directory if it doesn't have adequate space.

Configure the backup destination

The Destination tab allows you to change where Simple Backup will store backups. You can select the default of /var/backup, choose a custom directory somewhere on your system, or even select a remote FTP or SSH share. To back up to a remote share, use the "Connect to Server" dialog shown in Figure 8-8 or type ssh:// or ftp:// followed by the username:password, then @, then the remote host to connect to and the remote directory.

Figure 8-8. Back up to a remote share

Configure recurring backups

If you want to have Simple Backup automatically back up your system periodically, click the Time tab to configure how frequently to back up (Figure 8-9). You can choose anything from never backing up to backing up hourly, daily, weekly, or monthly. You can also configure the time, day of week, or day of month to back up, depending on how frequently you want to back up the system. Finally, you can configure how often to do a full backup. By default, a full backup is performed weekly, with incremental backups being performed otherwise to save space.

Figure 8-9. Schedule your backups in the Time tab

When you are finished with your backup configuration, click Save to save the configuration and exit, or click Backup Now to start the backup right this instant based on your configuration.

Figure 8-10. The "Restore files/directories" window

Restore from Backup

So you have accidentally deleted a file, or you need to roll back to a previous version of it. If you have already configured recurring backups, or have manually performed backups using Simple Backup, click SystemAdministrationSimple Backup Restore. Figure 8-10 shows the main window with a particular backup set selected. You can choose from any previous backups from the "Available backups" drop-down menu, and you can then browse through the directory structure of the backup to find the file or directories you want to restore. Once you have found the files or directories you want to restore, click Restore to overwrite any current version of the file with the backed-up version, or click Restore As to restore the file or directory to a new location with a new name so you don't overwrite the current version.

Hack 80. Clone an Installation

Export a list of installed packages on one Ubuntu system, and import them into another to build a duplicate system.

We've installed many different Debian-based distributions over the years, and one thing we've found handy to have is a complete list of packages you have installed. If you want to create a system that is similar to a different system you have already set up, it can be difficult to remember each and every package you had installed. In this hack, we cover a method to export your current package list and import it into a new system.

Export the List of Installed Packages

The first step in cloning an installation is to grab the complete list of installed packages from the first system. To do so, you basically instruct dpkg to dump its entire list of packages, filter out any packages that aren't currently installed, and then redirect the output to a file:

$ sudo dpkg --get-selections | grep '[[:space:]]install$' | \\

awk '{print $1}' > package_list

Next, copy this text file to the destination system over the network, via a USB key or whatever method you prefer. You may also want to copy over the /etc/apt/sources.list file from the base system, since the new system may not have all of the same repositories enabled (if the repositories aren't the same, the destination system may not be able to find some of the packages in the list).

Prepare the Destination System

Now you need to prepare the destination system. If both systems are running the same release of Ubuntu, this may be as simple as just copying the initial system's /etc/apt/sources.list on top of the one on the current system. Otherwise, if the systems are from different releases, you will want to compare the initial system's sources.list with the /etc/apt/sources.list file on your destination system and see if there are any extra third-party repositories or repository subcategories that need to be enabled. (Read "Modify the List of Package Repositories" [Hack #60] for more information on Ubuntu's repositories and how to edit sources.list.)

Once your sources.list file is settled, update your package list to make sure you get the latest version of the packages:

$ sudo apt-get update

Import the Package List

To import the package list, pipe the entire list to xargs, which then splits it into manageable chunks for the apt-get command:

$ cat package_list | xargs sudo apt-get install

If you are migrating to a different Ubuntu release, this will require a bit of trial and error, since you will likely get complaints about packages no longer existing. The simple fix for this is to edit the package list and remove the package from the list, save your changes, and then rerun the command. You will likely need to do this a number of times, but eventually you will have a list of valid packages. A more complicated but thorough fix would involve checking the latest list of packages for a potential update or replacement for the package that no longer exists and installing them one by one.

Once apt-get completes, you are finished. All of the files from the package list will have been imported into the new system. Now keep in mind that this doesn't mean that all of the settings have transferred over. To do that, you will likely need to copy settings from the /etc directory or possibly other directories, depending on the package.

Hacking the Hack

It can be rather handy to have a complete list of installed packages for backup and restore purposes. An easy way to automatically generate a complete list is to have a cron job update the list of installed packages every night. To do so, create the following script and save it in /etc/cron.daily/package_list:

$!/bin/sh

dpkg --get-selections | grep '[[:space:]]install$' | \\

awk '{print $1}' > /etc/package_list

Now make the script executable:

$ sudo chmod a+x /etc/cron.daily/package_list

This /etc/package_list file can then be backed up [Hack #79] with the rest of your system settings.

Hack 81. Rescue an Unbootable System

When you've had a little too much fun with your new Ubuntu system and it no longer boots, here are a few options to get your computer back in running order.

If you are reading this hack, we want to offer you our condolences. It can be stressful and upsetting when your system won't boot. We're sure you are thinking right now about all of those files you wished you backed up (We know this is a bad time, but once your system is back up and running, you might want to check out "Back Up Your System" [Hack #79]). We've certainly been in the same situation far too many times, but so far we've also been able to bring our systems back to life. While it would be impossible to cover every possible scenario that might cause a system to no longer boot, we're going to go over how to use the Ubuntu install CD in rescue mode to fix the problem and describe some common rescue scenarios.

Boot into Rescue Mode

First, find your trusty Ubuntu install CD and reboot your system to its initial boot screen. Among the number of options is "Recover a broken system." Select this option, and Ubuntu will start what appears at first to be the default installation program. You will be prompted for some standard language and network settings as you are during the installation, but these steps are only to set up the initial rescue environment. Notice that in the top-left corner of the screen, "Rescue mode" appears.

Continue through these dialogs until you are prompted to choose your root device. In Figure 8-11, you see a sample of this dialog with a lot of different drives. How do you pick the right one? If you aren't too familiar with the partition layout of your drives, it might require a bit of trial and error, but here's a basic rule of thumb:

Figure 8-11. The rescue-disk root-device dialog

· If you installed Ubuntu as a standalone system and had Ubuntu overwrite everything on the current drive, your root filesystem is probably the very first partition on the list.

· If you installed Ubuntu in a dual-boot configuration with Windows, your root filesystem is probably the second partition in the list.

After you choose your root filesystem, the Ubuntu CD will attempt to mount it, and it will continue to the next dialog if it was successful. If it wasn't successful, you might have chosen the wrong partition, so pick another and try again.

Common Rescue Scenarios

The next dialog will present you with a number of different rescue options (see Figure 8-12). The operation you choose here will vary greatly depending on the symptoms of your unbootable system, so instead of describing everything possible with these options, we're going to describe a few common rescue scenarios and what to do from this dialog to fix the problem.

Figure 8-12. The rescue-mode chooser

The system won't go past the boot menu, or the menu is missing

This is a common symptom for an unbootable system. Perhaps you reinstalled Windows or some other operating system as a dual-boot system. Whatever the reason, your default GRUB boot menu is now missing and needs to be restored.

From the rescue operations menu, select the "Reinstall GRUB boot loader" option. The next screen that appears is the same one you might remember being prompted with when you first installed Ubuntu. It asks you where you would like to install the GRUB boot loader. Unless you remember setting up GRUB somewhere in particular, chances are you probably installed it to the Master Boot Record on your first hard drive. If so, type (hd0) and then continue. The rescue mode will reinstall GRUB and drop you back to the main "Rescue operations" menu. Select "Reboot the system," and you should hopefully be presented with your standard boot menu.

The system goes past the boot menu but can't access the root filesystem

This can be a tricky symptom, as a number of different problems can be the cause, from corrupted filesystems to new problematic kernels. Before you go through any steps in the rescue mode, first try to boot and select a different kernel option in the boot menu. If that kernel boots, then there is likely some sort of hardware support problem for your latest kernel. Check for updates to your kernel package and, in the meantime, boot from the working kernel. If you experience the same problem with the different kernels in your boot menu, continue to the following steps.

For this sort of problem, the troubleshooting will need to be done directly from the rescue CD, so select the "Execute a shell in the installer environment" option. You will then be dropped to a basic BusyBox shell where you can perform some basic diagnostics and repairs:

Test the root filesystem

First, test and see if you can read your default root filesystem. The rescue-mode CD mounted this filesystem at /target earlier in the boot process. Type:

~ # ls /target

and see if you get a list of files somewhat similar to what you see in Figure 8-13 has sample df output you can reference. On a default Ubuntu install, your root filesystem will likely be on /dev/discs/disc0/part1.

Figure 8-13. Some sample diagnostics from the BusyBox shell

Repair the filesystem

Now use the fsck tool to attempt to repair your filesystem. First, you will need to unmount the /target partition if it is currently mounted:

~ # umount /target

The rescue-mode environment has fsck utilities only for ext2, ext3, and jfs, so if you chose another filesystem during the install, you will have to resort to a different rescue disk. Ubuntu defaults to ext3, so if you went with the default, type the following to check and repair your drive:

~ # /sbin/

fsck.ext3

-f

/dev/discs/disc0/part1

Replace fsck.ext3 with fsck.ext2 or fsck.jfs if you have an ext2 or jfs partition, respectively, and replace the partition with your particular root partition. Once the fsck completes, attempt to mount the partition under /target again and try to access it with ls:

~ # mount

/dev/discs/disc0/part1

/target

~ # ls /target

If you see a number of files and directories, congratulations, your drive is repaired! Type exit to leave the BusyBox shell and reboot, and see if you can boot into your system successsfully.

The system boots and continues through part of the boot process, but hangs at a particular service

Whether it is due to a poorly written init script or some other bug, on some rare occaisions, an init script might stall and refuse to continue through the boot process. When this happens, you can't drop to a shell to repair the problem, so you must resort to a rescue CD. Make a note of the service that is stalling, and boot into the rescue mode of your Ubuntu install CD.

To fix this sort of issue, you will need to actually drop to a shell within your root filesystem's environment, so select "Execute a shell in /dev/discs/disc0/part1" from the Ubuntu rescue operations menu. (This option might be named something slightly different if you picked a different partition as your root filesystem.)

This option will drop you to a shell from within your own root filesystem. You can navigate this filesystem and run commands as though you had actually booted into it. At this point, you will want to identify the init service that is causing you problems and disable it. "Control Startup Services" [Hack #77] has instructions on how to identify and disable startup services from the command line, so read through that hack for the specifics. In a default Ubuntu install, you will likely want to look for your init service in either /etc/rc2.d or /etc/rcS.d. Once you find the offending init script, rename it, changing the S at the beginning of the filename to a D.

Once the init script has been disabled, type exit to exit out of the shell and then reboot and attempt to boot back into your Ubuntu system. The disabled init script should not stall you this time, so you should be able to get to your default login screen.

Conclusion

It's true that these are only a few of the possible problems that could render a system unbootable. If you have tried all of these rescue options and still can't fix your system, don't give up! Read "Get Help" [Hack #2] for some information on how to enlist the help of some fellow Ubuntu users.

Hack 82. Check the Captain's Log

Find out where Ubuntu logs important system information so you can track down the cause of startup and system errors.

When you use a system long enough, eventually you find you need to put on your detective hat. Maybe you've added a piece of hardware or plugged in a new device, and you aren't sure whether the system is recognizing it. Maybe you upgraded some software, and now it's not working quite right. Whatever the reason, when you want to track down what's really going on under the hood of your Ubuntu system, it's time to look through logs.

A normal desktop system generates a surprising amount of logs in a day, even if nothing is wrong. When you connect to a network, plug in a new device, log in, or do any number of things, the system generates logs. The majority of system logs reside in the /var/log directory. Some of these logs overlap; for instance, logs from daemons will show up both in the daemon.log and in syslog. Here are some of the main logs you will find under /var/log, along with their uses:

syslog

syslog is the primary system log and contains log output from daemons and other programs running on the system, such as dhclient, cron, init, xscreensaver, and some kernel logs. This log is the first place to look when trying to track down general system errors.

dmesg

This log traditionally lists all of the boot-time kernel logging for a system, along with any other kernel logs related to devices and module loading. Check here to see what sorts of devices the kernel detected at boot time, as well as to track down any errors the kernel might have had when loading a module.

kern.log

Like dmesg, this log contains kernel log output; however, it has the advantage of being timestamped.

Xorg.?.log

The Xorg logs contain very detailed output from Xorg and are numbered according to which display it is running. The default Xorg session runs on display 0, so for the most part you would look through Xorg.0.log to trace down errors in Xorg.

messages

This log contains some kernel log messages along with log output from certain system programs. For instance, gconfd and some other programs log here.

daemon.log

Here, you will find the same daemon output that you would normally find in syslog, but without log output from kernels or other systems that log to syslog.

auth.log

Inside auth.log, you will find information about user authentication including every time a user logs in to the system or executes sudo.

mail.log

If you use your system as an email server, this log will contain information about incoming and outgoing messages, along with any errors.

apache/

If you have installed the Apache web server on your system, /var/log/apache contains access_log, error_log, and all of the other primary Apache logs.

cups/

CUPS is the printing system for Ubuntu. This directory contains all of the different logs for the CUPS service, so look here to debug any issues with printing on the system.

gdm/

GDM is the graphical login manager for GNOME. If you notice any problems with the main login screen, look at the logs in this directory for clues.

View the Logs

Now that you have an idea where to look for information, there are a number of different methods you can use to actually look through the logs. Ubuntu provides a nice graphical tool for log viewing called the System Log Viewer. To start this program, click SystemAdministrationSystem Log. The default window displays a sidebar to the left listing various system logs you have open and a main window listing the contents of the selected log. By default, the System Log Viewer lists only syslog, but you can click LogOpen to add a logfile to the sidebar (Figure 8-14).

Figure 8-14. Default System Log Viewer window

A nice feature of the System Log Viewer is that it splits logs up according to date. Along the bottom-lefthand side of the window is a calendar with days displayed in bold if the current log contains entries for that day. Select a day from the calendar, and it will appear in the main window. You can also collapse a particular day from view in the main window by clicking the arrow next to the date.

You can also view logs the old-fashioned way from a terminal. The program less is a great, no-frills way to open up a logfile and page through it. Just type:

$ less /var/log/syslog

to open up the syslog (you can also replace that file with the path of the file you wish to open). You can use the arrow keys to navigate up and down the file, or you can hit G to scroll down to the very bottom of the file, or g to scroll to the top. To search within the logfile for a keyword, type / followed by the keyword to find. The keyword will be highlighted within the terminal, and you can type n to advance to the next match, or N to go back to the previous match. Hit F to have less continue to update the log as new lines are appended (a lot like running tail -f on the file).

grep is also a very useful tool for logs. grep accepts a pattern as an argument and will look for that pattern within a file. For instance:

$ grep dhclient /var/log/syslog

will return all of the log entries containing the word "dhclient". Grep is particularly useful for Xorg logs, as the warnings and errors in these logs are preceded by WW and EE, respectively. To grep out all of the warnings and errors from Xorg.0.log, type:

$ grep -E '(WW|EE)' /var/log/Xorg.0.log

Note that the -E option turned on extended regular expressions so that you could use a more advanced pattern.

Hack 83. Mount Removable Devices with Persistent Names

Mount your USB drive so it appears as /media/music every time. Mount your FireWire drive so it appears as /media/data every time.

My storage setup on my Ubuntu box at home is a bit unusual. I have an external drive dedicated to music and another external drive that backs up the music drive. I also have an external drive to hold all my personal documents, pictures, and movies, with another external drive to back up that drive. Those four drives are all connected to my desktop via FireWire. Finally, I have an external drive that's a temp drive: when I download a new movie or set of pictures, or rip a new CD, I keep those files on the temp drive until I can properly place them on the music or personal data drive. Unlike the other four, the temp drive uses USB.

Five drives sounds cool, but there's a major annoyance associated with them. When I reboot Ubuntu (a rare occasion, to be sure, but it does happen), their mount points shift around. Sometimes FireWire drive number one gets /media/sdb1, and sometimes /media/sdc1. The same thing happens to the other drives as well, which wreaks havoc with my backup scripts and my attempts to SSH in from other machines, to name just two problems I've experienced. I want persistent naming of those drives, so that FireWire number one is always/media/<something>, FireWire number two is always /media/<somethingelse>, and so on.

After much searching, I found the answer on the Ubuntu forums, at http://ubuntuforums.org/showthread.php?t=91948, and in an excellent overview by Daniel Drake titled "Writing udev rules" (http://www.reactivated.net/writing_udev_rules.html). Basically, Ubuntu includes a technology called udev, which manages dynamic devices. I can tell udev how I want it to label each separate drive when I plug it in, a far more efficient and usable method than the default. Here's how to do so.

You're going to be editing a file found at /etc/udev/rules.d/10-local.rules. Check to see if that file already exists on your hard drive:

$ ls /etc/udev/rules.d/10-local.rules

If it's already there, then you're going to edit it; if it's not there currently, go ahead and create a blank file with the proper name and permissions:

$ sudo touch /etc/udev/rules.d/10-local.rules

$ sudo chmod 777 /etc/udev/rules.d/10-local.rules

At the end of the process described in this hack, the devices will be listed in the 10-local.rules file. The USB drive, for instance, will look like this:

BUS=="usb", SYSFS{serial}=="6R0000065086", NAME{all_partitions}=="temp"

The first partBUS=="usb"is obvious; that's how that drive connects to the machine. The third partNAME{all_partitions}=="temp"makes sense too, since "temp" is the name I want the drive to use in the future as its mount point, instead of /media/sdb1 or whatever else Ubuntu decides to use. But what about that second part? Where is "6R0000065086" coming from?

In order for udev to work, it has to know which specific drive I want to label "temp". When I plug in a drive (or any dynamic device), the drive identifies itself in a whole host of ways. I need to pick a unique data point that I can pass to udev to use for identification. To see what udev sees when you plug in a drive, I run the following command (notice that the path given is the current path that Ubuntu has assigned to the drivein this case, /dev/sda):

$ udevinfo -a -p $(udevinfo -q path -n /dev/sda)

Once I press Enter, an enormous amount of data streams by, looking something like this (this listing has been greatly truncated for length):

device '/sys/block/sda' has major:minor 8:0

looking at class device '/sys/block/sda':

SUBSYSTEM=="block"

SYSFS{dev}=="8:0"

SYSFS{range}=="16"

SYSFS{removable}=="0"

SYSFS{size}=="488397168"

SYSFS{stat}==" 2159 102 16586 1871738 28 17 360 204 0 14257 1871942"

...

looking at the device chain at

'/sys/devices/pci0000:00/0000:00:1e.0/0000:05:09.2/usb4/4-1/4-1.2/4-1.2:1.0':

BUS=="usb"

ID=="4-1.2:1.0"

DRIVER=="usb-storage"

SYSFS{bAlternateSetting}==" 0"

SYSFS{bInterfaceClass}=="08"

SYSFS{bInterfaceNumber}=="00"

SYSFS{bInterfaceProtocol}=="50"

SYSFS{bInterfaceSubClass}=="06"

SYSFS{bNumEndpoints}=="02"

SYSFS{modalias}=="usb:v04B4p6830d0001dc00dsc00dp00ic08isc06ip50"

looking at the device chain at

'/sys/devices/pci0000:00/0000:00:1e.0/0000:05:09.2/usb4/4-1/4-1.2':

BUS=="usb"

ID=="4-1.2"

DRIVER=="usb"

SYSFS{bConfigurationValue}=="1"

SYSFS{bDeviceClass}=="00"

SYSFS{bDeviceProtocol}=="00"

SYSFS{bDeviceSubClass}=="00"

SYSFS{bMaxPower}==" 0mA"

SYSFS{bNumConfigurations}=="1"

SYSFS{bNumInterfaces}==" 1"

SYSFS{bcdDevice}=="0001"

SYSFS{bmAttributes}=="c0"

SYSFS{configuration}==""

SYSFS{devnum}=="3"

SYSFS{idProduct}=="6830"

SYSFS{idVendor}=="04b4"

SYSFS{manufacturer}=="Cypress Semiconductor"

SYSFS{maxchild}=="0"

SYSFS{product}=="USB2.0 Storage Device"

SYSFS{serial}=="6R0000065086"

SYSFS{speed}=="480"

SYSFS{version}==" 2.00"

...

See the line in bold that says SYSFS{serial}=="6R0000065086"? That's the unique serial number for this device, so that's exactly what I'll use in the line I'm adding to 10-local.rules. Notice also that the line BUS=="usb" is also present, in case I don't know how /dev/sda is connected to my box (hey, with five drives, it could happen!). Once again, then, the new line in /etc/udev/rules.d/10-local.rules is:

BUS=="usb", SYSFS{serial}=="6R0000065086", NAME{all_partitions}=="temp"

I need to repeat the process for any other drives as well. Although the next four drives are all connected with FireWire, the process is the same; it's just the listings that are different:

$ udevinfo -a -p $(udevinfo -q path -n /dev/sdb)

device '/sys/block/sdb' has major:minor 8:16

looking at class device '/sys/block/sdb':

SUBSYSTEM=="block"

SYSFS{dev}=="8:16"

SYSFS{range}=="16"

SYSFS{removable}=="0"

SYSFS{size}=="234441648"

SYSFS{stat}=="1166 102 8386 739882 27 16 344 52 0 6575 739934"

...

looking at the device chain at

'/sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/fw-host0/0030e0f4e020dca0/0030e0f4e020dca0-0':

BUS=="ieee1394"

ID=="0030e0f4e020dca0-0"

DRIVER=="sbp2"

SYSFS{address}=="0x0000fffff0000830"

SYSFS{ignore_driver}=="0"

SYSFS{length}=="0"

SYSFS{model_id}=="0x000001"

SYSFS{model_name_kv}=="OXFORD IDE Device LUN 0 "

SYSFS{specifier_id}=="0x00609e"

SYSFS{version}=="0x010483"

looking at the device chain at

'/sys/devices/pci0000:00/0000:00:1e.0/0000:05:04.0/fw-host0/0030e0f4e020dca0':

BUS=="ieee1394"

ID=="0030e0f4e020dca0"

DRIVER=="unknown"

SYSFS{bus_options}=="IRMC_0_ CMC_0_ ISC_0_ BMC_0_ PMC_0_ GEN_0_

LSPD_2_ MAX_REC_64_ MAX_ROM_0_ CYC_CLK_ACC_255_"

SYSFS{capabilities}=="0x0083c0"

SYSFS{guid_vendor_id}=="0x0030e0"

SYSFS{guid}=="0x0030e0f4e020dca0"

SYSFS{nodeid}=="0xffc0"

SYSFS{tlabels_allocations}=="1262"

SYSFS{tlabels_free}=="64"

SYSFS{tlabels_mask}=="0x05965272090596527209"

SYSFS{vendor_id}=="0x0030e0"

SYSFS{vendor_name_kv}=="Oxford Semiconductor Ltd. "

...

In this instance, udevinfo tells me how this drive is connected to my Ubuntu desktop: BUS=="ieee1394" (remember that IEEE1394 is the official name for FireWire). FireWire drives don't include SYSFS{serial} like USB drives do; instead, their unique identifiers are SYSFS{guid}. This particular drive holds my personal files, so I want it to show up as /media/data. Put all that together, and I add this line to /etc/udev/rules.d/10-local.rules:

BUS=="ieee1394", SYSFS{guid}=="0x0030e0f4e020e229", NAME{all_

partitions}=="data"

After performing the same task three more times, /etc/udev/rules.d/10-local.rules looks like this:

BUS=="usb", SYSFS{serial}=="6R0000065086", NAME{all_partitions}=="temp"

BUS=="ieee1394", SYSFS{guid}=="0x0030e0f4e020e229",

NAME{all_partitions}=="data"

BUS=="ieee1394", SYSFS{guid}=="0x0030e0f4e020dca0",

NAME{all_partitions}=="data_copy"

BUS=="ieee1394", SYSFS{guid}=="0x0030e0f4e020912c",

NAME{all_partitions}=="music"

BUS=="ieee1394", SYSFS{guid}=="0x0030e0f4e020c727",

NAME{all_partitions}=="music_copy"

Now I need to test my changes to make sure they'll actually work. First, I need to restart udev, the subsystem that is managing all this craziness:

$ sudo /etc/init.d/udev restart

Now I'll test the rules for the USB external drive to make sure everything will work. To test, I use udevtest. The syntax of the command is:

$ sudo udevtest

sysfs_device_path

subsystem

How do I know what the sysfs_device_path and subsystem are? Look back when I ran udevinfo -a -p $(udevinfo -q path -n /dev/sda) and notice the first things reported back to me:

looking at class device '/sys/block/sda':

SUBSYSTEM=="block"

My udevtest command therefore looks like this:

$ sudo udevtest /sys/block/sda block

The results appear immediately below it (some error messages about other items in udev that aren't important here have been removed):

udevtest.c: looking at device '/block/sda' from subsystem 'block'

...

udevtest.c: opened class_dev->name='sda'

udev_rules.c: configured rule in '/etc/udev/rules.d/10-local.rules:1' applied, 'sda' becomes 'temp'

udev_add.c: creating device node '/dev/temp', major = '8', minor = '0', mode = '0640', uid = '0', gid = '46'

udev_add.c: creating device partition nodes '/dev/temp[1-15]'

Excellent! Now I know that udev would in fact create a device name /dev/tempand therefore a mount point at /media/tempinstead of /dev/sda and /media/sda, and will always use that in the future. By the way, don't worryeven though the output says it was creating nodes, it really wasn't. It was just showing me what it would do when it runs for real.

So I know the USB drive works, but what about one of the FireWire drives?

$ sudo udevtest /sys/block/sdd block

Here are the results for that drive (again, some unnecessary data has been removed):

udevtest.c: looking at device '/block/sdd' from subsystem 'block'

...

udevtest.c: opened class_dev->name='sdd'

udev_rules.c: configured rule in '/etc/udev/rules.d/10-local.rules:5' applied, 'sdd' becomes 'music_copy'

udev_add.c: creating device node '/dev/music_copy', major = '8', minor = '48', mode = '0640', uid = '0', gid = '46'

udev_add.c: creating device partition nodes '/dev/music_copy[1-15]'

Exactly what I wanted, so it looks like everything will work as intended. Now I restart the machine, and lo and behold, my drives appear, just as I wanted them:

/media/temp

/media/data

/media/data_copy

/media/music

/media/music_copy

Of course, there's nothing stopping you from setting up udev so that your digital camera always shows up as /media/camera. Or so your iRiver music jukebox consistently mounts as /media/iriver. And so on. Basically, if your system sees it as a dynamic device, you can use udev to map it to a specific mount point. Thanks to udev, Ubuntu just got a lot easier to use.

Scott Granneman

Hack 84. Mount Remote Directories Securely and Easily

If you can access it via SSH, you can mount it on your filesystem, giving you secure access to your favorite stuff.

Samba has transformed how Penguinistas interact on networks with Windows machines and even other Linux boxes. In particular, I often use smbfs (short for "Samba filesystem") to mount a Samba share on my box for easy access. For example, I have two computers on my network: a server named sophocles and a laptop named euripides. A huge collection of music can be found on sophocles at /var/music, but I want easy access to those goodies from euripides. Using Samba on sophocles, I share /var/music, and using smbfs on euripides, I mount /var/music on my server sophocles to /home/scott/tunes on my laptop euripides. By doing so, it appears to me while I'm using euripides as though tunes is just another local folder on my laptop, so I can read files from it, save to it, and do anything else I could if that folder were in fact on my machine.

This is great, except that there are some issues. Setting up Samba can be a royal pain, so any time I can use something simpler, I jump at the chance. Second, Samba shares aren't secure by default. Call me paranoid, but I don't like anything flowing over a network that isn't encrypted. Yes, it's possible to tunnel Samba using SSH, but that just adds more time and trouble on top of the royal pain that Samba sometimes causes. Finally, Samba was designed for LANs, not the wild and woolly Internet, so accessing shares remotely is out of the question (yes, there are ways to do it, but it's just not a good idea on today's Net...and it causes yet more complication!).

But I'm here to tell you that there's a better way: sshfs, the SSH filesystem. Instead of Samba, it uses SSH. All the problems I listed previously are obviated when you switch to SSH. SSH is a breeze to set up and use. All SSH traffic is encrypted, hence the name Secure SHell. And SSH was designed for use on LANs as well as the big, bad Internet, so it's a great choice no matter where you are.

To go back to my original example, I can SSH from euripides to sophocles (and vice versa, for that matter), so I now use sshfs to mount /var/music on sophocles to /home/scott/tuneson euripides. It's easy, it's encrypted, and if this is a connection I'm going to need all the time, I can set things up in /etc/fstab to automate the whole process. Ready to try it out?

Before doing anything else, make sure that you can SSH from the client to the server:

$ ssh scott@sophocles

Linux sophocles 2.6.15-18-386 #1 PREEMPT Thu Mar 9 14:41:49 UTC 2006 i686 GNU/Linux

The programs included with the Ubuntu system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by

applicable law.

Last login: Sun Mar 12 13:59:45 2006

$ exit

If you can't SSH in, you need to set that up:

· On an Ubuntu server, use sudo apt-get install ssh.

· On Windows, install Cygwin, available from http://www.cygwin.com, and install OpenSSH as a service.

· On the Mac, enable Remote Login in System PreferencesSharingServices.

If you can successfully SSH in to the server, you're finished with that machine (see how much simpler this is than it would be if you were using Samba?).

Now you need to focus on the local computer. Make sure you've enabled the universe archive in your /etc/apt/sources.list file, as detailed in "Modify the List of Package Repositories" [Hack #60]. Once you've done so, install sshfs:

$ sudo apt-get install sshfs

You'll be told that apt is going to have to install some additional packages in addition to sshfs (fuse-utils and libfuse2), so go ahead and approve apt's request.

When apt finishes installing your new software, it's time to fix some permissions so that normal users can mount and unmount using sshfs. If you leave out this step, mounting and unmounting will require the use of sudo and knowledge of the root password, which you probably don't want. Better to go ahead and run this command, which grants all users on your system execute permission for the fusermount command, used by sshfs to perform its magic:

$ sudo chmod +x /usr/bin/fusermount

Now you need to make the mount point you plan to use on your local machine. In my case, this would be /home/scott/tunes:

$ mkdir /home/scott/tunes

In order for sshfs to work, you have to load the fuse module. For now, go ahead and run this command:

$ sudo modprobe fuse

I know you don't want to have to run that command every time you want to mount with sshfs, so use the following command to tell Ubuntu to automatically load the fuse module when you boot:

$ sudo sh -c "echo 'fuse' >> /etc/modules"

Now for the final step: mount /var/music on sophocles to /home/scott/tunes on euripides. The syntax of the command is as follows:

$ sshfs

user@

[

IP

|

HOSTNAME of remote machine

]

:/shared/path /local/mount/point

If your username is the same on the remote and local machines, you can leave it off. DNS will work either on your LAN, if you have an entry for the server in your /etc/hosts file, or over the Net to a machine with a registered address. Otherwise, use the computer's IP address. If you don't specify a path after the colon, sshfs mounts the home directory for the SSH user, which may be all that you want. In my case, my username is the same on both boxes, I have an entry for sophocles in the /etc/hosts file, and I want to mount a specific directory on the computer, so my command looks like this:

$ sshfs sophocles:/var/music /home/scott/tunes

Now I can open my favorite music player, point it to /home/scott/tunes, and start enjoying jazz, rock, country, or whatever it is that floats my boat. Remember, it's all secure, so I don't have to worry about anyone sniffing packets and finding out what's moving around on my network.

To unmount, you have a choice. You can use the fusermount command with the -u option (for "unmount"):

$ fusermount -u /home/scott/tunes

Or it might be simpler just to use the good ol' umount command:

$ sudo umount /home/scott/tunes

If I knew that I wanted to make my connection to sophocles whenever I booted, I could add the following to /etc/fstab:

sophocles:/var/music /home/scott/tunes fuse defaults 0 0

The first three items are different from the usual fstab entries, with the first two exactly like the sshfs command you just used, and fuse indicating to your computer the kind of filesystem it's going to mount. If you want to change the other entries and you know what you're doing (run man fstab to learn more), go right ahead.

Once you discover sshfs, you're going to use it all the time. There's no reason not to: it works beautifully, and the added benefit of security is icing on the cake. Thank you, sshfs!

Scott Granneman

Hack 85. Make Videos of Your Tech-Support Questions

Have a friend or relative with tech-support troubles? Show him how to record a movie of his Ubuntu problems and send it to you, you ever-helpful, all-knowing Ubuntu guru, you.

We've all been there: Uncle Gussy is on the phone to you trying to explain a computer problem he's having, but his description isn't helping in the slightest: "Welllll, I click on the thingie, and then this doohickey opens, so I click on that, and then the computer does something weird." Gee, thanks, Uncle Gussy!

If you were smart enough to set up Uncle Gussy with Ubuntu (and being a good niece or nephew, that's what you did, right?), you could just tell him to use Istanbul to record what he's doing on his computer into a patent-free, open source Ogg Theora video file. Uncle Gussy could then email you the file, which would vastly simplify your job as family tech support.

When you first set up Uncle Gussy's Ubuntu box, make sure Istanbul is installed. First look for it, to see if it's already on the machine:

$ whereis istanbul

istanbul:

Nope. OK, you need to put it on there. Make sure you have the universe repository enabled [Hack #60], and then install Istanbul:

$ sudo apt-get install istanbul

You may find that apt wants to install several other packages as well, chiefly related to gstreamer, so go ahead and indicate your approval. Once apt has finished its business, you need to start Istanbul so you can configure it, so select ApplicationsSound & VideoIstanbul Desktop Session Recorder. You'll know Istanbul has loaded when you see a large red dot to the left of the clock on the top panel, as shown in Figure 8-15.

Figure 8-15. Istanbul on the top-edge panel

Right-click on Istanbul's icon and choose Preferences, which brings up the window shown in Figure 8-16.

Figure 8-16. Istanbul Preferences

It's a good idea to check the box next to "Encode after stopping record (uses less CPU)" for the reason given in the setting's name. Also, to reduce file size, change Frames Per Second from 10.0 to 1.0. Of course, this also means that Uncle Gussy will have to move slowly and deliberately as he records his movie, but that can only help you when you're viewing it. As for the Network Stream section, that would more than likely be far too complicated for the likes of Uncle Gussy, so just leave those prefs alone. Click Close, head on home, and then wait patiently until the next phone call from Uncle Gussy.

The Ogg Theora encoder loves to gobble up CPU cycles, so whatever you can do to reduce that load would be good. Checking the box next to "Encode after stopping record" was a good start, but you should also walk Uncle Gussy through temporarily changing his screen resolution to 800x600 if at all possible.

Now for Uncle Gussy's debut as Cecil B. DeMille. Tell him to start Istanbul by going to ApplicationsSound & VideoIstanbul Desktop Session Recorder. To start recording, he clicks on the large red circle, which changes to a stop box to indicate that recording is in progress. He then performs the actions he wants you to watch and decipher. Tell him to go slow and make all his actions deliberate, so that you can easily comprehend what he's doing.

When he's finished, he clicks on the stop box, and the icon changes once again, to a gray icon that GNOME uses to represent a hard drive, to let him know that Istanbul is encoding. If your uncle has a fast computer, or his movie is short, he may not even see that last icon, in which case he'll be back to the red circle. If Uncle Gussy is done with Istanbul, he can close the program by right-clicking on the icon and choosing Quit.

There should now be a new file named desktop-recording.ogg in Uncle Gussy's home directory. Tell him to email that to you, and shortly thereafter you'll be able to see exactly what's going on with his Ubuntu machine. It's the next best thing to sitting next to him while he screws upI mean, useshis computer.

You may find that Istanbul is still a bit buggy, but it's getting a lot of attention from developers, so keep an eye on it. More and more, developers of various Linux distros are going to rely on Istanbul when users complain about various bugs and problems.

Scott Granneman

Hack 86. Synchronize Files Across Machines

If you switch between multiple machines, it's a real hassle to keep track of which system has the latest version of a file. Use the free Unison File Synchronizer to keep all your systems in sync, even Windows and Mac machines.

There's an old saying that everyone should take to heart: "There are two kinds of computer users: those who have lost data and those who are about to." I belong in that first camp, but it only took one nasty experience with an accidentally deleted term paper to force me to resolve never to let that happen again. That night long ago, when I lost the fruits of my labors analyzing the great film Raising Arizona, taught me the hard way that backing up is essential. And while traditional backups [Hack #79] are vital, keeping two or more machines in sync not only gives you some extra security, but it also makes it easy to switch from system to system without having to remember which one has the latest copy of a file.

If a user new to Linux starts asking more experienced users about a good way to sync his data, he will soon hear about a wonderful tool named rsync. rsync was developed by Andrew "Tridge" Tridgell, the same man behind Samba; in fact, Tridge has stated that he believes he'll be remembered through posterity for rsync far more than for Samba, and he just may be right. However, rsync, while absolutely excellent, is not necessarily the best software to use for syncing two repositories that you're actively working with.

The Problem with rsync

rsync is truly awesome software, and it really is worth your time to check it out. However, I don't use rsync for my day-to-day sync needs. Instead, I use Unison, which uses rsync as its base. Why Unison instead of rsync? Because Unison synchronizes in two directions at one time, while rsync goes only in one direction.

Let me give another example, from my own setup. At home I have a desktop machine, but I also use a laptop. I work a lot outside of my home, so my laptop travels with me constantly; when I'm home, though, I sometimes leave my laptop in my backpack and use my desktop instead. I have a large amount of data that I need to have available to me at all times: web pages I've read, instruction manuals, articles I'm working on, photos, and so on. All told, it's about 10 GB worth of stuff. I keep all of this work on my desktop, and I keep a mirror of the same thing on my laptop.

Obviously, I need to keep all 10 GB in sync between my two machines. If I delete a file on the desktop, I need to delete it on the laptop. If I change a file on the laptop, it wouldn't do to open the original, unchanged file on the desktop and make changes there. That way lies madness.

So why can't I use rsync? Because my goal is to synchronize between two machines, each of which acts as a source and a destination at the same time. Here's one example: on my laptop, I move the file widgets from the directory foo to the directory bar. On the desktop, however, widgets is still in foo. If I run rsync with the laptop as the source and the desktop as the destination, so far so good: widgets is now in bar on both the laptop and the desktop. However, widgets is still in foo on the desktop. The next time I run rsync with the desktop as the source and the laptop as the destination, widgets gets copied over from foo on the desktop back into foo on the laptop, leaving me with widgets in both foo and bar, which produces a real mess.

Of course, I could run rsync with the --delete option set, so that files deleted on the source are also deleted on the destination, but that can be a very dangerous practice. What if I delete foo on the laptop but change foo on the desktop? If I run rsync with the laptop as source, then foo is deleted on the desktop, which is not what I want. Instead, I have to remember to run rsync with the desktop as source so that the changed foo is copied over to the laptop. But what if there were files I changed on the laptop and deleted on the desktop? Then I need to run rsync with the laptop as source...and on and on, ad infinitum, back and forth.

Unison solves this problem. You define a source and a destination, and then you run Unison. After some length of time, Unison starts asking you questions about the files it has found: copy this file from laptop to desktop? Copy this other file from desktop to laptop? Delete this file on the laptop since it was deleted on the desktop? You can accept Unison's guesses, specify a direction for the copy to go, or even tell Unison to skip the file entirely until another day. With Unison, synchronizing two directories, each on a different machine, becomes a far simpler task. And, as icing on the cake, Unison is cross-platform, as it runs on Linux, Mac OS X (with a few caveats; see "Further Information About Unison" at the end of this hack), Unix, and Windows machines.

Synchronize Files on Two Machines with Unison Using SSH

Installing Unison is easy as pie (however, make sure you have the universe repository enabled [Hack #60]). Just run the following command, and you're done:

$ sudo apt-get install unison unison-gtk

Now you have both the basic, command-line-only Unison program and the GTK-based GUI.

It's time to sync a group of files with another copy of those same files on another machine. For added safety, I'm going to use SSH for the transport, a capability built into Unison. Unfortunately, Unison doesn't create entries in the GNOME Applications menu or the KDE K menu, so you'll need to add those yourself. In the meantime, you can type unison-gtk on the command line. When the program opens, you'll see Figure 8-17, which asks you to either select or create a profile.

Figure 8-17. Select or create a Unison profile

Click "Create new profile," and you're asked to give a name for the new profile. Since I'm going to sync up my folder that contains the work I've done on this book, I'm going to cleverly name this profile ubuntu_hacks and click OK. Now both profiles, default and ubuntu_hacks, appear. I double-click on ubuntu_hacks, and a small window opens, shown in Figure 8-18, asking me to enter the local directory that I want to sync.

Figure 8-18. Select the local directory you want to sync

Since I don't like to type when I can avoid it, I instead click Browse and navigate to the folder:

/home/rsgranne/documents/clientele/aaa_current/OReilly/ubuntu_hacks

I click OK, and another window opens, shown in Figure 8-19. This one wants to know about the remote folder.

Figure 8-19. Select the remote directory you want to sync

Notice that I can sync to a local folder (which could also be a folder accessed through Samba or NFS that I've mounted on this machine), or to another machine via SSH, RSH, or a socket. In this case, I'm using SSH, so I enter the address of the remote hostin this case, 192.168.0.14and the username, which is rsgranne. Finally, I type in the remote directory I want to sync:

/media/data/zzz_rsg/documents/clientele/aaa_current/OReilly/ubuntu_hacks

I click OK, and since this is the first time I've synched these two folders, Unison presents me with a warning that it is going to have to create archives for those two folders from scratch. No problem, so I click OK. The main Unison window opens, and Unison goes to work, comparing the two directories (and all their subdirectories as well, since the program automatically recurses). Be patient during this process. Unison may appear to be locked up, but it's just taking its time to digest all the files and folders. The more data, the longer Unison will take. Eventually, Figure 8-20 appears.

Figure 8-20. You can sync files in a number of directions

Unison's interface is pretty self-explanatory, with the arrows showing the direction in which copying will occur. If I want to switch the direction in which a file is copied, I simply select it and then click the "Right to Left" or "Left to Right" button. If I want Unison to ignore a file, I click Skip. If I want to invoke a diff-type program to actually compare the contents of the two files (this obviously works only with text files and the like) and then selectively merge the differences, I click Diff and then Merge. When I'm sure that everything is the way I want it, I click the Go button to tell Unison to copy files and folders. There are actually lots of other options available through the Actions menu, so it's a good idea to check it out.

Tweaking Your Profiles

When I first started Unison, I had to create a profile, which keeps track of the directories I want to compare and the transport method, among other configurations. Profiles live in a hidden directory (.unison), which itself is in your home directory, and they're just text files that end with the .prf extension. Before I finish looking at Unison, I want to mention a few other tweaks that you should add manually to those files in order to make your sync processes work a little more smoothly.

In order to add these tweaks, you're going to need to open your profiles with your favorite text editor. In my case, I'm going to open and change ubuntu_hacks.prf, located in /home/rsgranne/.unison.

To start, I'll add the following to the file (it doesn't matter where you put these preferences, but I like to put them at the top of the file, after my two roots and before any "ignore" lines):

times = true

If you use Unison for any length of time, you're going to notice that a lot of files need their "props" synchronized. Complaining about "props" is just Unison's way of telling you that the date- and timestamps for your fileswhich indicate when the files were modifiedare not exact. By setting times to TRue, you are copying over not just the contents of the files, but their modification dates and times as well, which will vastly reduce any "props" complaints that you'll see.

In a similar vein, you may want to add the following:

owner = true

group = true

When you add these two settings to your profile, you order Unison to synchronize the owners and groups of files, along with their contents, which can really help to keep things straight as you move files around between machines.

If you want to synchronize users and groups by the real numbers that your Ubuntu system uses instead of the names that we humans use (in other words, by 1000 instead of rsgranne; if you don't know what I'm talking about, take a look at your /etc/passwd file and run the command man 5 passwd), then add this preference to your profile as well:

numericids = true

If you're using Unison to sync only Linux boxes, then you can skip this next preference. But if you're involving Windows machines in your syncsay, from a Windows workstation to a Linux machine, or from a Windows PC to a Linux workstationthen you probably want to use fastcheck (but read the following paragraphs carefully):

fastcheck = yes

You'll probably want to use fastcheck = yes most of the time, but every once in a while you'll want to comment out that line with a pound sign in front of it (#), run Unison, and then remove the pound sign. Let me explain why.

You may already have asked yourself a pretty important question: how the heck does Unison know that a file has changed? For files on a Linux box, Unison looks at their inode numbers and modtimes; if either of those is different, then Unison thinks that the file has changed. Windows boxes, of course, don't have inode numbers, so Unison's default for Windows is to scan the contents of each file to see if anything has changed. This is certainly a safe method, but if you have a lot of files to scan on a Windows machine, it's going to take a long, long time. Failing to set the fastcheck preference then, or setting it to fastcheck = default, results in this behavior, which probably is not what you want to do.

If you set fastcheck to yes, however, then Unison acts differently on your Windows machines: it looks at the file modification times instead of scanning the contents of each file (Linux files still get the same treatment: their inode numbers and modtimes are checked). This results in a much faster scan of your Windows machine, but in very rare circumstances (a file has been changed, but you have somehow managed not to change the create time, the modification time, and the length of the filepretty hard to do), Unison may refuse to make an update of some of the files on the Windows machine.

Now, practically speaking, you don't have to worry about this very much. First, you'd have to really jump through hoops to fool Unison, and second, even if Unison thought that it should possibly overwrite such a file, it won't, as the Unison manual explains: "Unison will never overwrite such an update with a change from the other replica, since it always does a safe check for updates just before propagating a change."

Since this is the case, I would recommend leaving fastcheck set to yes most of the time, but, every once in a while, comment out that preference with a pound sign, so that the full contents of your Windows files will be checked on the next run. Then, after you've satisfied yourself that things are copacetic, uncomment fastcheck = yes and go back to the faster method.

Changing the Location of Your Logfile Directory

When you run Unison, the program generates a logfile, which can be a very useful thing. The default, however, is to create this logfilenamed unison.login your home directory, which I find annoying since I try to keep my home directory neat and tidy. If you don't mind this, then leave the logfile preference alone. But if you, like me, want to change the location in which the logfile goes, then set this preference:

logfile = $HOME/.unison/unison.log

I like the logfile to go into the hidden .unison directory, but you can change this any way you like.

Further Information About Unison

The main site for all things Unison can be found at http://www.cis.upenn.edu/~bcpierce/unison/. Hopefully, it'll get a cooler URL someday.

On the main Unison site, you can read the very complete and very informative Unison documentation (if you really get into Unison, then you should look into running it sans GUI; the documentation will tell you everything you need to know, although the GUI actually provides a good way to learn the basics). If that doesn't help, there is a good FAQ that covers several important questions, especially one that I was wondering myself: "Does Unison work on Mac OS X?" Be sure to read that answer in full before trying to use Unison on that OS!

For further support, there are a few listservs you can joinor just search, if you don't want to joinat Yahoo! Groups. Unison-users is just that: a list for users of Unison. It currently has over 600 members, and they produce in the neighborhood of 100 messages a month, so it's not too overwhelming. You can get more information about the list at http://groups.yahoo.com/group/unison-users/. If you're a developer, then you might want to check out Unison-hackers at http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers. If all you want are the announcements when a new version is released, then you should subscribe to the low-volume (one message per month) Unison-announce, which you can find at http://groups.yahoo.com/group/unison-announceor just send a blank email to unison-announce-subscribe@groups.yahoo.com.

Finally, Open magazine, a "weekly e-zine for Linux and Open Source computing in the enterprise," has a nice article about Unison available at http://www.open-mag.com/features/Vol_53/synch/synch.htm. It focuses on getting Unison to work with Windows as well as Linux, and includes some technical details about fail-safe provisions that Unison provides.

Scott Granneman