Backup and Recovery - Network Services - Running Linux, 5th Edition (2009)

Running Linux, 5th Edition (2009)

Part IV. Network Services

Chapter 27. Backup and Recovery

After reading the previous three chapters, you now have all the skills you need to start using your system. But eventually you'll want the information in this chapter too. Some of the activities, such as making backup tapes, are important habits to develop. You may also find it useful to have access to files and programs on Windows. Finally, we'll help you handle events that you hope will never happen, but sometimes do—system panics and corruption.

Making Backups

Making backups of your system is an important way to protect yourself from data corruption or loss in case you have problems with your hardware, or you make a mistake such as deleting important files inadvertently. During your experiences with Linux, you're likely to make quite a few customizations to the system that can't be restored by simply reinstalling from your original installation media. However, if you happen to have your original Linux CD-ROM or DVD-ROM handy, it may not be necessary to back up your entire system. Your original installation media already serve as an excellent backup.

Under Linux, as with any Unix-like system, you can make mistakes while logged in as root that would make it impossible to boot the system or log in later. Many newcomers approach such a problem by reinstalling the system entirely from backup, or worse, from scratch. This is seldom, if ever, necessary. In "What to Do in an Emergency," later in this chapter, we talk about what to do in these cases.

If you do experience data loss, it is sometimes possible to recover that data using the filesystem maintenance tools described in "Checking and Repairing Filesystems" in Chapter 10. Unlike some other operating systems, however, it's generally not possible to "undelete" a file that has been removed by rm or overwritten by a careless cp or mv command (for example, copying one file over another destroys the file to which you're copying). In these extreme cases, backups are key to recovering from problems.

Backups are usually made to tape, floppy, CD-R(W), or DVD-R(W). None of these media is 100% reliable, although tape, CD-R(W), and DVD-R(W) are more dependable than floppies in the long term. These days, with the cost of hard disks plummeting and the capacity increasing, backing up to a hard disk is also an option.

Many tools are available to help you make backups. In the simplest case, you can use a combination of gzip (or bzip2) and tar to back up files from your hard drive to removable media. This is the best method to use when you make only occasional backups—no more often than, say, once a month.

If you have numerous users on your system or you make frequent changes to the system configuration, it makes more sense to employ an incremental backup scheme. Under such a scheme, you would make a "full backup" of the system only about once a month. Then, every week, you would back up only those files that changed in the last week. Likewise, each night, you could back up just those files that changed over the previous 24 hours. There are several tools to aid you in this type of backup.

The idea behind an incremental backup is that it is more efficient to make backups in small steps; you use fewer tapes or CDs, and the weekly and nightly backups are shorter and easier to run. With this method, you have a backup that is at most a day old. If you were to, say, accidentally delete your entire system, you would restore it from backup in the following manner:

1. Restore from the most recent monthly backup. For instance, if you wiped the system on July 17, you would restore the July 1 full backup. Your system now reflects the state of files when the July 1 backup was made.

2. Restore from each weekly backup made so far this month. In our case, we could restore from the two weekly backups from July 7 and 14. Restoring each weekly backup updates all the files that changed during that week.

3. Restore from each daily backup during the last week—that is, since the last weekly backup. In this case, we would restore the daily backups from July 15 and 16. The system now looks as it did when the daily backup was taken on July 16; no more than a day's worth of files have been lost.

Depending on the size of your system, the full monthly backup might require 4 GB or more of backup storage—often not more than one DVD-R(W) or tape, but quite a few Zip disks. However, the weekly and daily backups would generally require much less storage space . Depending on how your system is used, you might decide to make the weekly backup on Sunday night and not bother with daily backups for the weekend.

One important characteristic that backups should (usually) have is the ability to select individual files from the backup for restoration. This way, if you accidentally delete a single file or group of files, you can simply restore those files without having to do a full system restoration. Depending on how you make backups , however, this task will be either very easy or painfully difficult.

It's also highly desirable to keep the backup media physically separate from the computer. If you choose to back up to hard disk, consider an external USB, FireWire, or SCSI drive. USB and FireWire are particularly nice because they can be easily plugged into or removed from the system as needed. If you choose to back up to a second, internal hard disk, it would be wise to at least keep the disk unmounted when it's not in use, so that if you were to accidentally delete one or more filesystems, your backup would be spared. An important decision you need to make is evaluating the relative importance of your data's safety and recoverability versus the cost and convenience of the backup media you choose, as well as how you use it.

In this section, we talk about the use of tar, gzip, and a few related tools for making backups to CD and tape. We even cover the use of tape drives, as well as CD-R in the bargain. These tools allow you to make backups more or less "by hand"; you can automate the process by writing shell scripts and even schedule your backups to run automatically during the night using cron. All you have to do is flip tapes. Other software packages provide a nice menu-driven interface for creating backups, restoring specific files from backup, and so forth. Many of these packages are, in fact, nice frontends to tar and gzip. You can decide for yourself what kind of backup system suits you best.

Simple Backups

The simplest means of making a backup is to use tar to archive all the files on the system or only those files in a set of specific directories. Before you do this, however, you need to decide what files to back up. Do you need to back up every file on the system? This is rarely necessary, especially if you have your original installation disks or CD-ROM. If you have made important changes to the system, but everything else is just the way it was found on your installation media, you could get by with only archiving those files you have made changes to. Over time, however, it is difficult to keep track of such changes.

In general, you will be making changes to the system configuration files in /etc. There are other configuration files as well, and it can't hurt to archive directories such as /usr/lib and /etc/X11 (which contains the XFree86 configuration files, as we saw in "Installing" in Chapter 16).

You should also back up your kernel sources (if you have upgraded or built your own kernel); these are found in /usr/src/linux.

During your Linux adventures it's a good idea to keep notes on what features of the system you've made changes to so that you can make intelligent choices when making backups . If you're truly paranoid, go ahead and back up the whole system; that can't hurt, but the cost of backup media might.

Of course, you should also back up the home directories for each user on the system; these are generally found in /home. If you have your system configured to receive electronic mail (see "The Postfix MTA" in Chapter 23), you might want to back up the incoming mail files for each user. Many people tend to keep old and "important" electronic mail in their incoming mail spool, and it's not difficult to accidentally corrupt one of these files through a mailer error or other mistake. These files are usually found in /var/spool/mail. Of course, this applies only if you are using the local mail system, not if you access mail directly via POP3 or IMAP.

Backing up to tape

Assuming you know what files or directories to back up, you're ready to roll. You can use the tar command directly, as we saw in "Using tar" in Chapter 12, to make a backup. For example, the command:

tar cvf /dev/qft0 /usr/src /etc /home

archives all the files from /usr/src, /etc, and /home to /dev/qft0. /dev/qft0 is the first "floppy-tape" device—that is, a tape drive that hangs off of the floppy controller. Many popular tape drives for the PC use this interface. If you have a SCSI tape drive, the device names are /dev/st0, /dev/st1, and so on, based on the drive number. Those tape drives with another type of interface have their own device names; you can determine these by looking at the documentation for the device driver in the kernel.

You can then read the archive back from the tape using a command such as:

tar xvf /dev/qft0

This is exactly as if you were dealing with a tar file on disk, as discussed in "Archive and Compression Utilities" in Chapter 12.

When you use the tape drive, the tape is seen as a stream that may be read from or written to in one direction only. Once tar is done, the tape device will be closed, and the tape will rewind. You don't create a filesystem on a tape, nor do you mount it or attempt to access the data on it as files. You simply treat the tape device itself as a single "file" from which to create or extract archives.

Be sure your tapes are formatted before you use them. This ensures that the beginning-of-tape marker and bad-blocks information have been written to the tape. For formatting QIC-80 tapes (those used with floppy-tape drivers), you can use a tool called ftformat that is either already included with your distribution or can be downloaded from as part of the ftape package.

Creating one tar file per tape might be wasteful if the archive requires only a fraction of the capacity of the tape. To place more than one file on a tape, you must first prevent the tape from rewinding after each use, and you must have a way to position the tape to the next file marker, for bothtar file creation and extraction.

The way to do this is to use the nonrewinding tape devices , which are named /dev/nqft0, /dev/nqft1, and so on for floppy-tape drivers, and /dev/nst0, /dev/nst1, and so on for SCSI tapes. When this device is used for reading or writing, the tape will not be rewound when the device is closed (that is, once tar has completed). You can then use tar again to add another archive to the tape. The two tar files on the tape won't have anything to do with each other. Of course, if you later overwrite the first tar file, you may overwrite the second file or leave an undesirable gap between the first and second files (which may be interpreted as garbage). In general, don't attempt to replace just one file on a tape that has multiple files on it.

Using the nonrewinding tape device, you can add as many files to the tape as space permits. To rewind the tape after use, use the mt command. mt is a general-purpose command that performs a number of functions with the tape drive. The mt command, although very useful and powerful, is also fairly complicated. There's a lot to keep track of to locate a particular record on the tape, and it's easy to get confused. If you're particularly motivated to use your tapes as efficiently as possible, read up on mt; the manpage is quite concise. We include a few examples here. The command:

mt /dev/nqft0 rewind

rewinds the tape in the first floppy-tape device.

Similarly, the command:

mt /dev/nqft0 reten

retensions the tape by winding it to the end and then rewinding it.

When reading files on a multiple-file tape, you must use the nonrewinding tape device with tar and the mt command to position the tape to the appropriate file.

For example, to skip to the next file on the tape, use the command:

mt /dev/nqft0 fsf 1

This skips over one file on the tape. Similarly, to skip over two files, use:

mt /dev/nqft0 fsf 2


mt device fsf 1

to move to the next file.

Be sure to use the appropriate nonrewinding tape device with mt. Note that this command does not move to "file number two" on the tape; it skips over the next two files based on the current tape position. Just use mt to rewind the tape if you're not sure where the tape is currently positioned. You can also skip back; see the mt(1) manual page for a complete list of options.

Backing up to CD-R

You can back up your files to recordable CD perhaps more easily than to tape. Blank CDs are very inexpensive, widely available, readable on just about any system, and much easier to transport and store than tapes. In this section, we show you the basics of writing backups to CD-R, as well as a couple of tricks. Almost all of the techniques that work for CD-Rs work equally well for the various flavors of recordable DVDs that are available.

By far the most common way to write data to a CD is to create a CD image file on your hard disk, then burn that to CD. This is easy to do, but has one slight disadvantage: you need at least 650 or 700 MB of free disk space to create a full-sized CD image. On modern systems that generally shouldn't be a problem.

CD-ROMs use the ISO 9660 filesystem standard, which can be mounted and read on just about any operating system in common use today. The program mkisofs is a full-featured and robust tool for creating such filesystems, which can be used in a number of ways, including burning them to CD-R. The actual burning can be done with cdrecord. Both of these programs are usually included with most Linux systems.

Here's how to create an ISO 9660 image and burn it to CD. Let's say you have a directory called /data that you want to put on CD. Enter:

# mkisofs -T -r -o /tmp/mycd.iso /data

#cdrecord -v -eject -fs=4M speed=8 dev=0,0,0 /tmp/mycd.iso

Some of the parameters of cdrecord are system-specific. You can run cdrecord -scanbus to search for the CD burner on your machine. On the machine used for testing the material in this section, the CD burner shows up as device 0,0,0. Even though the author has a 52 × burner, he still chooses to record CDs at only 8 ×, to make sure he doesn't underflow the drive and make a bad disk. Experiment with your hardware and determine what works—you may or may not be able to burn reliably at higher speeds.

A slick, if somewhat less reliable, way to create a CD without writing an image file first is to simply pipe the output of mkisofs directly to cdrecord:

mkisofs -T -r /data | cdrecord -v -eject -fs=4M speed=8 dev=0,0,0 -

That's not the only possible optimization. If, for some reason, you wanted to treat a CD like a tape, you could skip creating an ISO 9660 filesystem, and just write a tar file directly to a CD. This won't be mountable as a normal CD, and you won't be able to put it in a Windows system, but if you prefer this, it works:

tar -czf - /data | cdrecord -v -eject -fs=4M speed=8 dev=0,0,0 -

It is important to note that although CD burners are much better than they were a few years ago, it's still quite possible to produce a useless disk if there is anything more than a brief interruption in the flow of data from the source to the burner. These problems are even more apparent when burning from a pipeline as in the last two examples. For this reason, we urge you to check your backups after making them!

In addition to cdrecord, tar, and mkisofs, there are a large number of other programs available on the Web that provide an easy-to-use frontend for creating backups. Some of them are able to span multiple CDs, or manage a rotation of CD-RW disks. If you find that creating CD backups as we've described here doesn't fit your needs, chances are someone has created another program that will work for you.

Backing up to hard disks

As hard disks get bigger and cheaper, one problem is that backup media often don't keep up. Now that 500-GB hard disks are available and affordable by normal people, it may have occurred to you that the only thing you can back a disk that big up to is...another disk that big!

Just about all of the techniques you can use for media in general can apply to hard disks, but there are a few special considerations as well.

If you have a disk mounted at /data, and you want to back it up to a second hard disk (of equal or greater size) mounted at /backup, you could do a tar and un-tar pipeline, like this:

cd / ; tar -cvf - /data | (cd /backup ; tar -xf -)

If you have room on your backup disk, you can use the remaining space to store incremental backups using the techniques described elsewhere in this chapter. The nice thing about hard disk backups is that you can create any kind of directory structure that makes sense to you. You could have a disk mounted at /backup, with subdirectories /backup/full and /backup/incremental, or any other scheme you choose.

With a combination of find, cron, tar, and gzip, you could create a fairly small but powerful script that would mount your backup hard disk, tar up the files that have changed since the last time your backup ran, delete the backups older than the last full backup, and unmount the backup disk.

To compress or not to compress?

There are good arguments both for and against compression of tar archives when making backups. The overall problem is that neither tar nor the compression tools gzip and bzip2 are particularly fault-tolerant, no matter how convenient they are. Although compression using gzip or bzip2 can greatly reduce the amount of backup media required to store an archive, compressing entire tar files as they are written to CD-R or tape makes the backup prone to complete loss if one block of the archive is corrupted, say, through a media error (not uncommon in the case of CD-Rs and tapes). Most compression algorithms, gzip and bzip2 included, depend on the coherency of data across many bytes in order to achieve compression. If any data within a compressed archive is corrupt, gunzip may not be able to uncompress the file from that point on, making it completely unreadable to tar.

This is much worse than if the tar file were uncompressed on the tape. Although tar doesn't provide much protection against data corruption within an archive, if there is minimal corruption within a tar file, you can usually recover most of the archived files with little trouble, or at least those files up until the corruption occurs. Although far from perfect, it's better than losing your entire backup.

A better solution is to use an archiving tool other than tar to make backups. Several options are available. cpio is an archiving utility that packs files together, similar in fashion to tar. However, because of the simpler storage method used by cpio, it recovers cleanly from data corruption in an archive. (It still doesn't handle errors well on gzipped files.)

The best solution may be to use a tool such as afio. afio supports multivolume backups and is similar in some respects to cpio. However, afio includes compression and is more reliable because each individual file is compressed. This means that if data in an archive is corrupted, the damage can be isolated to individual files, instead of to the entire backup.

These tools should be available with your Linux distribution, as well as from all the Internet-based Linux archives. A number of other backup utilities, with varying degrees of popularity and usability, have been developed or ported for Linux. If you're serious about backups, you should look into them.[*] Among these programs are the freely available taper, tob, and Amanda, as well as commercial programs such as ARKEIA (free for use with up to two computers), BRU, and Arcserve. Lots of free backup tools can also be found at

Incremental Backups

Incremental backups, as described earlier in this chapter, are a good way to keep your system backups up-to-date. For example, you can make nightly backups of only those files that changed in the last 24 hours, weekly backups of all files that changed in the last week, and monthly backups of the entire system.

You can create incremental backups using the tools mentioned previously: tar, gzip, cpio, and so on. The first step in creating an incremental backup is to produce a list of files that have changed since a certain amount of time ago. You can do this easily with the find command.[*] If you use a special backup program, you will most likely not have to do this, but can set some option somewhere that you want to do an incremental backup.

For example, to produce a list of all files that were modified in the last 24 hours, we can use the command:

find / -mtime -1 \! -type d -print > /tmp/filelist.daily

The first argument to find is the directory to start from—here, /, the root directory. The -mtime -1 option tells find to locate all files that changed in the last 24 hours.

The \! -type d bit is complicated (and optional), but it cuts some unnecessary stuff from your output. It tells find to exclude directories from the resulting file list. The ! is a negation operator (meaning here, "exclude files of type d"), but put a backslash in front of it because otherwise the shell interprets it as a special character.

The -print option causes all filenames matching the search to be printed to standard output. We redirect standard output to a file for later use. Likewise, to locate all files that changed in the last week, use:

find / -mtime -7 -print > /tmp/filelist.weekly

Note that if you use find in this way, it traverses all mounted filesystems. If you have a CD-ROM mounted, for example, find attempts to locate all files on the CD-ROM as well (which you probably do not wish to backup). The -xdev option can be used to limit find's traversal to the local filesystem. Another approach would be to use find multiple times with a first argument other than /. See the manual page for find(1) for details.

Now you have produced a list of files to back up. Previously, when using tar, we specified the files to archive on the command line. However, this list of files may be too long for a single command line (which is usually limited to around 2048 characters), and the list itself is contained within a file.

You can use the -T option with tar to specify a file containing a list of files for tar to back up. In order to use this option, you have to use an alternate syntax to tar in which all options are specified explicitly with dashes. For example, to back up the files listed in /tmp/filelist.daily to the device/dev/qft0, use the following command:

tar -cv -T /tmp/filelist.daily -f /dev/qft0

You can now write a short shell script that automatically produces the list of files and backs them up using tar. You can use cron to execute the script nightly at a certain time; all you have to do is make sure there's a tape in the drive. You can write similar scripts for your weekly and monthly backups. cron is covered in Chapter 10.

[*] Of course, this section was written after the author made the first backup of his Linux system in nearly four years of use!

[*] If you're not familiar with find, become so soon. find is a great way to locate files across many directories that have certain filenames, permissions, or modification times. find can even execute a program for each file that it locates. In short, find is your friend, and all good system administrators know how to use it well.

What to Do in an Emergency

It's not difficult to make a simple mistake as root that can cause real problems on your system, such as not being able to log in or losing important files. This is especially true for novice system administrators who are beginning to explore the system. Nearly all new system administrators learn their lessons the hard way: by being forced to recover from a real emergency. In this section, we give you some hints about what to do when the inevitable happens.

You should always be aware of preventive measures that reduce the impact of such emergencies. For example, make backups of all important system files, if not the entire system. If you happen to have a Linux distribution on CD-ROM, the CD-ROM itself acts as a wonderful backup for most files (as long as you have a way to access the CD-ROM in a tight situation—more on this later). Backups are vital to recovering from many problems; don't let the many weeks of hard work configuring your Linux system go to waste.

Also, be sure to keep notes on your system configuration, such as your partition table entries, partition sizes and types, and filesystems. If you were to trash your partition table somehow, fixing the problem might be a simple matter of rerunning fdisk, but this helps only as long as you can remember what your partition table used to look like. (True story: one of the authors once created this problem by booting a blank floppy, and had no record of the partition table contents. Needless to say, some guesswork was necessary to restore the partition table to its previous state!)

In fact, it's not a bad idea to actually back up the partition tables of each disk in your system. The sfdisk program is a very useful tool for viewing, saving, and manipulating partition data. You can capture this data and store it in a file with a command such as

sfdisk -d > /partitions.lst

This dumps out the partition tables of all the disks in your system and saves them into the file /partitions.lst (or whatever you want to call it). This output is readable not only by humans, but by sfdisk as well. If you need to restore a partition table, you can edit the partitions.lst file, remove all the partition info you don't want to restore, and rebuild the partition table (for hda, for example) like this:

sfdisk /dev/hda < partitions.lst

Of course, for any of these measures to work, you'll need a way to boot the system and access your files, or recover from backups, in an emergency. This is best accomplished with an "emergency disk ," or "root disk." Typically, this is a bootable CD-ROM containing at least enough of a Linux system to be able to recover filesystems and do most kinds of repair work. There are also many full-featured CD-ROMs, such as Knoppix (http://, that boot into a system with a graphical desktop, Web browser, and everything else you'd want to work comfortably. Either type can be very useful when you need to recover from a disaster.

For systems that can boot only from a floppy, you need a small root filesystem with the basics required to run a Linux system from floppy—just the essential commands and system files, as well as tools to repair problems. Use such a disk by booting a kernel from another floppy (see "Using a Boot Floppy" in Chapter 17) and telling the kernel to use the emergency disk as the root filesystem.

Most distributions of Linux include such a boot/root floppy combination with the original installation floppies. The installation disks usually contain a small Linux system that can be used to install the software as well as perform basic system maintenance. Some systems include both the kernel and root filesystem on one floppy, but this severely limits the number of files that can be stored on the emergency disk. How useful these disks are as a maintenance tool depends on whether they contain the tools (such as fsck, fdisk, a small editor such as vi, and so on) necessary for problem recovery. Some distributions have such an elaborate installation process that the installation floppies don't have room for much else.

At any rate, you can create such a root floppy yourself. Being able to do this from scratch requires an intimate knowledge of what's required to boot and use a Linux system, and exactly what can be trimmed down and cut out. For example, you could dispose of the startup programs init, getty, and login, as long as you know how to rig things so that the kernel starts a shell on the console instead of using a real boot procedure. (One way to do this is to have /etc/init be a symbolic link to /sbin/bash, all on the floppy filesystem.)

Although we can't cover all the details here, the first step in creating an emergency floppy is to use mkfs to create a filesystem on a floppy (see the section "Creating Filesystems" in Chapter 10). You then mount the floppy and place on it whatever files you'll need, including appropriate entries in /dev (most of which you can copy from /dev on your hard drive root filesystem). You'll also need a boot floppy, which merely contains a kernel. The kernel should have its root device set to /dev/fd0, using rdev. This is covered in "Using a Boot Floppy" in Chapter 17. You'll also have to decide whether you want the root floppy filesystem loaded into a ramdisk (which you can set using rdev as well). If you have more than 4 MB of RAM, this is a good idea because it can free up the floppy drive to be used for, say, mounting another floppy containing additional tools. If you have two floppy drives, you can do this without using a ramdisk.

If you feel that setting up an emergency floppy is too hard for you now after reading all this, you might want to try some of the scripts available that do it for you (e.g., tomsrtbt at But whatever you do, be sure to try the emergency floppy before disaster happens!

At any rate, the best place to start is your installation floppies. If those floppies don't contain all the tools you need, create a filesystem on a separate floppy and place the missing programs on it. If you load the root filesystem from floppy into a ramdisk, or have a second floppy drive, you can mount the other floppy to access your maintenance tools.

What tools do you need? In the following sections, we talk about common emergencies and how to recover from them; this should guide you as to what programs are required for various situations. It is best if the tools you put on the floppy are statically linked in order to avoid problems with shared libraries not being available at emergency time.

Repairing Filesystems

As discussed in "Checking and Repairing Filesystems" in Chapter 10, you can use fsck to recover from several kinds of filesystem corruption. Most of these filesystem problems are relatively minor and can be repaired by booting your system in the usual way and running fsck from the hard drive. However, it is usually better to check and repair your root filesystem while it is unmounted. In this case, it's easier to run fsck from an emergency floppy.

There are no differences between running fsck from floppy and from the hard drive; the syntax is exactly the same as described earlier in the chapter. However, remember that fsck is usually a frontend to tools such as fsck.ext3. On other systems, you'll need to use e2fsck (for Second Extended filesystems ).

It is possible to corrupt a filesystem so that it cannot be mounted. This is usually the result of damage to the filesystem's superblock, which stores information about the filesystem as a whole. If the superblock is corrupted, the system won't be able to access the filesystem at all, and any attempt to mount it will fail (probably with an error to the effect of "can't read superblock").

Because of the importance of the superblock, the filesystem keeps backup copies of it at intervals on the filesystem. Second Extended filesystems (and ext3, which is mostly the same) are divided into "block groups ," where each group has, by default, 8192 blocks. Therefore, there are backup copies of the superblock at block offsets 8193, 16385 (that's 8192 × 2 + 1), 24577, and so on. If you use the ext2 or ext3 filesystem, check that the filesystem has 8192-block groups with the following command:

dumpe2fs device | more

(Of course, this works only when the master superblock is intact.) This command will print a great deal of information about the filesystem, and you should see something like the following:

Blocks per group: 8192

If another offset is given, use it for computing offsets to the superblock copies, as mentioned earlier.

If you can't mount a filesystem because of superblock problems, chances are that fsck (or e3fsck) will fail as well. You can tell e3fsck to use one of the superblock copies, instead, to repair the filesystem. The command is:

e3fsck -f -b offset device

where offset is the block offset to a superblock copy; usually, this is 8193. The -f switch is used to force a check of the filesystem; when using superblock backups, the filesystem may appear "clean," in which case no check is needed. -f overrides this. For example, to repair the filesystem on /dev/hda2 with a bad superblock, we can say:

e3fsck -f -b 8193 /dev/hda2

Superblock copies save the day. The previous commands can be executed from an emergency floppy system and will hopefully allow you to mount your filesystems again.

Now that journaling filesystems such as ext3, reiserfs (the Reiser filesystem), and jfs come with most Linux distributions by default, it is very unlikely that you will ever have to resort to the superblock-wrangling tricks just described. Because of the "journal," which is a log kept internally by the filesystem of all changes as they are made, modern filesystems are quite resistant to superblock damage. Still, it can happen, and it's good to know how to recover from it easily, without having to rebuild your entire filesystem.

Accessing Damaged Files

You might need to access the files on your hard drive filesystems when booting from an emergency floppy or CD-ROM. To do this, simply use the mount command as described in "Mounting Filesystems" in Chapter 10, mounting your filesystems under a directory such as /mnt. (This directory must exist on the root filesystem contained on the rescue disk.) For example,

mount -t ext3 /dev/hda2 /mnt

will allow us to access the files on the ext3 filesystem stored on /dev/hda2 in the directory /mnt. You can then access the files directly and even execute programs from your hard drive filesystems. For example, if you wish to execute vi from the hard drive, normally found in /usr/bin/vi, you would use the command

/mnt/usr/bin/vi filename

You could even place subdirectories of /mnt on your path to make this easier.

Be sure to unmount your hard drive filesystems before rebooting the system. If your emergency disks don't have the ability to do a clean shutdown, unmount your filesystems explicitly with umount, to be safe.

Two problems that can arise when doing this are forgetting the root password or trashing the contents of /etc/passwd. In either case, it might be impossible to log in to the system or su to root. To repair this problem, simply boot from your emergency disks, mount your root filesystem under/mnt, and edit /mnt/etc/passwd. (It might be a good idea to keep a backup copy of this file somewhere in case you delete it accidentally.) For example, to clear the root password altogether, change the entry for root to

root::0:0:The root of all evil:/:/bin/bash

Now root will have no password; you can reboot the system from the hard drive and use the passwd command to reset it.

If you are conscientious about system security, you might have shivered by now. You have read correctly: if somebody has physical access to your system, he or she can change your root password by using a boot floppy. Luckily, there are ways to protect your system against possible assaults. Most effective are, of course, the physical ones: if your computer is locked away, nobody can access it and put a boot floppy into it. There are also locks for the floppy drive only, but notice that you need such protection for the CD-ROM drive as well for floppy-drive locks to be useful. If you don't want to use physical protection, you can also use the BIOS password if your computer supports that: configure the BIOS so that it does not try to boot from CD-ROM or floppy (even if a CD or floppy disk is inserted at boot time) and protect the BIOS settings with a BIOS password. This is not as secure because it is possible to reset the BIOS password with hardware means, but it still protects you against casual would-be intruders. Actually, of course, somebody could steal the whole computer.

Another common problem is corrupt links to shared system libraries. The shared library images in /lib are generally accessed through symbolic links, such as /lib/, which point to the actual library, /lib/ If this link is removed or is pointing to the wrong place, many commands on the system won't run. You can fix this problem by mounting your hard drive filesystems and relinking the library with a command such as:

cd /mnt/lib; ln -sf

to force the link to point to Remember that symbolic links use the pathname given on the ln command line. For this reason, the command:

ln -sf /mnt/lib/ /mnt/lib/

won't do the right thing: will point to /mnt/lib/ When you boot from the hard drive, /mnt/lib can't be accessed, and the library won't be located. The first command works because the symbolic link points to a file in the same directory.

Restoring Files from Backup

If you have deleted important system files, it might be necessary to restore backups while booting from an emergency disk. For this reason, it's important to be sure your emergency disk has the tools you need to restore backups; this includes programs such as tar and gzip, as well as the drivers necessary to access the backup device. For instance, if your backups are made using the floppy-tape device driver, be sure that the ftape module and insmod command are available on your emergency disk. See "Loading Modules Automatically" in Chapter 18 for more about this.

All that's required to restore backups to your hard drive filesystems is to mount those filesystems, as described earlier, and unpack the contents of the archives over those filesystems (using the appropriate tar and gzip commands, for example; see "Making Backups" earlier in this chapter). Remember that every time you restore a backup you will be overwriting other system files; be sure you're doing everything correctly so that you don't make the situation worse. With most archiving programs, you can extract individual files from the archive.

Likewise, if you want to use your original CD-ROM to restore files, be sure the kernel used on your emergency disks has the drivers necessary to access the CD-ROM drive. You can then mount the CD-ROM (remember the mount flags -r -t iso9660) and copy files from there.

The filesystems on your emergency disks should also contain important system files; if you have deleted one of these from your system, it's easy to copy the lost file from the emergency disk to your hard drive filesystem.

For more information, including some scripts, examples, advice about backups in general, and more, take a look at Charles Curley's Linux Complete Backup and Recovery HOWTO, available at: