Linux Administration (2016)
System Logging
Linux uses the syslog standard for message logging. This allows programs and applications to generate messages that can be captured, processed, and stored by the system logger. It eliminates the need for each and every application having to implement a logging mechanism. It also means that logging can be configured and controlled in a central location.
The syslog standard uses facilities and severities to categorize messages. Each message is labeled with a facility code and a severity level. The various combinations of facilities and severities can be used to determine how a message is handled.
Facilities are used to indicate what type of program or what part of the system the message originated from. For example, messages that are labeled with the kern facility originate from the Linux kernel. Messages that are labeled with the mail facility come from applications involved in handling mail.
Each facility has a number and a keyword associated with it. This table lists the syslog facilities.
Number Keyword Description
0 kern kernel messages
1 user user-level messages
2 mail mail system
3 daemon system daemons
4 auth security/authorization messages
5 syslog messages generated by syslogd
6 lpr line printer subsystem
7 news network news subsystem
8 uucp UUCP subsystem
9 clock daemon
10 authpriv security/authorization messages
11 ftp FTP daemon
12 - NTP subsystem
13 - log audit
14 - log alert
15 cron clock daemon
16 local0 local use 0 (local0)
17 local1 local use 1 (local1)
18 local2 local use 2 (local2)
19 local3 local use 3 (local3)
20 local4 local use 4 (local4)
21 local5 local use 5 (local5)
22 local6 local use 6 (local6)
23 local7 local use 7 (local7)
This table lists each of the severities, including their code, keyword, and description.
Code Severity Keyword Description
0 Emergency emerg (panic) System is unusable
1 Alert alert Action must be taken
immediately
2 Critical crit Critical conditions
3 Error err (error) Error conditions
4 Warning warning (warn) Warning conditions
5 Notice notice Normal but
significant condition
6 Info info Informational
messages
7 Debug debug Debug-level messages
A syslog server accepts syslog messages and process those messages based on a set of configurable rules. Traditionally, the syslogd daemon filled this role, but many Linux distributions today ship with alternatives such as rsyslog and syslog-ng. For the remainder of this chapter, we are going to focus on rsyslog; however, the concepts apply to any syslog server.
Logging Rules
The main configuration file for rsyslog is /etc/rsyslog.conf. You can include additional configuration files by using the $IncludeConfig directive. For example, this line will cause rsyslog to read and include all the configuration files that end in .conf in the /etc/rsyslog.d directory.
$IncludeConfig /etc/rsyslog.d/*.conf
Logging rules consist of two fields. The first field is called the selector field and it lists the facilities and severities of messages to include in the rule. The second field is called the action field and it determines what will happen to the messages matched by the selector field. The most common action is to write the messages to a log file. The selector field and action field can be separated by one or more spaces or tabs.
The format of the selector field is FACILITY.SEVERITY. Note that wildcards are supported. For example, use the mail.* selector to match all the mail messages. You can also omit the .SEVERITY if you want to include all messages from the facility. In short, mail.* and mail are equivalent. If you do not want to match any messages from a facility use FACILITY.none. If you want to match multiple facility/severity pairs, separate each pair with a semicolon.
This example rule matches messages that have the facility of mail and any severity. It writes all the matching messages to /var/log/mail.log.
mail.* /var/log/mail.log
If the path starts with a minus sign that tells rsyslog that it doesn’t have to perform a sync() operation for each log message. This is sometimes called caching mode. When using caching mode, know that some message might be lost if a system crashes immediately after a write attempt. However, you may see performance improvements during normal operations if you have a system that performs many logging operations. The default configurations that ship with a Linux distribution will probably have a mix of caching (-/path) and non-caching (/path) rules with less critical messages using caching. Here’s an example.
mail.* -/var/log/mail.log
In this example, there are separate actions for different severities of mail messages. Notice that the less critical mail messages are using caching mode.
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
This example is taken from an Ubuntu system. The first line ensures all messages from the auth and authpriv facilities are written to /var/log/auth.log. The second line writes all messages except ones originating from the auth and authpriv facilities to /var/log/syslog.
auth,authpriv.* /var/log/auth.log
*.*;auth.none,authpriv.none -/var/log/syslog
This example is taken from a RedHat Enterprise Linux system. The rule tells rsyslog to write all messages except mail, authpriv, and cron to /var/log/messages.
*.info;mail.none;authpriv.none;cron.none /var/log/messages
Hopefully you can see why it’s important to understand how to determine where messages are being sent instead of simply memorizing that system messages are stored in /var/log/messages. Different distributions ship with different configurations and different companies, organizations, and system administrators may alter the default rules to suit their particular needs.
Creating Your Own Syslog Messages
You can use the logger command to generate syslog messages. This can prove useful if you want to test any configuration changes you’ve made to the system logger or if you want to generate log message from your own shell scripts. Use the -p option to provide a FACILITY.SEVERITYlevel. If you don’t specify a facility/severity pair, it will default to user.notice. You can also use the -t option to tag your message.
logger [options] message
In this example, we generate a message with the mail facility at a severity level of info. You can see that the message made its way to the proper log file.
$ logger -p mail.info -t mailtest "Testing 123."
$ sudo tail -1 /var/log/mail.log
Apr 4 14:33:16 linuxsvr mailtest: Testing 123.
Rotating Log Files
You can use the logrotate tool to rotate, compress, remove, and even mail log files. This provides an automated way to manage log files and can help prevent filling up your storage space with log messages.
The configuration file for logrotate is located at /etc/logrotate.conf. Like many other configuration files it may use an include directive. This line tells logrotate to read configuration files located in the /etc/logrotate.d directory.
include /etc/logrotate.d
Here is a sample logrotate.conf file.
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating
# old ones
create
# uncomment if you want your log files compressed
#compress
# packages drop log rotation information
# into this directory
include /etc/logrotate.d
The configuration in the main file contains some defaults. For example, the weekly keyword ensures that log files will be rotated every week. The rotate 4 line tells logrotate to keep 4 weeks worth of logs. Logs older than this will be removed. The create line makes sure that a new empty log file is created after it is rotated.
This is a sample logrotate configuration file from an Ubuntu system. It is located at /etc/logrotate.d/rsyslog. It handles the log rotation for all the files associated with rsyslog.
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
Notice that the format is a single log file, or list of log files, followed by the configuration that controls those log files. The configuration is enclosed in brackets. Here are the configuration options used.
rotate count Rotate the files by count times before removing them.
daily Rotate log files every day.
weekly Rotate log files weekly.
missingok Ignore missing log files. (Do not issue an error.)
notifempty Do not rotate the log file if it is empty.
compress Compress rotated log files.
postrotate The lines between postrotate and endscript are executed using /bin/sh. These commands are executed after the rotation.
Testing Your Logrotate Configuration
If you make changes to your logrotate configuration and want to test it, use the following command.
# logrotate -fv /etc/logrotate.conf
The -f option tells logrotate to force a rotation while the -v option enables verbose logging.
Summary
In this chapter, you learned about the syslog standard and how it assigns a facility and severity to each message. You also learned that syslog servers employ the use of logging rules to determine what action to perform on a given message. Typically, the action is to simply store the message in a log file. You also learned how to test the syslog server configuration by generating messages with the logger utility. Finally, you learned how to use logrotate to automatically prune system logs.
Quiz
1. The syslog standard:
1. Aids in the processing of messages.
2. Allows logging to be centrally controlled.
3. Uses facilities and severities to categorize messages.
4. All of the above.
2. Which command can you use to generate log messages?
1. logger
2. log
3. logit
4. There is no such command.
Quiz Answers
1. D
2. A