Making better use of your Linux logs

Linux systems maintain quite a collection of log files, many of which you are probably rarely tempted to view. Some of these log files are quite valuable, though, and options for exploring them might be more interesting and varied than you imagine. Let’s look at some system logs and get a handle on some of the ways in which log data might be easier to probe.

Log file rotation

First, there’s the issue of log rotation. Some Linux log files are “rotated.” In other words, the system stores more than one “generation” of these files, mostly to keep them from using too much disk space. The older logs are then compressed but left available for a while. Eventually, the oldest in a series of rotated log files will be automatically deleted in the log rotation process, but you’ll still have access to a number of the older logs so that you can examine log entries that were added in the last few days or weeks when and if you need to look a little further back into some issue you’re tracking.

To get a feel for what types of system information are being saved, simply cd over to the /var/log directory and list its contents.

/var/log# ls
alternatives.log       btmp.1           kern.log.2.gz      syslog.3.gz
alternatives.log.1     cups             kern.log.3.gz      syslog.4.gz
alternatives.log.2.gz  dist-upgrade     kern.log.4.gz      syslog.5.gz
alternatives.log.3.gz  dpkg.log         lastlog            syslog.6.gz
alternatives.log.4.gz  dpkg.log.1       mail.err           syslog.7.gz
alternatives.log.5.gz  dpkg.log.2.gz    mail.err.1         sysstat
apport.log             dpkg.log.3.gz    mail.err.2.gz      tallylog
apport.log.1           dpkg.log.4.gz    mail.err.3.gz      ufw.log
apt                    dpkg.log.5.gz    mail.err.4.gz      ufw.log.1
atop                   faillog          mail.log           ufw.log.2.gz
auth.log               fontconfig.log   mail.log.1         ufw.log.3.gz
auth.log.1             gdm3             mail.log.2.gz      ufw.log.4.gz
auth.log.2.gz          gpu-manager.log  mail.log.3.gz      unattended-upgrades
auth.log.3.gz          hp               mail.log.4.gz      wtmp
auth.log.4.gz          installer        speech-dispatcher  wtmp.1
boot.log               journal          syslog
bootstrap.log          kern.log         syslog.1
btmp                   kern.log.1       syslog.2.gz

This is fairly large collection of logs and log directories — 69 files and directories in /var/log in this case, but 180 files when you include the files inside those directories.

$ cd /var/log
$ ls | wc -l
69
$ find . -type f -print | wc -l
180

When you examine your log files, you will see pretty clearly which are generations of the same basic log. For example, one of the primary log files — the syslog file — is broken into nine separate files. These represent what is basically a week’s worth of historical data along with the current file. Most of the older files are zipped to preserve space.

$ ls -l syslog*
-rw-r----- 1 syslog adm 588728 Oct 15 20:42 syslog
-rw-r----- 1 syslog adm 511814 Oct 15 00:09 syslog.1
-rw-r----- 1 syslog adm  31205 Oct 14 00:06 syslog.2.gz
-rw-r----- 1 syslog adm  34797 Oct 13 00:06 syslog.3.gz
-rw-r----- 1 syslog adm  61107 Oct 12 00:08 syslog.4.gz
-rw-r----- 1 syslog adm  31682 Oct 11 00:06 syslog.5.gz
-rw-r----- 1 syslog adm  32004 Oct 10 00:07 syslog.6.gz
-rw-r----- 1 syslog adm  32309 Oct  9 00:05 syslog.7.gz

The syslog files contain messages from many different system services — cron, sendmail and the kernel itself are just examples. You’ll also see evidence of user sessions and cron (scheduled tasks).

Most Linux systems no longer use the old messages and dmesg files that served as landing places for the bulk of our system messages for many years. Instead, a large variety of files and some special commands have become available to help present the log information that is likely to be most relevant to what you are looking for.

Depending on the file in question, you might simply use more or tail commands, or you might use a file-specific command like this use of the who command to pull user login data from the wtmp log.

$ who wtmp
shs      pts/1        2018-10-05 08:42 (192.168.0.10)
shs      pts/1        2018-10-08 09:41 (192.168.0.10)
shs      pts/1        2018-10-11 14:00 (192.168.0.10)
shs      :0           2018-10-14 19:11 (:0)
shs      pts/0        2018-10-14 19:16 (192.168.0.25)
shs      pts/0        2018-10-15 07:39 (192.168.0.25)
shs      :0           2018-10-15 19:58 (:0)
dory     pts/0        2018-10-15 20:01 (192.168.0.11)
shs      pts/0        2018-10-15 20:42 (192.168.0.6)
shs      pts/0        2018-10-16 07:18 (192.168.0.6)
nemo     pts/1        2018-10-16 07:46 (192.168.0.14)

Similarly, you might see nothing when you run a tail faillog command, but a command like this shows you that it’s simply full of zeroes:

# od -bc faillog
0000000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000