Unix commands: Troubleshooting with lsof

If you’ve never used the lsof command or used it only for one specific purpose, you might be delighted to learn how many ways the lsof (list open files) command can help you manage your servers. lsof is the Unix/Linux command that allows you to list open files or identify the processes that particular files have open. Handy for evaluating system security as well as troubleshooting, lsof features a large range of options that allow it to be used in numerous ways – sometimes even surpassing the ps command for looking at processes and the netstat command for examining network interfaces.

What are open files?

For starters, let’s consider what open files are and why you might be curious about them. Open files are files that some process is using. That process might be a command that you are running or an application that is running on a server you manage. The open files might include data files and libraries that supply shared routines. Many files are opened every time you log in. You might be surprised at how many. If you’re curious about how many files you have open right now, try this command.

$ lsof -u `whoami` | wc -l
55

And, if you’ve ever heard anyone say that for Unix, everything is a file, you might not be too surprised to learn that lsof works with things — like network interfaces — that most of us don’t generally think of as files.

Why do you care?

Sometimes you might want to know about open files because you try to remove a file and discover that it’s in use. Maybe it’s filling up your disk space like a desperate shopper at a dollar store. You want to know what process has the file open so that you can stop it and empty the file. At other times, you might want to know what some suspicious process is doing and examining the files that it has open can provide valuable information.

How does lsof work?

When used without options, lsof lists all files that are open (in use) on your system. If you run the lsof as yourself, you will get a long listing, but the output will include a lot of permission denied messages – many representing open files in the /proc file system that you’re not allowed to see. Run the command as root and you’ll see more output and all the data.

Note: The command output below (as well as the output for most of the commands shown in this post) has been truncated to keep this post from turning into a virtual ebook. The ellipsis (…) at the end of each patch of output is meant to indicate where output was omitted.

$ lsof
COMMAND PID USER   FD  TYPE      DEVICE  SIZE/OFF       NODE NAME
init      1 root  cwd  unknown  /proc/1/cwd (readlink:  Permission denied)
init      1 root  rtd  unknown  /proc/1/root (readlink: Permission denied)
init      1 root  txt  unknown  /proc/1/exe (readlink:  Permission denied)
...

$ sudo lsof
COMMAND PID USER   FD  TYPE      DEVICE  SIZE/OFF       NODE NAME
init      1 root  cwd  DIR       202,1      4096          2 /
init      1 root  rtd  DIR       202,1      4096          2 /
init      1 root  txt  REG       202,1    150360     397362 /sbin/init
init      1 root  DEL  REG       202,1               396485 /lib64/libnss_files-2.17.so
...

What else is there?

A quick glimpse at the rather length lsof man page will show you that we’ve only skimmed the surface. The lsof command has an extensive list of options.

 lsof [ -?abChlnNOPRtUvVX ] [ -A A ] [ -c c ] [ +c c ] [ +|-d d ] [ +|-D
 D ] [ +|-e s ] [ +|-f [cfgGn] ] [ -F [f] ] [ -g [s] ] [ -i [i] ] [ -k k
 ]  [  +|-L  [l]  ]  [  +|-m  m  ]  [  +|-M ] [ -o [o] ] [ -p s ] [ +|-r
 [t[m]] ] [ -s [p:s] ] [ -S [t] ] [ -T [t] ] [ -u s ] [ +|-w ] [ -x
 [fl] ] [ -z [z] ] [ -Z [Z] ] [ -- ] [names]

We’re only going to look at some of the more useful ones in this post.

To get started with all of these options, you should know that, if you use more than one, they get OR’ed together. In other words, you get a listing that combines the results of the options you specified. You can choose instead to use an option which AND’s your specifications together. In other words, you will see only the files or processes, etc. that match all of the options you specify. To use the AND option, add the -a option (there’s an example command below) to your command.

For a quick review of the implications of AND and OR, if you ask a programmer for a peanut butter AND jelly sandwich, you’ll probably end up with just two slices of bread – because the toppings can’t be both peanut butter and jelly at the same time. The logic goes something like this: if INGREDIENT = “peanut butter” AND INGREDIENT = “jelly”, SANDWICH-TOPPINGS = NULL.

Helpful lsof commands

The sample lsof commands below help illustrate some of the more useful things that you can do with the command. Again, most of the output is truncated to keep the length of this post reasonable.

The sample lsof command below will list all processes that have a particular file open.

$ sudo lsof /lib64/libcrypto.so.1.0.1k
COMMAND    PID     USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
ntpd      2148      ntp mem    REG  202,1  1971624 399258 /lib64/libcrypto.so.1.0.1k
sendmail  2163     root mem    REG  202,1  1971624 399258 /lib64/libcrypto.so.1.0.1k
sendmail  2170    smmsp mem    REG  202,1  1971624 399258 /lib64/libcrypto.so.1.0.1k
sshd     13774     root mem    REG  202,1  1971624 399258 /lib64/libcrypto.so.1.0.1k
sshd     13776 froggy mem    REG  202,1  1971624 399258 /lib64/libcrypto.so.1.0.1k
sshd     22551     root mem    REG  202,1  1971624 399258 /lib64/libcrypto.so.1.0.1k
...

This command will list all processes that have files within a particular directory open.

$ sudo lsof +D /lib64 | more
COMMAND     PID     USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
init          1     root mem    REG  202,1    89312 396417 /lib64/libgcc_s-4.8.2-20140120.so.1
init          1     root mem    REG  202,1   283184 397017 /lib64/libdbus-1.so.3.7.4
init          1     root mem    REG  202,1    39936 397358 /lib64/libnih-dbus.so.1.0.0
init          1     root mem    REG  202,1   101992 397360 /lib64/libnih.so.1.0.0
dhclient   1848     root mem    REG  202,1    18712 396864 /lib64/libcap-ng.so.0.0.0
auditd     1889     root mem    REG  202,1    89312 396417 /lib64/libgcc_s-4.8.2-20140120.so.1
auditd     1889     root mem    REG  202,1    13224 396979 /lib64/libkeyutils.so.1.5
…

In this command, we look at files opened by bash.

$ sudo lsof -c bash | more
COMMAND   PID   USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
bash    17811 froggy  cwd    DIR  202,1      4096 410779 /home/froggy
bash    17811 froggy  rtd    DIR  202,1      4096      2 /
bash    17811 froggy  txt    REG  202,1    898032 396531 /bin/bash
bash    17811 froggy  mem    REG  202,1 106065056 410803 /usr/lib/locale/local
e-archive
bash    17811 froggy  mem    REG  202,1     58288 396484 /lib64/libnss_files-2
.17.so
bash    17811 froggy  mem    REG  202,1   2107600 396466 /lib64/libc-2.17.so
bash    17811 froggy  mem    REG  202,1     19512 396472 /lib64/libdl-2.17.so
bash    17811 froggy  mem    REG  202,1    135616 396516 /lib64/libtinfo.so.5.
7
bash    17811 froggy  mem    REG  202,1    160240 396459 /lib64/ld-2.17.so
bash    17811 froggy  mem    REG  202,1     26254   1596 /usr/lib64/gconv/gcon
v-modules.cache
bash    17811 froggy    0u   CHR  136,0       0t0      3 /dev/pts/0
bash    17811 froggy    1u   CHR  136,0       0t0      3 /dev/pts/0
bash    17811 froggy    2u   CHR  136,0       0t0      3 /dev/pts/0
bash    17811 froggy  255u   CHR  136,0       0t0      3 /dev/pts/0

Below, we do the same thing but use a substring instead of the full process name.

$ sudo lsof -c bas
$ sudo lsof -c bas
COMMAND   PID     USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
bash    17811 ec2-user  cwd    DIR  202,1      4096 410779 /home/froggy
bash    17811 ec2-user  rtd    DIR  202,1      4096      2 /
bash    17811 ec2-user  txt    REG  202,1    898032 396531 /bin/bash
bash    17811 ec2-user  mem    REG  202,1 106065056 410803 /usr/lib/locale/locale-archive
…

Next, we list open files for a particular process ID.

$ sudo lsof -p 2178
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
crond   2178 root  cwd    DIR              202,1     4096      2 /
crond   2178 root  rtd    DIR              202,1     4096      2 /
crond   2178 root  txt    REG              202,1    64096 403907 /usr/sbin/crond
crond   2178 root  DEL    REG              202,1          396485 /lib64/libnss_files-2.17.so
crond   2178 root  DEL    REG              202,1          396511 /usr/lib/locale/locale-archive
crond   2178 root  DEL    REG              202,1          396467 /lib64/libc-2.17.so
crond   2178 root  mem    REG              202,1   113112 396681 /lib64/libaudit.so.1.0.0
crond   2178 root  DEL    REG              202,1          396473 /lib64/libdl-2.17.so
crond   2178 root  mem    REG              202,1    55928 398855 /lib64/libpam.so.0.83.1
crond   2178 root  mem    REG              202,1   126336 396609 /usr/lib64/libselinux.so.1
crond   2178 root  DEL    REG              202,1          396460 /lib64/ld-2.17.so
crond   2178 root    0u   CHR                1,3      0t0   5422 /dev/null
crond   2178 root    1u   CHR                1,3      0t0   5422 /dev/null
crond   2178 root    2u   CHR                1,3      0t0   5422 /dev/null
crond   2178 root    3u   REG              202,1        5 263404 /var/run/crond.pid
crond   2178 root    4u  unix 0xffff88003d754980      0t0   9655 socket
crond   2178 root    5r  0000                0,9        0   5420 anon_inode

Using lsof to look at your network

Looking at network connections We can also use lsof to look at network connections.

$ sudo lsof -i
COMMAND    PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
dhclient  1848     root    5u  IPv4     8573      0t0  UDP *:bootpc
ntpd      2148      ntp   16u  IPv4     9534      0t0  UDP *:ntp
ntpd      2148      ntp   17u  IPv4     9538      0t0  UDP localhost:ntp
ntpd      2148      ntp   18u  IPv4     9539      0t0  UDP 172.30.0.28:ntp
sendmail  2163     root    4u  IPv4     9613      0t0  TCP localhost:smtp (LISTEN)
sshd     15350     root    3u  IPv4 17247327      0t0  TCP 172.30.0.28:ssh>ip108.cable.shentel.net:37884 (ESTABLISHED)
sshd     15352 ec2-user    3u  IPv4 17247327      0t0  TCP 172.30.0.28:ssh>ip108.cable.shentel.net:37884 (ESTABLISHED)
sshd     22551     root    3u  IPv4    32640      0t0  TCP *:ssh (LISTEN)
sshd     22551     root    4u  IPv6    32642      0t0  TCP *:ssh (LISTEN)

We can look for listening ports or established connections.

$ sudo lsof -i -sTCP:LISTEN
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sendmail  2163 root    4u  IPv4   9613      0t0  TCP localhost:smtp (LISTEN)
sshd     22551 root    3u  IPv4  32640      0t0  TCP *:ssh (LISTEN)
sshd     22551 root    4u  IPv6  32642      0t0  TCP *:ssh (LISTEN)

$ sudo lsof -i -sTCP:ESTABLISHED
COMMAND   PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
sshd    13060     root    3u  IPv4 17234339      0t0  TCP 172.30.0.28:ssh->ip108.cable.shentel.net:37846 (ESTABLISHED)
sshd    13062 froggy    3u  IPv4 17234339      0t0  TCP 172.30.0.28:ssh->ip108.cable.shentel.net:37846 (ESTABLISHED)

View IPv6 traffic only

$ sudo lsof -i 6
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    22551 root    4u  IPv6  32642      0t0  TCP *:ssh (LISTEN)

$ sudo lsof -iUDP
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dhclient 1848 root    5u  IPv4   8573      0t0  UDP *:bootpc
ntpd     2148  ntp   16u  IPv4   9534      0t0  UDP *:ntp
ntpd     2148  ntp   17u  IPv4   9538      0t0  UDP localhost:ntp
ntpd     2148  ntp   18u  IPv4   9539      0t0  UDP 172.30.0.28:ntp

Or we can look at network connections for one particular source.

$ sudo lsof -i@172.30.0.28
COMMAND   PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
ntpd     2148      ntp   18u  IPv4     9539      0t0  UDP 172.30.0.28:ntp
sshd    13060     root    3u  IPv4 17234339      0t0  TCP 172.30.0.28:ssh->ip108.cable.shentel.net:37846 (ESTABLISHED)
sshd    13062 froggy    3u  IPv4 17234339      0t0  TCP 172.30.0.28:ssh->ip108.cable.shentel.net:37846 (ESTABLISHED)

We can also do the same thing but only for one particular network connection.

$ sudo lsof -u ec2-user -i @172.30.0.28
COMMAND   PID     USER   FD   TYPE             DEVICE  SIZE/OFF     NODE NAME
ntpd     2148      ntp   18u  IPv4               9539       0t0      UDP 172.30.0.28:ntp
sshd    15350     root    3u  IPv4           17247327       0t0      TCP 172.30.0.28:ssh->ip108.cable.shentel.net:37884 (ESTABLISHED)
sshd    15352 ec2-user  cwd    DIR              202,1      4096        2 /
sshd    15352 ec2-user  rtd    DIR              202,1      4096        2 /
sshd    15352 ec2-user  txt    REG              202,1    617128   404007 /usr/sbin/sshd
...

Looking at files by user

This lsof command allows you to view open files for a particular user.

$ sudo lsof -u froggy
COMMAND  PID     USER   FD   TYPE   DEVICE  SIZE/OFF     NODE NAME
sshd   13062 froggy  cwd    DIR    202,1      4096        2 /
sshd   13062 froggy  rtd    DIR    202,1      4096        2 /
sshd   13062 froggy  txt    REG    202,1    617128   404007 /usr/sbin/sshd
sshd   13062 froggy  DEL    REG      0,4    17234382 /dev/zero
sshd   13062 froggy  mem    REG    202,1     18640   398879 /lib64/security/pam_limits.so
sshd   13062 froggy  mem    REG    202,1     10264   398877 /lib64/security/pam_keyinit.so
sshd   13062 froggy  mem    REG    202,1     10288   398882 /lib64/security/pam_loginuid.so
sshd   13062 froggy  mem    REG    202,1     18736   398894 /lib64/security/pam_selinux.so
sshd   13062 froggy  mem    REG    202,1     38976   398532 /usr/lib64/libcrack.so.2.8.1
sshd   13062 froggy  mem    REG    202,1     14456   398863 /lib64/security/pam_cracklib.so
…

To view open files for all users except some particular user (in this case, root), use a ^ sign.

$ sudo lsof -u ^root | more
COMMAND     PID   USER   FD   TYPE   DEVICE  SIZE/OFF     NODE NAME
dbus-daem  1935   dbus  cwd    DIR   202,1      4096        2 /
dbus-daem  1935   dbus  rtd    DIR   202,1      4096        2 /
dbus-daem  1935   dbus  txt    REG   202,1    403416   404014 /bin/dbus-daemon
dbus-daem  1935   dbus  DEL    REG   202,1             396485 /lib64/libnss_files-2.17.so
dbus-daem  1935   dbus  DEL    REG   202,1             396473 /lib64/libdl-2.17.so
dbus-daem  1935   dbus  DEL    REG   202,1             396467 /lib64/libc-2.17.so
dbus-daem  1935   dbus  DEL    REG   202,1             396497 /lib64/librt-2.17.so
dbus-daem  1935   dbus  DEL    REG   202,1             396493 /lib64/libpthread-2.17.so
dbus-daem  1935   dbus  mem    REG   202,1     18712   396864 /lib64/libcap-ng.so.0.0.0
dbus-daem  1935   dbus  mem    REG   202,1    113112   396681 /lib64/libaudit.so.1.0.0

List process IDs for processes being run by a particular user.

$ sudo lsof -t -u froggy
15352
15353

Kill all processes belonging to a particular user

$ sudo kill `lsof -t -u froggy`

Switching to AND

Use the -a option to AND your conditions together, keeping in mind that this limits the output to only that which matches all specified conditions.

$ sudo lsof -i -a -p 2148
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ntpd    2148  ntp   16u  IPv4   9534      0t0  UDP *:ntp
ntpd    2148  ntp   17u  IPv4   9538      0t0  UDP localhost:ntp
ntpd    2148  ntp   18u  IPv4   9539      0t0  UDP 172.30.0.28:ntp
...

Wrap up

The lsof command is a lot more useful and versatile than many Unix admins realize. It could come in handy for a lot of tasks that you’ve been addressing in other ways. Try out some of these commands and let me know which you like the most. And don’t forget to make your own peanut butter and jelly sandwiches!