Examining Linux system performance with dstat

Want to do a quick performance check on your Linux system? You might want to take a look at the dstat command. Dstat provides valuable insights into Linux system performance, pretty much replacing a collection of older tools such as vmstat, netstat, iostat, and ifstat with a flexible and powerful command that combines their features.

With this one command, you can look at virtual memory, network connections and interfaces, CPU activity, input/output devices and more. In today’s post, we’ll examine some dstat commands and see what they can show you about your systems.

Dstat options and defaults

First, let’s start with a fairly simple command. With the dstat -c (CPU) option, dstat displays CPU stats. In the example below, we’re asking for two-second intervals and six reports.

$ dstat -c 2 6
--total-cpu-usage--
usr sys idl wai stl
  1   3  96   0   0
 33  67   0   0   0
 34  66   0   0   0
 35  66   0   0   0
 37  63   0   0   0
 36  64   0   0   0

Note that the first line of data in this report, which looks very different than the others, gives you the averages since the system was last booted and is returned immediately regardless of the specified interval. In this example, we see that the system on average has been largely idle (96%), but is now quite busy working between user and system processing tasks.

If you don’t supply any options with dstat, the command will use a default set (-cdngy) set of options. These include:

• c — cpu
• d — disk
• n — network
• g — paging stats
• y — system stats

The output of this command will look something like what you see below.

$ dstat 2 10
You did not select any stats, using -cdngy by default.
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  1   2  96   0   0|5568B 4040B|   0     0 |   0     0 |  58    63
 34  66   0   0   0|   0     0 | 174B  700B|   0     0 | 679   371
 34  66   0   0   0|   0     0 | 174B  407B|   0     0 | 680   377
 36  64   0   0   0|   0     0 |  64B  407B|   0     0 | 678   430
 35  65   0   0   0|   0     0 | 283B  407B|   0     0 | 680   374
 32  68   0   0   0|   0     0 | 238B  407B|   0     0 | 679   376
 33  67   0   0   0|   0     0 | 128B  407B|   0     0 | 680   374
 32  68   0   0   0|   0     0 | 251B  407B|   0     0 | 679   374
 33  67   0   0   0|   0     0 | 238B  407B|   0     0 | 676   376
 34  66   0   0   0|   0     0 | 173B  407B|   0     0 | 680   372

You probably noticed the “You did not select any stats” message near the top of the output displayed above. To overcome this with little effort, simply add the -a option. It will select the default options and omit the warning message.

$ dstat -a 2 5
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  4  10  86   0   0|5921B   12k|   0     0 |   0     0 | 127    99
 15  35  50   0   0|   0     0 | 302B  838B|   0     0 | 369   220
 15  35  50   0   0|   0    14k|  96B  407B|   0     0 | 365   216
 15  35  50   0   0|   0     0 | 246B  407B|   0     0 | 372   229
 18  32  50   0   0|   0     0 | 286B  407B|   0     0 | 359   208

In this “no options” approach, you can still set the timing for each interval in seconds and the number of intervals you want to see reported. If you don’t specify the number of intervals, the command will continue running until you stop it with a ^c.

What does this tell you?

In the output shown above, we saw evidence that the system being queried was fairly busy. No idle time was being reported; the CPU was spending all of its time between user and system tasks. Compare this with report, which shows the system is idle half the time.

$ dstat -a 2
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  2   4  94   0   0|7024B 5297B|   0     0 |   0     0 |  72    70
 14  36  50   0   0|   0     0 | 160B  809B|   0     0 | 381   229
 15  35  50   0   0|   0     0 | 238B  407B|   0     0 | 375   215
 16  34  50   0   0|   0     0 | 128B  346B|   0     0 | 369   204

The disks, on the other hand, are not busy at all with zero reads and writes.

One key to becoming adept at evaluating system performance is to run commands like these periodically — even when you don’t see the need to question how well a system is running. If  you come to know what normal performance looks like for a server, you will have a much easier time spotting problems.

Here’s another example, this one with some disk activity:

$ dstat -a 2 5
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  3   6  92   0   0|6631B 5133B|   0     0 |   0     0 |  89    79
 16  34  50   0   0|   0    16k| 270B  809B|   0     0 | 384   256
 16  34  50   0   0|   0     0 | 141B  407B|   0     0 | 358   207
 16  34  50   0   0|   0     0 | 238B  407B|   0     0 | 364   222
 15  35  50   0   0|2048B   18k| 350B  415B|   0     0 | 379   253

In all these samples, we’re not seeing any paging (loading executable images into a process’s virtual memory) activity. There is a fairly constant amount of interrupts and context switching going on, but the numbers are all quite modest.

In the command below, we’re looking at a memory usage report. Notice the amount of free memory compared to the memory in use. This system is not being challenged.

$ dstat -m 2 3
------memory-usage-----
 used  free  buff  cach
 372M 4659M  145M  681M
 373M 4659M  145M  681M
 373M 4659M  145M  681M

In the next command, we’re looking at an advanced memory usage report. Some additional memory statistics are provided.

$ dstat --mem-adv
-------------advanced-memory-usage-------------
total  used  free  buff  cach dirty shmem  recl
5960M  372M 4660M  144M  681M    0  1616k  104M
5960M  372M 4660M  144M  681M    0  1616k  104M
5960M  372M 4660M  144M  681M    0  1616k  104M
5960M  372M 4660M  144M  681M    0  1616k  104M^C

In this next command, we’re looking at open files and inodes in use.

$ dstat --fs
--filesystem-
files  inodes
 4704  73925
 4704  73925
 4704  73925
 4704  73925 ^C

In this last example, we’re generating the standard report, but adding one thing. We’re also writing the report to a .csv file so that it can be used in other tools such as Excel.

$ dstat --output /tmp/stats.csv -a 2 5
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  4  10  86   0   0|5918B   12k|   0     0 |   0     0 | 128    99
 18  32  50   0   0|   0     0 | 504B  923B|   0     0 | 377   237
 19  31  50   0   0|   0     0 | 355B  407B|   0     0 | 368   224
 15  36  50   0   0|   0    14k| 160B  407B|   0     0 | 372   227
 18  32  50   0   0|   0     0 | 270B  407B|   0     0 | 366   221

Here’s what the csv file looks like:

$ cat /tmp/stats.csv
"Dstat 0.7.3 CSV output"
"Author:","Dag Wieers >dag@wieers.com
What is dstat?
As mentioned, dstat is a great tool for looking at just about all aspects of system performance. But another answer to this question is that it's a Python script and one you're free to peruse if you'd like to see how it works.
$ which dstat
/usr/bin/dstat
$ file /usr/bin/dstat
/usr/bin/dstat: Python script, ASCII text executable
$ more /usr/bin/dstat | head -6
#!/usr/bin/env python2

### This program is free software; you can redistribute it and/or
### modify it under the terms of the GNU General Public License
### as published by the Free Software Foundation; either version 2
### of the License, or (at your option) any later version.