FIO - Flexible IO Tester Review

v0.2 This work is licensed under a Creative Commons License.

0. TOC and Purpose
1. Installation
2. Use
3. Assessment
4. Resources
5. Contact
6. Keywords
7. Disclaimer

0. TOC and Purpose

The purpose of this review is to review the install and use of FIO v1.36 to test disk IO.

1. Installation

Installation was not an ordeal.  I downloaded the source tarball from Freshmeat, and proceeded to untar and make.  My make failed on my test Ubuntu Desktop version 9.10, since I did not have the libaio-dev package.

After installing libaio-dev, I was able to make FIO and it was ready for use.  See compile and install below.  Note that to make install you need to have sudo privileges.

souljah@bluepunx:/usr/src/fio$ tar -zxvf fio-1.36.tar.gz 
souljah@bluepunx:/usr/src/fio/fio-1.36$ make
    CC gettime.o
In file included from fio.h:23,
                 from gettime.c:8:
os/os.h:15:20: error: libaio.h: No such file or directory
In file included from fio.h:29,
                 from gettime.c:8:
ioengine.h:19: error: field ‘iocb’ has incomplete type
make: *** [gettime.o] Error 1
souljah@bluepunx:/usr/src/fio/fio-1.36$ sudo apt-get install libaio-dev
souljah@bluepunx:/usr/src/fio/fio-1.36$ make
    CC gettime.o
    CC fio.o
    CC ioengines.o
    CC init.o
    CC stat.o
    CC log.o
    CC time.o
    CC filesetup.o
    CC eta.o
    CC verify.o
    CC memory.o
    CC io_u.o
    CC parse.o
    CC mutex.o
    CC options.o
    CC rbtree.o
    CC diskutil.o
    CC fifo.o
    CC blktrace.o
    CC smalloc.o
    CC filehash.o
    CC helpers.o
    CC cgroup.o
    CC crc/crc7.o
    CC crc/crc16.o
    CC crc/crc32.o
    CC crc/crc32c.o
    CC crc/crc32c-intel.o
    CC crc/crc64.o
    CC crc/sha1.o
    CC crc/sha256.o
    CC crc/sha512.o
    CC crc/md5.o
    CC engines/cpu.o
    CC engines/libaio.o
    CC engines/mmap.o
    CC engines/posixaio.o
    CC engines/sg.o
    CC engines/splice.o
    CC engines/sync.o
    CC engines/null.o
    CC engines/net.o
    CC engines/syslet-rw.o
    CC engines/guasi.o
    DEP depend
    CC fio
souljah@bluepunx:/usr/src/fio/fio-1.36$ make install
    DEP depend
    CC fio
install -m755 -d /usr/local/bin
install fio fio_generate_plots /usr/local/bin
install: cannot create regular file `/usr/local/bin/fio': Permission denied
install: cannot create regular file `/usr/local/bin/fio_generate_plots': Permission denied
make: *** [install] Error 1
souljah@bluepunx:/usr/src/fio/fio-1.36$ sudo make install
    DEP depend
    CC fio
install -m755 -d /usr/local/bin
install fio fio_generate_plots /usr/local/bin
install -m 755 -d /usr/local/man/man1
install -m 644 fio.1 /usr/local/man/man1
install -m 644 fio_generate_plots.1 /usr/local/man/man1

2. Use

While I am not an IO expert, I found the results fascinating.  I tested a variety of ioengines on my laptop, and desktop and found the results intriguing and useful in comparing hard drive performance.

This provides so much more granular data than the built-in utility hdparm, which only provides buffered reads and cached reads. Buffered reads provides an idea of how fast the disk reads under optimal linear reads without any filesystem overhead. The cache read provides an indication of data throughput of the cpu, cache, memory and bus of the system.

$ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:  156 MB in  3.01 seconds =  51.78 MB/sec

$ sudo hdparm -T /dev/sda1

/dev/sda1:
 Timing cached reads:   5756 MB in  1.99 seconds = 2886.59 MB/sec

The HOWTO included with the source outlined almost anything you could want to know about the program, and details the definitions related to IO testing.  This HOWTO is a definite must read for anyone trying to utilize FIO.

Example output:

souljah@bluepunx:/mnt/storage$ fio --name=global --rw=randrw --size=128m --name=job1 --name=job2
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
job2: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 2 processes
Jobs: 1 (f=1): [_m] [99.6% done] [516K/565K /s] [126/138 iops] [eta 00m:01s]
job1: (groupid=0, jobs=1): err= 0: pid=5117
  read : io=65880KB, bw=298486B/s, iops=72, runt=226011msec
    clat (usec): min=119, max=977960, avg=13487.89, stdev=37647.61
    bw (KB/s) : min=    4, max=  603, per=51.98%, avg=300.42, stdev=113.69
  write: io=65192KB, bw=295368B/s, iops=72, runt=226011msec
    clat (usec): min=10, max=196312, avg=219.16, stdev=4909.93
    bw (KB/s) : min=    4, max=  736, per=51.64%, avg=295.87, stdev=132.08
  cpu          : usr=0.05%, sys=0.57%, ctx=16529, majf=0, minf=555
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=16470/16298, short=0/0
     lat (usec): 20=44.48%, 50=4.92%, 100=0.21%, 250=0.79%, 500=0.07%
     lat (usec): 750=0.01%
     lat (msec): 2=0.58%, 4=10.53%, 10=33.08%, 20=2.33%, 50=0.56%
     lat (msec): 100=0.29%, 250=1.96%, 500=0.18%, 750=0.01%, 1000=0.01%
job2: (groupid=0, jobs=1): err= 0: pid=5118
  read : io=65716KB, bw=295669B/s, iops=72, runt=227596msec
    clat (usec): min=118, max=974169, avg=13608.29, stdev=38346.81
    bw (KB/s) : min=   19, max=  766, per=51.65%, avg=298.54, stdev=122.36
  write: io=65356KB, bw=294049B/s, iops=71, runt=227596msec
    clat (usec): min=9, max=171343, avg=229.40, stdev=5166.00
    bw (KB/s) : min=    5, max=  763, per=51.24%, avg=293.60, stdev=138.74
  cpu          : usr=0.07%, sys=0.54%, ctx=16489, majf=0, minf=1324
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=16429/16339, short=0/0
     lat (usec): 10=0.01%, 20=44.36%, 50=5.16%, 100=0.19%, 250=0.91%
     lat (usec): 500=0.05%, 750=0.02%, 1000=0.01%
     lat (msec): 2=0.64%, 4=10.23%, 10=33.26%, 20=2.38%, 50=0.41%
     lat (msec): 100=0.17%, 250=2.01%, 500=0.20%, 750=0.01%, 1000=0.02%

Run status group 0 (all jobs):
   READ: io=131596KB, aggrb=578KB/s, minb=295KB/s, maxb=298KB/s, mint=226011msec, maxt=227596msec
  WRITE: io=130548KB, aggrb=573KB/s, minb=294KB/s, maxb=295KB/s, mint=226011msec, maxt=227596msec

Disk stats (read/write):
  sdb: ios=32878/8303, merge=0/1095, ticks=445868/20400284, in_queue=20964472, util=99.98%

3. Assessment

FIO exceeded my expectations.  Compilation was simple, use is relatively intuitive, and the output is understandable.  I don't think you could ask for more from any product.  If you need to do any IO testing, this program should be in your toolbox.

My only compliant/suggestion would be to add an indication of the defaults for many of the run-time parameters which can be configured.  Without this, I'm not sure if I'm changing it from the default or specifying the default explicitly.  Additionally, it would be great to see a repository of results from end users.  This would provide invaluable relative information for those of use whom do not have access to a variety of hardware.

Either way, bravo to the author and maintainer Jens Axboe.  This is a fantastic project which provides a needed utility for the Unix community.

4. Resources

• Freshmeat project page for FIO.
• Git repo for FIO development.
• hdparm man page.

5. Contact

Thanks for reading. If you have any comments, suggestions or gripes please feel free to send them along. I'm always hungry for feedback. Thanks. Contact me via email at This e-mail address is being protected from spambots. You need JavaScript enabled to view it .

6. Keywords

Linux, Solaris, ,Ubuntu, Debian, IO, disk, disk IO, testing, baseline, review, async, sync, aio, libaio

7. Disclaimer

Obsecurities is furnishing this item "as is". Obsecurities does not provide any warranty of the item whatsoever, whether express, implied, or statutory, including, but not limited to, any warranty of merchantability or fitness for a particular purpose or any warranty that the contents of the item will be error-free.

In no respect shall obsecurities incur any liability for any damages, including, but limited to, direct, indirect, special, or consequential damages arising out of, resulting from, or any way connected to the use of the item, whether or not based upon warranty, contract, tort, or otherwise; whether or not injury was sustained by persons or property or otherwise; and whether or not loss was sustained from, or arose out of, the results of, the item, or any services that may be provided by Obsecurities.