Bug 837354 - gluster can't handle fio iometer simulation
gluster can't handle fio iometer simulation
Status: CLOSED WORKSFORME
Product: GlusterFS
Classification: Community
Component: core (Show other bugs)
3.3.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Raghavendra Bhat
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-03 11:23 EDT by Rob.Hendelman
Modified: 2012-09-26 02:45 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-26 02:45:07 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rob.Hendelman 2012-07-03 11:23:45 EDT
Description of problem:
When running the "fio" (1.38) from Ubuntu/natty packages, the program errors out ONLY when writing to a gluster mount.

Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Always

Steps to Reproduce:
1.Install an Ubuntu 11.04 System
2.apt-get update/upgrade
3.Install the gluster 3.3.0 final deb from gluster.org for Ubuntu 11.04
4.Create, start & mount a gluster replica filesystem
5.Mount the gluster filesystem from the server, as a client
6.apt-get install fio
7.copy /usr/share/doc/fio/examples/iometer-file-access-server to the gluster mount
8. cd to gluster mount & run "fio ./iometer-file-access-server" 
  
Actual results:

# fio ./iometer-file-access-server 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=libaio, iodepth=64
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
fio: pid=16213, err=22/file:filesetup.c:455, func=open(iometer.1.0), error=Invalid argument

Run status group 0 (all jobs):

Expected results:
# fio ./iometer-file-access-server 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=libaio, iodepth=64
Starting 1 process
Jobs: 1 (f=1): [m] [100.0% done] [3181K/697K /s] [543/126 iops] [eta 00m:00s]  
iometer: (groupid=0, jobs=1): err= 0: pid=16037
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3281MB, bw=4552KB/s, iops=596, runt=738192msec
    slat (usec): min=3, max=186810, avg=17.00, stdev=281.64
    clat (usec): min=33, max=1019K, avg=88752.43, stdev=77299.60
    bw (KB/s) : min= 1629, max=17954, per=100.16%, avg=4558.40, stdev=2269.12
  write: io=834509KB, bw=1130KB/s, iops=148, runt=738192msec
    slat (usec): min=4, max=290588, avg=23.44, stdev=876.82
    clat (usec): min=1, max=739207, avg=74488.51, stdev=76909.52
    bw (KB/s) : min=    0, max= 4693, per=99.60%, avg=1125.50, stdev=613.35
  cpu          : usr=0.74%, sys=2.33%, ctx=532547, majf=0, minf=1051
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued r/w: total=440001/109822, short=0/0
     lat (usec): 2=0.01%, 50=0.05%, 100=0.97%, 250=1.09%, 500=0.03%
     lat (usec): 750=0.03%, 1000=0.03%
     lat (msec): 2=0.18%, 4=0.64%, 10=4.05%, 20=7.65%, 50=22.06%
     lat (msec): 100=33.10%, 250=25.59%, 500=4.32%, 750=0.22%, 1000=0.01%
     lat (msec): 2000=0.01%

Run status group 0 (all jobs):
   READ: io=3281MB, aggrb=4551KB/s, minb=4661KB/s, maxb=4661KB/s, mint=738192msec, maxt=738192msec
  WRITE: io=834508KB, aggrb=1130KB/s, minb=1157KB/s, maxb=1157KB/s, mint=738192msec, maxt=738192msec

Disk stats (read/write):
  sda: ios=439892/109870, merge=16/2833, ticks=39030620/18786390, in_queue=58338910, util=100.00%

Additional info:
Ubuntu 11.04 Natty, kernel 2.6.38-15-generic x86_64
8GB ram
DL360 G7, P812 raid controller for gluster, P410i for OS.
Comment 1 Rob.Hendelman 2012-07-03 11:56:44 EDT
# fio --debug=all ./iometer-file-access-server 
fio: set all debug options
parse    16507 filling default options
parse    16507 handle_option=kb_base, ptr=1024
parse    16507 __handle_option=kb_base, type=7, ptr=1024
parse    16507 handle_option=lockfile, ptr=none
parse    16507 __handle_option=lockfile, type=1, ptr=none
parse    16507 handle_option=rw, ptr=read
parse    16507 __handle_option=rw, type=1, ptr=read
parse    16507 handle_option=ioengine, ptr=sync
parse    16507 __handle_option=ioengine, type=5, ptr=sync
parse    16507 handle_option=iodepth, ptr=1
parse    16507 __handle_option=iodepth, type=7, ptr=1
parse    16507 handle_option=iodepth_batch, ptr=1
parse    16507 __handle_option=iodepth_batch, type=7, ptr=1
parse    16507 handle_option=iodepth_batch_complete, ptr=1
parse    16507 __handle_option=iodepth_batch_complete, type=7, ptr=1
parse    16507 handle_option=fill_device, ptr=0
parse    16507 __handle_option=fill_device, type=8, ptr=0
parse    16507 handle_option=offset, ptr=0
parse    16507 __handle_option=offset, type=3, ptr=0
parse    16507 handle_option=bs, ptr=4k
parse    16507 __handle_option=bs, type=7, ptr=4k
parse    16507 handle_option=randrepeat, ptr=1
parse    16507 __handle_option=randrepeat, type=8, ptr=1
parse    16507 handle_option=softrandommap, ptr=0
parse    16507 __handle_option=softrandommap, type=8, ptr=0
parse    16507 handle_option=nrfiles, ptr=1
parse    16507 __handle_option=nrfiles, type=7, ptr=1
parse    16507 handle_option=file_service_type, ptr=roundrobin
parse    16507 __handle_option=file_service_type, type=1, ptr=roundrobin
parse    16507 handle_option=fallocate, ptr=1
parse    16507 __handle_option=fallocate, type=8, ptr=1
parse    16507 handle_option=fadvise_hint, ptr=1
parse    16507 __handle_option=fadvise_hint, type=8, ptr=1
parse    16507 handle_option=fsync, ptr=0
parse    16507 __handle_option=fsync, type=7, ptr=0
parse    16507 handle_option=fdatasync, ptr=0
parse    16507 __handle_option=fdatasync, type=7, ptr=0
parse    16507 handle_option=direct, ptr=0
parse    16507 __handle_option=direct, type=8, ptr=0
parse    16507 handle_option=buffered, ptr=1
parse    16507 __handle_option=buffered, type=8, ptr=1
parse    16507 handle_option=overwrite, ptr=0
parse    16507 __handle_option=overwrite, type=8, ptr=0
parse    16507 handle_option=loops, ptr=1
parse    16507 __handle_option=loops, type=7, ptr=1
parse    16507 handle_option=numjobs, ptr=1
parse    16507 __handle_option=numjobs, type=7, ptr=1
parse    16507 handle_option=startdelay, ptr=0
parse    16507 __handle_option=startdelay, type=7, ptr=0
parse    16507 handle_option=runtime, ptr=0
parse    16507 __handle_option=runtime, type=4, ptr=0
parse    16507 handle_option=mem, ptr=malloc
parse    16507 __handle_option=mem, type=1, ptr=malloc
parse    16507 handle_option=iomem_align, ptr=0
parse    16507 __handle_option=iomem_align, type=7, ptr=0
parse    16507 handle_option=verify, ptr=0
parse    16507 __handle_option=verify, type=1, ptr=0
parse    16507 handle_option=do_verify, ptr=1
parse    16507 __handle_option=do_verify, type=8, ptr=1
parse    16507 handle_option=verifysort, ptr=1
parse    16507 __handle_option=verifysort, type=8, ptr=1
parse    16507 handle_option=verify_offset, ptr=0
parse    16507 __handle_option=verify_offset, type=7, ptr=0
parse    16507 handle_option=verify_fatal, ptr=0
parse    16507 __handle_option=verify_fatal, type=8, ptr=0
parse    16507 handle_option=verify_async, ptr=0
parse    16507 __handle_option=verify_async, type=7, ptr=0
parse    16507 handle_option=zonesize, ptr=0
parse    16507 __handle_option=zonesize, type=3, ptr=0
parse    16507 handle_option=zoneskip, ptr=0
parse    16507 __handle_option=zoneskip, type=3, ptr=0
parse    16507 handle_option=lockmem, ptr=0
parse    16507 __handle_option=lockmem, type=3, ptr=0
parse    16507 handle_option=rwmixread, ptr=50
parse    16507 __handle_option=rwmixread, type=7, ptr=50
parse    16507 handle_option=rwmixwrite, ptr=50
parse    16507 __handle_option=rwmixwrite, type=7, ptr=50
parse    16507 handle_option=nice, ptr=0
parse    16507 __handle_option=nice, type=7, ptr=0
parse    16507 handle_option=thinktime, ptr=0
parse    16507 __handle_option=thinktime, type=7, ptr=0
parse    16507 handle_option=thinktime_spin, ptr=0
parse    16507 __handle_option=thinktime_spin, type=7, ptr=0
parse    16507 handle_option=thinktime_blocks, ptr=1
parse    16507 __handle_option=thinktime_blocks, type=7, ptr=1
parse    16507 handle_option=ratecycle, ptr=1000
parse    16507 __handle_option=ratecycle, type=7, ptr=1000
parse    16507 handle_option=invalidate, ptr=1
parse    16507 __handle_option=invalidate, type=8, ptr=1
parse    16507 handle_option=sync, ptr=0
parse    16507 __handle_option=sync, type=8, ptr=0
parse    16507 handle_option=bwavgtime, ptr=500
parse    16507 __handle_option=bwavgtime, type=7, ptr=500
parse    16507 handle_option=create_serialize, ptr=1
parse    16507 __handle_option=create_serialize, type=8, ptr=1
parse    16507 handle_option=create_fsync, ptr=1
parse    16507 __handle_option=create_fsync, type=8, ptr=1
parse    16507 handle_option=create_on_open, ptr=0
parse    16507 __handle_option=create_on_open, type=8, ptr=0
parse    16507 handle_option=pre_read, ptr=0
parse    16507 __handle_option=pre_read, type=8, ptr=0
parse    16507 handle_option=cpuchunks, ptr=50000
parse    16507 __handle_option=cpuchunks, type=7, ptr=50000
parse    16507 handle_option=end_fsync, ptr=0
parse    16507 __handle_option=end_fsync, type=8, ptr=0
parse    16507 handle_option=fsync_on_close, ptr=0
parse    16507 __handle_option=fsync_on_close, type=8, ptr=0
parse    16507 handle_option=unlink, ptr=0
parse    16507 __handle_option=unlink, type=8, ptr=0
parse    16507 handle_option=hugepage-size, ptr=2097152
parse    16507 __handle_option=hugepage-size, type=7, ptr=2097152
parse    16507 handle_option=disk_util, ptr=1
parse    16507 __handle_option=disk_util, type=8, ptr=1
parse    16507 handle_option=gtod_reduce, ptr=0
parse    16507 __handle_option=gtod_reduce, type=8, ptr=0
parse    16507 handle_option=disable_clat, ptr=0
parse    16507 __handle_option=disable_clat, type=8, ptr=0
parse    16507 handle_option=disable_slat, ptr=0
parse    16507 __handle_option=disable_slat, type=8, ptr=0
parse    16507 handle_option=disable_bw_measurement, ptr=0
parse    16507 __handle_option=disable_bw_measurement, type=8, ptr=0
parse    16507 handle_option=continue_on_error, ptr=0
parse    16507 __handle_option=continue_on_error, type=8, ptr=0
parse    16507 handle_option=block, ptr=4k
parse    16507 __handle_option=block, type=7, ptr=4k
parse    16507 handle_option=description, ptr=Emulation of Intel IOmeter File Server Access Pattern
parse    16507 __handle_option=description, type=5, ptr=Emulation of Intel IOmeter File Server Access Pattern
file     16507 dup files: 0
parse    16507 handle_option=bssplit, ptr=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
parse    16507 __handle_option=bssplit, type=1, ptr=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
parse    16507 handle_option=rw, ptr=randrw
parse    16507 __handle_option=rw, type=1, ptr=randrw
parse    16507 handle_option=rwmixread, ptr=80
parse    16507 __handle_option=rwmixread, type=7, ptr=80
parse    16507 handle_option=direct, ptr=1
parse    16507 __handle_option=direct, type=8, ptr=1
parse    16507 handle_option=size, ptr=4g
parse    16507 __handle_option=size, type=3, ptr=4g
parse    16507 handle_option=ioengine, ptr=libaio
parse    16507 __handle_option=ioengine, type=5, ptr=libaio
parse    16507 handle_option=iodepth, ptr=64
parse    16507 __handle_option=iodepth, type=7, ptr=64
io       16507 load ioengine libaio
file     16507 add file iometer.1.0
file     16507 resize file array to 2 files
file     16507 file 0x7f5b8b23c028 "iometer.1.0" added at 0
iometer: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=libaio, iodepth=64
Starting 1 process
file     16507 setup files
file     16507 get file size for 0x7f5b8b23c028/0/0x7f5b8b23c108
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
file     16507 layout unlink iometer.1.0
file     16507 open file iometer.1.0, flags 241
file     16507 fallocate file iometer.1.0 size 4294967296
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
file     16507 truncate file iometer.1.0, size 4294967296
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
diskutil 16507 update io ticks
io       16507 invalidate cache iometer.1.0: 0/4294967296
diskutil 16507 update io ticks
diskutil 16507 iometer.1.0 belongs to maj/min 0/20
process  16507 pid=0: runstate 0 -> 1
process  16507 will fork
mutex    16507 wait on startup_mutex
process  16512 jobs pid=16512 started
process  16512 pid=16512: runstate 1 -> 2
mutex    16512 up startup_mutex
mutex    16512 wait on td->mutex
mutex    16507 done waiting on startup_mutex
process  16507 pid=16512: runstate 2 -> 4
mutex    16512 done waiting on td->mutex
mem      16512 malloc 4198399 0x7f5b8a48d010
mem      16512 io_u alloc 0x15c6180, index 0
mem      16512 io_u 0x15c6180, mem 0x7f5b8a48e000
mem      16512 io_u alloc 0x15c6380, index 1
mem      16512 io_u 0x15c6380, mem 0x7f5b8a49e000
mem      16512 io_u alloc 0x15c6500, index 2
mem      16512 io_u 0x15c6500, mem 0x7f5b8a4ae000
mem      16512 io_u alloc 0x15c66c0, index 3
mem      16512 io_u 0x15c66c0, mem 0x7f5b8a4be000
mem      16512 io_u alloc 0x15c6840, index 4
mem      16512 io_u 0x15c6840, mem 0x7f5b8a4ce000
mem      16512 io_u alloc 0x15c6a40, index 5
mem      16512 io_u 0x15c6a40, mem 0x7f5b8a4de000
mem      16512 io_u alloc 0x15c6bc0, index 6
mem      16512 io_u 0x15c6bc0, mem 0x7f5b8a4ee000
mem      16512 io_u alloc 0x15c6d80, index 7
mem      16512 io_u 0x15c6d80, mem 0x7f5b8a4fe000
mem      16512 io_u alloc 0x15c6f00, index 8
mem      16512 io_u 0x15c6f00, mem 0x7f5b8a50e000
mem      16512 io_u alloc 0x15c7100, index 9
mem      16512 io_u 0x15c7100, mem 0x7f5b8a51e000
mem      16512 io_u alloc 0x15c7280, index 10
mem      16512 io_u 0x15c7280, mem 0x7f5b8a52e000
mem      16512 io_u alloc 0x15c7440, index 11
mem      16512 io_u 0x15c7440, mem 0x7f5b8a53e000
mem      16512 io_u alloc 0x15c7a00, index 12
mem      16512 io_u 0x15c7a00, mem 0x7f5b8a54e000
mem      16512 io_u alloc 0x15c7c00, index 13
mem      16512 io_u 0x15c7c00, mem 0x7f5b8a55e000
mem      16512 io_u alloc 0x15c7d80, index 14
mem      16512 io_u 0x15c7d80, mem 0x7f5b8a56e000
mem      16512 io_u alloc 0x15c7f40, index 15
mem      16512 io_u 0x15c7f40, mem 0x7f5b8a57e000
mem      16512 io_u alloc 0x15c80c0, index 16
mem      16512 io_u 0x15c80c0, mem 0x7f5b8a58e000
mem      16512 io_u alloc 0x15c82c0, index 17
mem      16512 io_u 0x15c82c0, mem 0x7f5b8a59e000
mem      16512 io_u alloc 0x15c8440, index 18
mem      16512 io_u 0x15c8440, mem 0x7f5b8a5ae000
mem      16512 io_u alloc 0x15c8600, index 19
mem      16512 io_u 0x15c8600, mem 0x7f5b8a5be000
mem      16512 io_u alloc 0x15c8780, index 20
mem      16512 io_u 0x15c8780, mem 0x7f5b8a5ce000
mem      16512 io_u alloc 0x15c8980, index 21
mem      16512 io_u 0x15c8980, mem 0x7f5b8a5de000
mem      16512 io_u alloc 0x15c8b00, index 22
mem      16512 io_u 0x15c8b00, mem 0x7f5b8a5ee000
mem      16512 io_u alloc 0x15c8cc0, index 23
mem      16512 io_u 0x15c8cc0, mem 0x7f5b8a5fe000
mem      16512 io_u alloc 0x15c8e40, index 24
mem      16512 io_u 0x15c8e40, mem 0x7f5b8a60e000
mem      16512 io_u alloc 0x15c9040, index 25
mem      16512 io_u 0x15c9040, mem 0x7f5b8a61e000
mem      16512 io_u alloc 0x15c91c0, index 26
mem      16512 io_u 0x15c91c0, mem 0x7f5b8a62e000
mem      16512 io_u alloc 0x15c9380, index 27
mem      16512 io_u 0x15c9380, mem 0x7f5b8a63e000
mem      16512 io_u alloc 0x15c9500, index 28
mem      16512 io_u 0x15c9500, mem 0x7f5b8a64e000
mem      16512 io_u alloc 0x15c9700, index 29
mem      16512 io_u 0x15c9700, mem 0x7f5b8a65e000
mem      16512 io_u alloc 0x15c9880, index 30
mem      16512 io_u 0x15c9880, mem 0x7f5b8a66e000
mem      16512 io_u alloc 0x15c9a40, index 31
mem      16512 io_u 0x15c9a40, mem 0x7f5b8a67e000
mem      16512 io_u alloc 0x15c9bc0, index 32
mem      16512 io_u 0x15c9bc0, mem 0x7f5b8a68e000
mem      16512 io_u alloc 0x15c9dc0, index 33
mem      16512 io_u 0x15c9dc0, mem 0x7f5b8a69e000
mem      16512 io_u alloc 0x15c9f40, index 34
mem      16512 io_u 0x15c9f40, mem 0x7f5b8a6ae000
mem      16512 io_u alloc 0x15ca100, index 35
mem      16512 io_u 0x15ca100, mem 0x7f5b8a6be000
mem      16512 io_u alloc 0x15ca280, index 36
mem      16512 io_u 0x15ca280, mem 0x7f5b8a6ce000
mem      16512 io_u alloc 0x15ca480, index 37
mem      16512 io_u 0x15ca480, mem 0x7f5b8a6de000
mem      16512 io_u alloc 0x15ca600, index 38
mem      16512 io_u 0x15ca600, mem 0x7f5b8a6ee000
mem      16512 io_u alloc 0x15ca7c0, index 39
mem      16512 io_u 0x15ca7c0, mem 0x7f5b8a6fe000
mem      16512 io_u alloc 0x15ca940, index 40
mem      16512 io_u 0x15ca940, mem 0x7f5b8a70e000
mem      16512 io_u alloc 0x15cab40, index 41
mem      16512 io_u 0x15cab40, mem 0x7f5b8a71e000
mem      16512 io_u alloc 0x15cacc0, index 42
mem      16512 io_u 0x15cacc0, mem 0x7f5b8a72e000
mem      16512 io_u alloc 0x15cae80, index 43
mem      16512 io_u 0x15cae80, mem 0x7f5b8a73e000
mem      16512 io_u alloc 0x15cb000, index 44
mem      16512 io_u 0x15cb000, mem 0x7f5b8a74e000
mem      16512 io_u alloc 0x15cb200, index 45
mem      16512 io_u 0x15cb200, mem 0x7f5b8a75e000
mem      16512 io_u alloc 0x15cb380, index 46
mem      16512 io_u 0x15cb380, mem 0x7f5b8a76e000
mem      16512 io_u alloc 0x15cb540, index 47
mem      16512 io_u 0x15cb540, mem 0x7f5b8a77e000
mem      16512 io_u alloc 0x15cb6c0, index 48
mem      16512 io_u 0x15cb6c0, mem 0x7f5b8a78e000
mem      16512 io_u alloc 0x15cb8c0, index 49
mem      16512 io_u 0x15cb8c0, mem 0x7f5b8a79e000
mem      16512 io_u alloc 0x15cba40, index 50
mem      16512 io_u 0x15cba40, mem 0x7f5b8a7ae000
mem      16512 io_u alloc 0x15cbc00, index 51
mem      16512 io_u 0x15cbc00, mem 0x7f5b8a7be000
mem      16512 io_u alloc 0x15cbd80, index 52
mem      16512 io_u 0x15cbd80, mem 0x7f5b8a7ce000
mem      16512 io_u alloc 0x15cbf80, index 53
mem      16512 io_u 0x15cbf80, mem 0x7f5b8a7de000
mem      16512 io_u alloc 0x15cc100, index 54
mem      16512 io_u 0x15cc100, mem 0x7f5b8a7ee000
mem      16512 io_u alloc 0x15cc2c0, index 55
mem      16512 io_u 0x15cc2c0, mem 0x7f5b8a7fe000
mem      16512 io_u alloc 0x15cc440, index 56
mem      16512 io_u 0x15cc440, mem 0x7f5b8a80e000
mem      16512 io_u alloc 0x15cc640, index 57
mem      16512 io_u 0x15cc640, mem 0x7f5b8a81e000
mem      16512 io_u alloc 0x15cc7c0, index 58
mem      16512 io_u 0x15cc7c0, mem 0x7f5b8a82e000
mem      16512 io_u alloc 0x15cc980, index 59
mem      16512 io_u 0x15cc980, mem 0x7f5b8a83e000
mem      16512 io_u alloc 0x15ccb00, index 60
mem      16512 io_u 0x15ccb00, mem 0x7f5b8a84e000
mem      16512 io_u alloc 0x15ccd00, index 61
mem      16512 io_u 0x15ccd00, mem 0x7f5b8a85e000
mem      16512 io_u alloc 0x15cce80, index 62
mem      16512 io_u 0x15cce80, mem 0x7f5b8a86e000
mem      16512 io_u alloc 0x15cd040, index 63
mem      16512 io_u 0x15cd040, mem 0x7f5b8a87e000
file     16512 trying file iometer.1.0 0
file     16512 fd open iometer.1.0
file     16512 file not found in hash iometer.1.0
file     16512 get file iometer.1.0, ref=0
io       16512 invalidate cache iometer.1.0: 0/4294967296
file     16512 fd close iometer.1.0
file     16512 error 1 on open of iometer.1.0
file     16512 get_next_file_rr: (nil)
fio: pid=16512, got signal=11
process  16507 pid=16512: runstate 4 -> 9
process  16507 terminate group_id=-1
process  16507 setting terminate on iometer/16512
diskutil 16507 update io ticks

Run status group 0 (all jobs):
io       16507 ioengine cpuio unregistered
io       16507 ioengine libaio unregistered
io       16507 ioengine mmap unregistered
io       16507 ioengine posixaio unregistered
io       16507 ioengine sg unregistered
io       16507 ioengine splice unregistered
io       16507 ioengine sync unregistered
io       16507 ioengine psync unregistered
io       16507 ioengine vsync unregistered
io       16507 ioengine null unregistered
io       16507 ioengine net unregistered
io       16507 ioengine netsplice unregistered
io       16507 ioengine syslet-rw unregistered
io       16507 ioengine guasi unregistered
profile  16507 unregister profile 'tiobench'
Comment 2 Niels de Vos 2012-07-03 12:01:43 EDT
Could you try with a newer version of fio? I do not seem to hit any error with fio-2.07.

Updated versions of fio might be available from links listed in the README (http://git.kernel.dk/?p=fio.git;a=blob;f=README;hb=HEAD).

Thanks!
Comment 3 Rob.Hendelman 2012-07-03 14:18:24 EDT
Same result...

# pwd
/mnt/gluster/client
# /usr/src/fio/fio --version
fio-2.0.8-9-gfb9f0
# /usr/src/fio/fio ./iometer-file-access-server 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.0.8-9-gfb9f0
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
fio: looks like your file system does not support direct=1/buffered=0
fio: pid=18441, err=22/file:filesetup.c:534, func=open(iometer.1.0), error=Invalid argument


Run status group 0 (all jobs):
Comment 4 Anand Avati 2012-07-03 17:29:22 EDT
(In reply to comment #3)
> Same result...
> 
> # pwd
> /mnt/gluster/client
> # /usr/src/fio/fio --version
> fio-2.0.8-9-gfb9f0
> # /usr/src/fio/fio ./iometer-file-access-server 
> iometer: (g=0): rw=randrw, bs=512-64K/512-64K, ioengine=libaio, iodepth=64
> fio-2.0.8-9-gfb9f0
> Starting 1 process
> iometer: Laying out IO file(s) (1 file(s) / 4096MB)
> fio: looks like your file system does not support direct=1/buffered=0
> fio: pid=18441, err=22/file:filesetup.c:534, func=open(iometer.1.0),
> error=Invalid argument
> 
> 
> Run status group 0 (all jobs):

Correllating the 'direct=i' and 'error=Invalid argument' I am guessing it is due to the lack of O_DIRECT support in FUSE which is resulting in EINVAL (Invalid Argument) error -- which is expected. O_DIRECT support in FUSE was merged in 3.5 (I think). It should also be available in RHEL 6.2z
Comment 5 Raghavendra Bhat 2012-07-12 07:46:02 EDT
Executed fio on a replicate volume got the same error of EINVAL. Changed the configuration file of iometer-file-access-server by setting direct=0 (i.e disabling O_DIRECT). Now fio executed correctly. It seems to be a O_DIRECT issue of fuse which has got fixed in higher kernel version.
Comment 6 Amar Tumballi 2012-09-26 02:45:07 EDT
The problem is evidently the O_DIRECT issue on fuse driver. Which is fixed in linux kernel 3.4 or higher (and is backported to many distributions).

http://lwn.net/Articles/476978/ 
https://lkml.org/lkml/2012/4/18/301

Note You need to log in before you can comment on or make changes to this bug.