Bug 1656507

Summary: [RHEL.8] qcow2 cache is too small
Product: Red Hat Enterprise Linux 8 Reporter: Kevin Wolf <kwolf>
Component: qemu-kvmAssignee: Kevin Wolf <kwolf>
Status: CLOSED CURRENTRELEASE QA Contact: Tingting Mao <timao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0CC: chayang, coli, ddepaula, eblake, juzhang, kwolf, rbalakri, timao, virt-maint
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-2.12.0-46.module+el8+2351+e14a4632 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1656913 (view as bug list) Environment:
Last Closed: 2019-06-14 00:52:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1656913    

Description Kevin Wolf 2018-12-05 16:29:40 UTC 
The qcow2 metadata cache is too small for some use cases (covering only a 8 GB region for random I/O). Originally, the idea was that management tools would set the cache size options, but this has never happened. Upstream recently switched to a more intelligent default at the QEMU level that can make a big difference.


The difference is easily demonstrated:

1. Create an image that is noticeably larger than 8G (with 64k clusters)
   $ qemu-img create -f qcow2 /tmp/test.qcow2 32G

2. Initialise the image with some data
   $ qemu-img bench -w -c 1000 -S 512M /tmp/test.qcow2

3. Benchmark reads with 512 MB steps between each request:
   $ qemu-img bench -c 1000000 -S 512M /tmp/test.qcow2


On RHEL qemu-kvm (qcow2):

$ qemu-img bench -c 1000000 -S 512M /tmp/test.qcow2
Sending 1000000 read requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 536870912)
Run completed in 20.913 seconds.


On RHEL qemu-kvm (raw):

$ qemu-img bench -c 1000000 -S 512M /tmp/test.raw 
Sending 1000000 read requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 536870912)
Run completed in 2.572 seconds.


On upstream qemu (qcow2):
$ qemu-img bench -c 1000000 -S 512M /tmp/test.qcow2
Sending 1000000 read requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 536870912)
Run completed in 1.571 seconds.


The difference should also be very visible in fio, with random I/O across a 32 GB image.
Comment 2 Danilo de Paula 2018-12-11 16:11:18 UTC 
Fix included in qemu-kvm-2.12.0-46.module+el8+2351+e14a4632
Comment 8 Tingting Mao 2019-01-09 08:43:14 UTC 
Hi Kevin,

After doing i/o performance compare(qemu-kvm-2.12.0-45.* VS qemu-kvm-2.12.0-52.*) with fio, random read improved a lot(IOPS=1085 VS IOPS=24.3k), while random write seems not changed(IOPS=15.7k VS IOPS=15.7k). And for random read and write simultaneously, it improved(IOPS=333(r)/333(w) VS IOPS=899(r)/898(w)).

So could you please help to check whether the result is okay?



In qemu-kvm-2.12.0-52.module+el8+2617+8f1b8f17

Steps:

1. Create qcow2 file
# qemu-img create -f qcow2 data.qcow2 40G

2. Boot guest with qcow2 file as data disk
# /usr/libexec/qemu-kvm \
        -name 'guest-rhel8.0' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -object secret,id=sec0,data=base \
        -device virtio-blk-pci,id=virtio_blk_pci0,drive=my_file,bus=pcie.0,addr=05,bootindex=0 \
        -blockdev driver=luks,cache.direct=off,cache.no-flush=on,node-name=my_file,file.driver=file,file.filename=base.luks,key-secret=sec0 \
        -device virtio-blk-pci,id=virtio_blk_pci1,drive=my_file1,bus=pcie.0,addr=06 \
        -blockdev driver=qcow2,cache.direct=off,cache.no-flush=on,node-name=my_file1,file.driver=file,file.filename=data.qcow2 \
        -vnc :0 \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \

3. Random read/write for the disk file in guest
# lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda           252:0    0  20G  0 disk
├─vda1        252:1    0   1G  0 part /boot
└─vda2        252:2    0  19G  0 part
  ├─rhel-root 253:0    0  17G  0 lvm  /
  └─rhel-swap 253:1    0   2G  0 lvm  [SWAP]
vdb           252:16   0  40G  0 disk

3.1 Write
# fio --filename=/dev/vdb --direct=1 --rw=randwrite --bs=8K --name=my_test --iodepth=1 --ioengine=libaio --size=5G
my_test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=133MiB/s][r=0,w=17.1k IOPS][eta 00m:00s]
my_test: (groupid=0, jobs=1): err= 0: pid=16341: Tue Jan  8 21:53:53 2019
  write: IOPS=15.7k, BW=123MiB/s (129MB/s)(5120MiB/41730msec)
    slat (nsec): min=6866, max=137472, avg=8076.88, stdev=709.06
    clat (usec): min=25, max=1385, avg=50.97, stdev=14.23
     lat (usec): min=39, max=1393, avg=60.05, stdev=14.30
    clat percentiles (usec):
     |  1.00th=[   39],  5.00th=[   41], 10.00th=[   43], 20.00th=[   45],
     | 30.00th=[   46], 40.00th=[   46], 50.00th=[   47], 60.00th=[   48],
     | 70.00th=[   49], 80.00th=[   51], 90.00th=[   67], 95.00th=[   90],
     | 99.00th=[  105], 99.50th=[  110], 99.90th=[  125], 99.95th=[  137],
     | 99.99th=[  165]
   bw (  KiB/s): min=90016, max=150064, per=99.97%, avg=125595.06, stdev=13188.49, samples=83
   iops        : min=11252, max=18758, avg=15699.37, stdev=1648.56, samples=83
  lat (usec)   : 50=78.24%, 100=18.59%, 250=3.17%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=7.62%, sys=18.97%, ctx=655364, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,655360,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=5120MiB (5369MB), run=41730-41730msec

Disk stats (read/write):
  vdb: ios=50/653992, merge=0/0, ticks=4/34463, in_queue=2, util=99.81%

3.2 Read
# fio --filename=/dev/vdb --direct=1 --rw=randread --bs=8K --name=my_test --iodepth=1 --ioengine=libaio --size=20G
my_test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=188MiB/s,w=0KiB/s][r=24.1k,w=0 IOPS][eta 00m:00s]
my_test: (groupid=0, jobs=1): err= 0: pid=16346: Tue Jan  8 21:56:04 2019
   read: IOPS=24.3k, BW=190MiB/s (199MB/s)(20.0GiB/107895msec)
    slat (nsec): min=6585, max=83173, avg=7869.84, stdev=758.14
    clat (usec): min=2, max=666, avg=28.73, stdev=10.24
     lat (usec): min=23, max=675, avg=37.57, stdev=10.29
    clat percentiles (usec):
     |  1.00th=[   20],  5.00th=[   22], 10.00th=[   22], 20.00th=[   22],
     | 30.00th=[   22], 40.00th=[   23], 50.00th=[   24], 60.00th=[   26],
     | 70.00th=[   27], 80.00th=[   44], 90.00th=[   46], 95.00th=[   47],
     | 99.00th=[   50], 99.50th=[   52], 99.90th=[   65], 99.95th=[   77],
     | 99.99th=[  229]
   bw (  KiB/s): min=187216, max=206256, per=100.00%, avg=194363.89, stdev=2941.39, samples=215
   iops        : min=23402, max=25782, avg=24295.48, stdev=367.70, samples=215
  lat (usec)   : 4=0.01%, 10=0.01%, 20=1.27%, 50=97.94%, 100=0.77%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=11.61%, sys=28.54%, ctx=2621408, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=190MiB/s (199MB/s), 190MiB/s-190MiB/s (199MB/s-199MB/s), io=20.0GiB (21.5GB), run=107895-107895msec

Disk stats (read/write):
  vdb: ios=2615536/0, merge=0/0, ticks=79767/0, in_queue=0, util=99.94%

3.3 Read and write
# fio --filename=/dev/vdb --direct=1 --rw=randrw --bs=8K --name=my_test --iodepth=1 --ioengine=libaio --size=32G
my_test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=6632KiB/s,w=6672KiB/s][r=829,w=834 IOPS][eta 00m:01s]
my_test: (groupid=0, jobs=1): err= 0: pid=16372: Tue Jan  8 22:35:25 2019
   read: IOPS=899, BW=7196KiB/s (7369kB/s)(16.0GiB/2332682msec)
    slat (usec): min=6, max=164, avg=15.16, stdev= 7.97
    clat (usec): min=6, max=3347.2k, avg=968.93, stdev=10998.93
     lat (usec): min=27, max=3347.2k, avg=986.00, stdev=10999.67
    clat percentiles (usec):
     |  1.00th=[    22],  5.00th=[    25], 10.00th=[    26], 20.00th=[    42],
     | 30.00th=[    46], 40.00th=[    48], 50.00th=[    66], 60.00th=[    97],
     | 70.00th=[   105], 80.00th=[   116], 90.00th=[   135], 95.00th=[  2737],
     | 99.00th=[ 15270], 99.50th=[ 37487], 99.90th=[149947], 99.95th=[189793],
     | 99.99th=[256902]
   bw (  KiB/s): min=   32, max=69936, per=100.00%, avg=7329.78, stdev=13684.44, samples=4579
   iops        : min=    4, max= 8742, avg=916.21, stdev=1710.56, samples=4579
  write: IOPS=898, BW=7188KiB/s (7361kB/s)(15.0GiB/2332682msec)
    slat (usec): min=6, max=130, avg=15.62, stdev= 8.01
    clat (usec): min=25, max=1607.2k, avg=94.15, stdev=1474.30
     lat (usec): min=41, max=1607.2k, avg=111.68, stdev=1474.48
    clat percentiles (usec):
     |  1.00th=[   42],  5.00th=[   45], 10.00th=[   47], 20.00th=[   49],
     | 30.00th=[   57], 40.00th=[   89], 50.00th=[  100], 60.00th=[  105],
     | 70.00th=[  111], 80.00th=[  120], 90.00th=[  139], 95.00th=[  147],
     | 99.00th=[  204], 99.50th=[  215], 99.90th=[  235], 99.95th=[  241],
     | 99.99th=[  265]
   bw (  KiB/s): min=   48, max=70272, per=100.00%, avg=7322.09, stdev=13675.15, samples=4579
   iops        : min=    6, max= 8784, avg=915.25, stdev=1709.39, samples=4579
  lat (usec)   : 10=0.01%, 20=0.03%, 50=33.83%, 100=23.33%, 250=39.80%
  lat (usec)   : 500=0.04%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.22%, 4=0.72%, 10=1.34%, 20=0.28%, 50=0.22%
  lat (msec)   : 100=0.10%, 250=0.09%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.85%, sys=4.06%, ctx=4194357, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2098266,2096038,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=7196KiB/s (7369kB/s), 7196KiB/s-7196KiB/s (7369kB/s-7369kB/s), io=16.0GiB (17.2GB), run=2332682-2332682msec
  WRITE: bw=7188KiB/s (7361kB/s), 7188KiB/s-7188KiB/s (7361kB/s-7361kB/s), io=15.0GiB (17.2GB), run=2332682-2332682msec

Disk stats (read/write):
  vdb: ios=2098132/2095869, merge=0/0, ticks=2038196/202929, in_queue=1831834, util=24.13%



In qemu-kvm-2.12.0-45.module+el8+2313+d65431a0, same qcow2 data disk file and same boot scripts. The test result is listed like below:

Do random read/write on the data disk
 
1.1 Write: 
# fio --filename=/dev/vdb --direct=1 --rw=randwrite --bs=8K --name=my_test --iodepth=1 --ioengine=libaio --size=5G
my_test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=122MiB/s][r=0,w=15.6k IOPS][eta 00m:00s]
my_test: (groupid=0, jobs=1): err= 0: pid=1723: Wed Jan  9 00:13:39 2019
  write: IOPS=15.7k, BW=122MiB/s (128MB/s)(5120MiB/41816msec)
    slat (nsec): min=6902, max=92545, avg=8300.00, stdev=862.23
    clat (usec): min=17, max=1158.2k, avg=50.84, stdev=1430.64
     lat (usec): min=42, max=1158.2k, avg=60.14, stdev=1430.64
    clat percentiles (usec):
     |  1.00th=[   38],  5.00th=[   45], 10.00th=[   45], 20.00th=[   47],
     | 30.00th=[   48], 40.00th=[   49], 50.00th=[   49], 60.00th=[   50],
     | 70.00th=[   50], 80.00th=[   51], 90.00th=[   53], 95.00th=[   56],
     | 99.00th=[   62], 99.50th=[   71], 99.90th=[   91], 99.95th=[   97],
     | 99.99th=[  137]
   bw (  KiB/s): min=98928, max=138496, per=100.00%, avg=128453.20, stdev=5458.55, samples=81
   iops        : min=12366, max=17312, avg=16056.63, stdev=682.32, samples=81
  lat (usec)   : 20=0.01%, 50=70.61%, 100=29.35%, 250=0.04%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=8.21%, sys=19.16%, ctx=655364, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,655360,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=5120MiB (5369MB), run=41816-41816msec

Disk stats (read/write):
  vdb: ios=50/652665, merge=0/0, ticks=5/34251, in_queue=1168, util=97.03%

1.2 Read:
# fio --filename=/dev/vdb --direct=1 --rw=randread --bs=8K --name=my_test --iodepth=1 --ioengine=libaio --size=20G
my_test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=12.5MiB/s,w=0KiB/s][r=1600,w=0 IOPS][eta 00m:00s]
my_test: (groupid=0, jobs=1): err= 0: pid=1735: Wed Jan  9 01:02:24 2019
   read: IOPS=1085, BW=8687KiB/s (8896kB/s)(20.0GiB/2413991msec)
    slat (usec): min=8, max=100, avg=23.63, stdev= 3.55
    clat (usec): min=46, max=180618, avg=883.41, stdev=2209.98
     lat (usec): min=58, max=180643, avg=910.06, stdev=2210.05
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[  104], 10.00th=[  112], 20.00th=[  126],
     | 30.00th=[  137], 40.00th=[  167], 50.00th=[  178], 60.00th=[  186],
     | 70.00th=[  192], 80.00th=[  202], 90.00th=[ 3589], 95.00th=[ 5669],
     | 99.00th=[ 7963], 99.50th=[10683], 99.90th=[18482], 99.95th=[25560],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1088, max=14656, per=100.00%, avg=8686.86, stdev=2884.55, samples=4827
   iops        : min=  136, max= 1832, avg=1085.85, stdev=360.57, samples=4827
  lat (usec)   : 50=0.01%, 100=3.79%, 250=80.54%, 500=0.57%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=1.28%, 4=4.77%, 10=8.43%, 20=0.52%, 50=0.08%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=1.77%, sys=3.83%, ctx=2621503, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=8687KiB/s (8896kB/s), 8687KiB/s-8687KiB/s (8896kB/s-8896kB/s), io=20.0GiB (21.5GB), run=2413991-2413991msec

Disk stats (read/write):
  vdb: ios=2621279/0, merge=0/0, ticks=2324096/0, in_queue=1763815, util=35.12%


1.3 Write and read:
# fio --filename=/dev/vdb --direct=1 --rw=randrw --bs=8K --name=my_test --iodepth=1 --ioengine=libaio --size=32G
my_test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=3664KiB/s,w=3760KiB/s][r=458,w=470 IOPS][eta 00m:01s]   
my_test: (groupid=0, jobs=1): err= 0: pid=1758: Wed Jan  9 02:48:14 2019
   read: IOPS=333, BW=2667KiB/s (2731kB/s)(16.0GiB/6293551msec)
    slat (usec): min=8, max=160, avg=24.19, stdev= 3.71
    clat (usec): min=41, max=2106.8k, avg=2742.37, stdev=20326.85
     lat (usec): min=66, max=2106.8k, avg=2769.53, stdev=20326.88
    clat percentiles (usec):
     |  1.00th=[    97],  5.00th=[   115], 10.00th=[   125], 20.00th=[   157],
     | 30.00th=[   176], 40.00th=[   184], 50.00th=[   192], 60.00th=[   198],
     | 70.00th=[   215], 80.00th=[  3589], 90.00th=[  5997], 95.00th=[  7242],
     | 99.00th=[ 34341], 99.50th=[ 74974], 99.90th=[193987], 99.95th=[229639],
     | 99.99th=[784335]
   bw (  KiB/s): min=   16, max= 7056, per=100.00%, avg=2767.99, stdev=1946.63, samples=12128
   iops        : min=    2, max=  882, avg=345.98, stdev=243.33, samples=12128
  write: IOPS=333, BW=2664KiB/s (2728kB/s)(15.0GiB/6293551msec)
    slat (usec): min=8, max=145, avg=24.94, stdev= 3.80
    clat (usec): min=44, max=93980, avg=179.18, stdev=85.89
     lat (usec): min=71, max=94007, avg=207.11, stdev=86.07
    clat percentiles (usec):
     |  1.00th=[  102],  5.00th=[  120], 10.00th=[  126], 20.00th=[  143],
     | 30.00th=[  174], 40.00th=[  184], 50.00th=[  190], 60.00th=[  196],
     | 70.00th=[  200], 80.00th=[  206], 90.00th=[  212], 95.00th=[  219],
     | 99.00th=[  229], 99.50th=[  233], 99.90th=[  247], 99.95th=[  265],
     | 99.99th=[  343]
   bw (  KiB/s): min=   16, max= 7072, per=100.00%, avg=2767.79, stdev=1963.47, samples=12116
   iops        : min=    2, max=  884, avg=345.96, stdev=245.43, samples=12116
  lat (usec)   : 50=0.01%, 100=1.04%, 250=84.98%, 500=0.25%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.63%, 4=3.95%, 10=7.72%, 20=0.68%, 50=0.39%
  lat (msec)   : 100=0.17%, 250=0.17%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.15%, sys=2.39%, ctx=4194381, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2098266,2096038,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2667KiB/s (2731kB/s), 2667KiB/s-2667KiB/s (2731kB/s-2731kB/s), io=16.0GiB (17.2GB), run=6293551-6293551msec
  WRITE: bw=2664KiB/s (2728kB/s), 2664KiB/s-2664KiB/s (2728kB/s-2728kB/s), io=15.0GiB (17.2GB), run=6293551-6293551msec

Disk stats (read/write):
  vdb: ios=2098306/2096035, merge=0/0, ticks=5761079/383578, in_queue=5207204, util=21.86%
Comment 10 Kevin Wolf 2019-01-11 09:24:37 UTC 
(In reply to Tingting Mao from comment #8)
> After doing i/o performance compare(qemu-kvm-2.12.0-45.* VS
> qemu-kvm-2.12.0-52.*) with fio, random read improved a lot(IOPS=1085 VS
> IOPS=24.3k), while random write seems not changed(IOPS=15.7k VS IOPS=15.7k).
> And for random read and write simultaneously, it improved(IOPS=333(r)/333(w)
> VS IOPS=899(r)/898(w)).
> 
> So could you please help to check whether the result is okay?

These results looked a bit strange, and I think I found two things that will lead to more meaningful benchmark results:

- The cache mode you used is not well-suited for benchmarks. It is what used to be cache=unsafe. Real-life cases use cache=none, i.e. cache.direct=on,cache=no-flush=off. This will result is more stable results, too.

- We don't want to measure time spent for cluster allocation, or time to read from unallocated clusters . Both will affect the benchmark result: For writes, the allocation cost will dominate, so we will get low results anyway and the cache size changes will likely be invisible. For reads, we get too good results because we don't actually have to access the disk. To fix this, you can write the whole disk before running fio, e.g. with dd if=/dev/zero of=/dev/vdb.