Bug 1272793

oVirt is using the iotune API in libvirt for QoS:
https://libvirt.org/formatdomain.html#elementsDisks

Does it support Gluster?

It's implemented using "block_set_io_throttle" QMP command and whether this is supported for Gluster disks is more a question for QEMU developers.

(In reply to Jiri Denemark from comment #2)
> It's implemented using "block_set_io_throttle" QMP command and whether this
> is supported for Gluster disks is more a question for QEMU developers.

Jeff, can you please check?

It should be supported - the throttle groups are independent of the specific block driver, and attached to the BlockDriverState.

As a follow-up:  I did a quick check with my local build of qemu for rhev 7.1, with a gluster server.  Varying the bps, and iops totals via block_set_io_throttle seemed to behave as expected, when running just qemu and feeding it the qmp commands via stdio.  I tested using dd with oflag=dsync, however, rather than fio.

I'll do another more in-depth test with fio, and report the results.

(In reply to Jeff Cody from comment #5)
> As a follow-up:  I did a quick check with my local build of qemu for rhev
> 7.1, with a gluster server.  Varying the bps, and iops totals via
> block_set_io_throttle seemed to behave as expected, when running just qemu
> and feeding it the qmp commands via stdio.  I tested using dd with
> oflag=dsync, however, rather than fio.
> 
> I'll do another more in-depth test with fio, and report the results.

Could this be something to do with the nature of the profile? dd is pure sequential - I'm using random rw.

Moving to libvirt for further investigation.

(In reply to Doron Fediuck from comment #8)
> Moving to libvirt for further investigation.

Moving to qemu, see comment 2.

(In reply to Jiri Denemark from comment #2)
> It's implemented using "block_set_io_throttle" QMP command and whether this
> is supported for Gluster disks is more a question for QEMU developers.

Paul, Can you somehow show that the QMP command is making it to QEMU? 

Jeff, Have you run the test with fio? Did you test with 7.1 or 7.2? The original report was 7.1. Thanks!

(In reply to Karen Noel from comment #11)
> (In reply to Jiri Denemark from comment #2)
> > It's implemented using "block_set_io_throttle" QMP command and whether this
> > is supported for Gluster disks is more a question for QEMU developers.
> 
> Paul, Can you somehow show that the QMP command is making it to QEMU? 
> 
> Jeff, Have you run the test with fio? Did you test with 7.1 or 7.2? The
> original report was 7.1. Thanks!

I'm running the test with fio now - I needed to add additional space to my gluster test server (I only had ~200M, I want to run fio with ~4GB).

I tested against 7.1.  I'll update the BZ with my fio results shortly.

Here is the version of qemu I used for my testing:

qemu version:
qemu-kvm-rhev-2.1.2-23.el7_1_1.10


qemu commandline:
qemu-system-x86_64 -enable-kvm -drive file=F18Test.qcow2,if=virtio,boot=on,node-name="test",aio=native,cache=none -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device virtio-serial -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -m 1024 -boot c -qmp stdio -netdev tap,id=br0,vhost=on,ifname=tap1,script=no,downscript=no  -device virtio-net-pci,mac=02:12:34:56:78:9c,netdev=br0 -drive file=gluster://192.168.15.99/gv0/test.qcow2,if=virtio,aio=native,cache=none


fio commandline:
# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=64M --readwrite=randrw --rwmixread=75


Test run, no throttling:
  read : io=49000KB, bw=804450 B/s, iops=196 , runt= 62373msec
  write: io=16536KB, bw=271477 B/s, iops=66 , runt= 62373msec

Throttle via QMP:
{"execute": "block_set_io_throttle", "arguments": {"device": "virtio1", "bps": 0, "bps_rd": 0, "bps_wr": 0, "iops": 0, "iops_rd": 15, "iops_wr": 15} }

Test run, with throttling:
  read : io=49204KB, bw=61453 B/s, iops=15 , runt=819888msec
  write: io=16332KB, bw=20397 B/s, iops=4 , runt=819888msec

Peter, Paul:
Are there any logs that show the QEMU commands issued?  As described in comment #13, it looks to me as if it should work from QEMU.  I'd like to verify that 1., the command is being sent to QEMU, and 2., what the actual qapi command is that is sent.  Thanks!

(In reply to Jeff Cody from comment #14)
> Peter, Paul:
> Are there any logs that show the QEMU commands issued?  As described in
> comment #13, it looks to me as if it should work from QEMU.  I'd like to
> verify that 1., the command is being sent to QEMU, and 2., what the actual
> qapi command is that is sent.  Thanks!

I'll take a look at the vdsm logs tomorrow.

Just picking up on this with the latest rhev 3.6 beta.

Several observations

1) When the default disk profile has iops limitation imposed, I can confirm that the iops limit (for example), is visible on the qemu invocation (e.g. aio=threads,iops=100)

2) I ran an fio run and confirmed that the iops limitation set by the profile was being adhered to - which is great.

3) In the UI when I create a new QoS, and then add a disk profile to a storage domain that uses it I can't seem to apply it to a vdisk. When I select a vm, and edit one of the disks, I can select the new disk profile to assign from the pulldown and click 'ok'...BUT, when I return to the disk properties the original disk profile setting is still in place.

(In reply to Jeff Cody from comment #14)
> Peter, Paul:
> Are there any logs that show the QEMU commands issued?  As described in
> comment #13, it looks to me as if it should work from QEMU.  I'd like to
> verify that 1., the command is being sent to QEMU, and 2., what the actual
> qapi command is that is sent.  Thanks!

AFAICS, we still don't have these logs... Paul?

I'll get the info to you as soon as I can. My test environment has been hosed by network changes - I need that to be resolved before I can get into this.

Anecdotally, what I'm seeing since 3.6GA is when the limit is placed in the unlimited profile - I see the additional parameters on the qemu process (e.g. aio=threads,iops=200). I'll confirm as soon as I can

However, if I try to change the settings to introduce a different profile it doesn't work - this is covered by BZ 1297734

(In reply to Jeff Cody from comment #13)
> Here is the version of qemu I used for my testing:
> 
> qemu version:
> qemu-kvm-rhev-2.1.2-23.el7_1_1.10
> 
> 
> qemu commandline:
> qemu-system-x86_64 -enable-kvm -drive
> file=F18Test.qcow2,if=virtio,boot=on,node-name="test",aio=native,cache=none
> -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device
> virtio-serial -device
> virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -m 1024 -boot c -qmp
> stdio -netdev tap,id=br0,vhost=on,ifname=tap1,script=no,downscript=no 
> -device virtio-net-pci,mac=02:12:34:56:78:9c,netdev=br0 -drive
> file=gluster://192.168.15.99/gv0/test.qcow2,if=virtio,aio=native,cache=none
> 
> 
> fio commandline:
> # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=64 --size=64M
> --readwrite=randrw --rwmixread=75
> 
> 
> Test run, no throttling:
>   read : io=49000KB, bw=804450 B/s, iops=196 , runt= 62373msec
>   write: io=16536KB, bw=271477 B/s, iops=66 , runt= 62373msec
> 
> Throttle via QMP:
> {"execute": "block_set_io_throttle", "arguments": {"device": "virtio1",
> "bps": 0, "bps_rd": 0, "bps_wr": 0, "iops": 0, "iops_rd": 15, "iops_wr": 15}
> }
> 
> Test run, with throttling:
>   read : io=49204KB, bw=61453 B/s, iops=15 , runt=819888msec
>   write: io=16332KB, bw=20397 B/s, iops=4 , runt=819888msec

Given that it works when testing only QEMU, we'll wait for the results of Paul's tests. In the meanwhile, I'm marking it condnak(reproducer).

Maybe it's actually a consequence of Bug 1297734.

OK. I have rerun the tests and can see on GA release of RHEV3.6 that the iops parameters are passed to the running process.

I can also confirm they're working well:) I tested with fio on unlimited and then limited to 100 iops, and fio results were 
Unlimited 
<snip>
read : io=359576KB, bw=2996.3KB/s, iops=749, runt=120009msec
</snip>

Limited
<snip>
read : io=48056KB, bw=409961B/s, iops=100, runt=120034msec
</snip>

With a limit in place the qemu process has the aio=threads,iops=100 parameters

So it looks like the issue now is solely BZ 1297734

Thanks for your patience!

Created attachment 1137276 [details]
qemu process output showing the iops limiting parameters

Thanks Paul.  Based on comment #21, closing this as a dupe of BZ 1297734.

*** This bug has been marked as a duplicate of bug 1297734 ***