RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1336564 - Extend <iotune> for QEMU group I/O throttling
Summary: Extend <iotune> for QEMU group I/O throttling
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: John Ferlan
QA Contact: yisun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-16 21:48 UTC by Stefan Hajnoczi
Modified: 2017-08-01 23:51 UTC (History)
10 users (show)

Fixed In Version: libvirt-3.1.0-1.el7
Doc Type: Enhancement
Doc Text:
Feature: Add support for QEMU group I/O throttling Reason: This prevents end-users from circumventing a hosting provider's throttling policy by splitting 1 large drive into N small drives and getting N times the normal throttling quota. Result: A new domain <disk> <iotune> subelement <group_name> will define the group name to be used from libvirt.
Clone Of:
Environment:
Last Closed: 2017-08-01 17:09:12 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Stefan Hajnoczi 2016-05-16 21:48:29 UTC 
In QEMU 2.4 the group I/O throttling feature was added to QEMU and in RHEL 7.3 the feature will be present in QEMU.  The libvirt <iotune> XML element requires extension to support the group ID.

The group I/O throttling feature shares I/O throttling quota between multiple drives.  This prevents end-users from circumventing a hosting provider's throttling policy by splitting 1 large drive into N small drives and getting N times the normal throttling quota.

Use the -drive throttling.group=<name> setting to put a drive into a throttling group.  Other drives using the same group name will share the throttling budget.  If no group name is given QEMU will internally use the drive ID as the group name.
Comment 1 Stefan Hajnoczi 2016-05-16 21:50:47 UTC 
Note the feature is not present in RHEL 7.2.  It's a new feature that became available in upstream QEMU 2.4.
Comment 2 Gu Nini 2016-06-07 10:01:49 UTC 
Stefan,

What should we do for burst_length? Should <iotune> also be exended?
Comment 3 John Ferlan 2016-06-22 15:07:19 UTC 
Move to consideration for 7.4
Comment 4 Stefan Hajnoczi 2016-06-24 13:24:37 UTC 
(In reply to Gu Nini from comment #2)
> What should we do for burst_length? Should <iotune> also be exended?

Yes, I have created a new BZ for burst_length: bz#1349898
Comment 5 John Ferlan 2016-12-05 23:40:45 UTC 
Looks like I forget to update this when I posted patches... In any case, here's the most recent v2 posting:

http://www.redhat.com/archives/libvir-list/2016-November/msg01022.html

Changes are now pushed:

$ git describe fbec5b949ba61cdf67dab2c70a385d3062306b83
v2.5.0-21-gfbec5b9
$
Comment 6 Martin Kletzander 2017-01-31 19:51:15 UTC 
Few more commits for this are v3.0.0-75-ge9d75343d4cf^..v3.0.0-78-g862bea96d9d9 and v3.0.0-100-geae7cfd42d44^..v3.0.0-101-gbb5d6379a009:

commit bb5d6379a009b2806936f341523f80c9fabbcd4c
Author: Martin Kletzander <mkletzan>
Date:   Tue Jan 24 15:50:17 2017 +0100

    qemu: Don't lose group_name
    
commit eae7cfd42d442ccb1b4d086a1ed792c27e4cdc54
Author: Martin Kletzander <mkletzan>
Date:   Mon Jan 30 20:37:48 2017 +0100

    conf: Add virDomainDiskSetBlockIOTune

commit 862bea96d9d9c75ba4cb6173fe484eb3d52d3540
Author: Martin Kletzander <mkletzan>
Date:   Wed Jan 25 09:38:09 2017 +0100

    virsh: Use consistent naming for blkdeviotune options
    
commit a20e8bcad5c17380b52cc3c2469660eb63a78304
Author: Martin Kletzander <mkletzan>
Date:   Wed Jan 25 09:35:47 2017 +0100

    virsh: Actually make blkdeviotune --group_name work
    
commit 87ee705183241a42ffd36d9f5d3934cacf91c343
Author: Martin Kletzander <mkletzan>
Date:   Thu Jan 26 15:13:27 2017 +0100

    qemu: Miscellaneous Block I/O tune cleanups
    
commit e9d75343d4cf552575a3b305fa00a36ee71e9309
Author: Martin Kletzander <mkletzan>
Date:   Tue Jan 24 15:50:00 2017 +0100

    qemu: Only set group_name when actually requested
Comment 8 yisun 2017-02-08 10:50:48 UTC 
hit a issue seems related to this bug

Version:
libvirt-3.0.0-1.el7.x86_64

Steps to Reproduce
1. having a running vm
# virsh list
 Id    Name                           State
----------------------------------------------------
 3     avocado-vt-vm2                 running


2. VM having a virtual disk - vda
# virsh domblklist avocado-vt-vm2
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/RHEL-7.2-x86_64-latest.qcow2


3. use blkdeviotune to set --read-bytes-sec to vm's vda
# virsh blkdeviotune avocado-vt-vm2 --live --device vda --read-bytes-sec 4096
error: Unable to change block I/O throttle
error: internal error: argument key 'group' must not have null value


expected result:
blkdeviotune should work, group is not a mandatory parameter as virsh manual says.

actual result:
failed
Comment 9 John Ferlan 2017-02-10 13:51:20 UTC 
See comment 6 - Martin found a few issues which would be fixed in libvirt 3.1.

I'll put this back in POST so that Jiri would mark it when he builds 3.1
Comment 10 yisun 2017-05-31 10:34:16 UTC 
The new parameters will trigger a libvirtd crash when setting together with other parameters

steps:
1. prepare a vm with blk devs
## virsh domblklist r
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/r.qcow2
vdb        /var/lib/libvirt/images/luks.disk

2. let's use vdb as test object
## virsh blkdeviotune r vdb
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : 
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

3. set group name and some other parameters together, such as --total-bytes-sec
## virsh blkdeviotune r vdb --group-name goodname --total-bytes-sec 100
error: Disconnected from qemu:///system due to I/O error
error: Unable to change block I/O throttle
error: End of file while reading data: Input/output error
<==== libvirtd crashed~~
( if setting the parameters separately, everything'll be fine - `virsh blkdeviotune r vdb --group-name goodname; virsh blkdeviotune r vdb --total-bytes-sec 100`)

I put the gdb backtrace in pastebin:
http://pastebin.test.redhat.com/489354


http://pastebin.test.redhat.com/489354
Comment 11 John Ferlan 2017-06-03 23:28:35 UTC 
I think comment 10 is fixed by bz 1455510.

It's not clear from the comment which libvirt version you used to test...

Can this be retried with:

Version-Release number of selected component:
libvirt-3.2.0-7.el7.x86_64
Comment 12 yisun 2017-06-05 07:56:42 UTC 
(In reply to John Ferlan from comment #11)
> I think comment 10 is fixed by bz 1455510.
> 
> It's not clear from the comment which libvirt version you used to test...
> 
> Can this be retried with:
> 
> Version-Release number of selected component:
> libvirt-3.2.0-7.el7.x86_64

Due to the time I added comment 10, it should be 3.2.0-6, I didn't notice there was another bug tracking this...
Should be same problem, so move this back to ON_QA and will test it soon. thx
Comment 13 yisun 2017-06-05 11:43:51 UTC 
Verified on libvirt-3.2.0-7.el7.x86_64

1. do not explicitly setting group name, but setting other parameters
## virsh start r
Domain r started

## virsh blkdeviotune r vda
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     :
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

## virsh blkdeviotune r vda --total-bytes-sec 100


 ## virsh blkdeviotune r vda
total_bytes_sec: 100
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 10
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     :*** drive-virtio-disk0 ***
total_bytes_sec_max_length: 1
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

<=== according to https://bugzilla.redhat.com/show_bug.cgi?id=1455834#c1 this is expected.

2. explicitly setting group name
## virsh blkdeviotune r vda --group-name goodname

## virsh blkdeviotune r vda
total_bytes_sec: 100
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 10
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : *** goodname ***
total_bytes_sec_max_length: 1
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0

## virsh dumpxml r | grep goodname -a2
      <iotune>
        <total_bytes_sec>100</total_bytes_sec>
        <group_name>goodname</group_name>
      </iotune>

## virsh dumpxml r --inactive| grep goodname -a2

3. start a vm with existing group name setting
## virsh blkdeviotune r vda --group-name goodname --config

## virsh destroy r
Domain r destroyed

## virsh dumpxml r --inactive| grep goodname -a2
      <iotune>
        <group_name>goodname</group_name>
      </iotune>

## virsh start r
error: Failed to start domain r
error: unsupported configuration: group_name can be configured only together with settings
<==== clear error message, acceptable

4. start a vm with existing group name and read_bytes_sec setting
## virsh dumpxml r --inactive | grep iotune -a3
...
      <iotune>
        <read_bytes_sec>200</read_bytes_sec>
        <group_name>goodname</group_name>
      </iotune>

##  virsh blkdeviotune r vda
total_bytes_sec: 0
read_bytes_sec : 200
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 20
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : *** goodname ***
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 1
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0
5. check --config option works fine for --group-name
## virsh blkdeviotune r vda --group-name goodname --read-iops-sec 100 --config
## virsh dumpxml r | grep iotune -a3
<=== nothing as expected
## virsh dumpxml r --inactive | grep iotune -a3
...
      <iotune>
        <read_iops_sec>100</read_iops_sec>
        <group_name>goodname</group_name>
      </iotune>
...
## virsh blkdeviotune r vda
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 0
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 0
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     :
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 0
write_iops_sec_max_length: 0
## virsh destroy r; virsh start r
Domain r destroyed

Domain r started

## virsh blkdeviotune r vda
total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 0
read_iops_sec  : 100
write_iops_sec : 0
total_bytes_sec_max: 0
read_bytes_sec_max: 0
write_bytes_sec_max: 0
total_iops_sec_max: 0
read_iops_sec_max: 10
write_iops_sec_max: 0
size_iops_sec  : 0
group_name     : *** goodname ***
total_bytes_sec_max_length: 0
read_bytes_sec_max_length: 0
write_bytes_sec_max_length: 0
total_iops_sec_max_length: 0
read_iops_sec_max_length: 1
write_iops_sec_max_length: 0

(some params combination tests were carried out in https://bugzilla.redhat.com/show_bug.cgi?id=1455510#c4)
Comment 14 errata-xmlrpc 2017-08-01 17:09:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 15 errata-xmlrpc 2017-08-01 23:51:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.