RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 770520 - blkiotune set weight on total and virtio device together will cause libvirtd hang
Summary: blkiotune set weight on total and virtio device together will cause libvirtd ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.3
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-12-27 08:32 UTC by weizhang
Modified: 2012-06-20 06:40 UTC (History)
11 users (show)

Fixed In Version: libvirt-0.9.10-1.el6
Doc Type: Bug Fix
Doc Text:
No Documentation needed
Clone Of:
Environment:
Last Closed: 2012-06-20 06:40:22 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:0748 0 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2012-06-19 19:31:38 UTC

Description weizhang 2011-12-27 08:32:01 UTC 
Description of problem:
when do blkiotune with --weight and --device-weights on virtio device, libvirtd will hang, and on libvirtd.log there will be lots of error logs like
"qemuDomainSetBlkioParameters:6131 : Unable to set io device weight for path /dev/vda: No such file or directory"
and occupy a large number of disk size


Version-Release number of selected component (if applicable):
kernel-2.6.32-223.el6.x86_64
libvirt-0.9.8-1.el6.x86_64
qemu-kvm-0.12.1.2-2.213.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. start a guest with virtio disk
# virsh dumpxml guest
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/kvm-rhel6u2-x86_64-new.img'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

2. Do command
# virsh blkiotune guest --device-weights /dev/vda,100 --weight 500

3. Check /var/log/libvirt/libvirtd.log
  
Actual results:
step 2 will hang and there are lots of error info continue writing on libvirtd.log

Expected results:
It may works well

Additional info:
Comment 2 Alex Jia 2011-12-27 09:15:44 UTC 
The programming will enter into a infinite loop due to 'i != ndevices' is true forever in this case:

static int qemuDomainSetBlkioParameters(virDomainPtr dom,
                                         virTypedParameterPtr params,
                                         int nparams,
                                         unsigned int flags)
{
......
        for (i = 0; i < nparams; i++) {
            int rc;
            virTypedParameterPtr param = &params[i];
......
                for (i = 0; i < ndevices; i++) {
                    rc = virCgroupSetBlkioDeviceWeight(group,
                                                       devices[i].path,
                                                       devices[i].weight);
                    if (rc < 0) {
                        virReportSystemError(-rc,
                                             _("Unable to set io device weight "
                                               "for path %s"),
                                             devices[i].path);
                        break;
                    }
                }
                if (i != ndevices) {
                    ret = -1;
                    continue;
                }
......
Comment 3 Eric Blake 2011-12-28 13:59:02 UTC 
commit 1a3f6608aa4c945bb3f392c25ff06b13f1dc5f30
Author: Eric Blake <eblake>
Date:   Wed Dec 28 06:53:27 2011 -0700

    qemu: fix inf-loop in blkio parameters
    
    https://bugzilla.redhat.com/show_bug.cgi?id=770520
    
    We had two nested loops both trying to use 'i' as the iteration
    variable, which can result in an infinite loop when the inner
    loop interferes with the outer loop.  Introduced in commit 93ab585.
    
    * src/qemu/qemu_driver.c (qemuDomainSetBlkioParameters): Don't
    reuse iteration variable across two loops.
Comment 4 zhpeng 2012-01-06 09:18:13 UTC 
I test it with:
libvirt-0.9.9-0rc1.el6.x86_64
qemu-kvm-0.12.1.2-2.209.el6_2.2.x86_64


Steps:
1. virsh blkiotune guest --device-weights /dev/sda,100 --weight 500

Results:
it hangs forever.


Check /var/log/libvirt/libvirtd.log:

2012-01-06 09:21:37.693+0000: 2294: warning : virKeepAliveTimer:182 : No response from client 0x234d270 after 5 keepalive messages in 30 seconds

2012-01-06 09:23:50.529+0000: 2294: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error (*When press Ctrl+C to terminate Step 1*)
Comment 5 zhpeng 2012-01-06 09:50:12 UTC 
Sorry, the PKGs are
libvirt-0.9.9-0rc1.el6.x86_64
qemu-kvm-0.12.1.2-2.213.el6.x86_64
Comment 7 Alex Jia 2012-01-10 10:01:03 UTC 
The issue still exists on libvirt-0.9.9-1.el6.x86_64, the root reason is a copy-paste error on previous patch, and I have committed a patch to fix the issue:

https://www.redhat.com/archives/libvir-list/2012-January/msg00309.html
Comment 8 Peter Krempa 2012-01-10 10:59:34 UTC 
Upstream fix for the typo in the previous patch:

commit d8d9b0e05844802d9f659f35e1c8a4653f32d5f2
Author: Alex Jia <ajia>
Date:   Tue Jan 10 17:55:01 2012 +0800

    qemu: fix a typo on qemuDomainSetBlkioParameters
    
    It should be a copy-paste error, the result is programming will result in an
    infinite loop again due to without iterating 'j' variable.
    
    * src/qemu/qemu_driver.c: fix a typo on qemuDomainSetBlkioParameters.
Comment 9 zhe peng 2012-02-15 06:34:16 UTC 
verify with:
libvirt-0.9.10-1.el6.x86_64
qemu-kvm-0.12.1.2-2.229.el6.x86_64

step:
1.start the guest
#virsh start v1-clone 
2.run command
#virsh blkiotune v1-clone --device-weights /dev/sda,100 --weight 500

no hang,the command finished w/o error
2. virsh dumpxml v1-clone
.........
<blkiotune>
    <device>
      <path>/dev/sda</path>
      <weight>100</weight>
    </device>
  </blkiotune>
.........
3. #virsh blkiotune v1-clone
weight         : 500
device_weight  : /dev/sda,100

verification passed.
move to verified.
Comment 10 Peter Krempa 2012-05-02 09:45:26 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No Documentation needed
Comment 12 errata-xmlrpc 2012-06-20 06:40:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html
Note You need to log in before you can comment on or make changes to this bug.