Red Hat Bugzilla – Bug 770520
blkiotune set weight on total and virtio device together will cause libvirtd hang
Last modified: 2012-06-20 02:40:22 EDT
Description of problem: when do blkiotune with --weight and --device-weights on virtio device, libvirtd will hang, and on libvirtd.log there will be lots of error logs like "qemuDomainSetBlkioParameters:6131 : Unable to set io device weight for path /dev/vda: No such file or directory" and occupy a large number of disk size Version-Release number of selected component (if applicable): kernel-2.6.32-223.el6.x86_64 libvirt-0.9.8-1.el6.x86_64 qemu-kvm-0.12.1.2-2.213.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1. start a guest with virtio disk # virsh dumpxml guest <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/kvm-rhel6u2-x86_64-new.img'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> 2. Do command # virsh blkiotune guest --device-weights /dev/vda,100 --weight 500 3. Check /var/log/libvirt/libvirtd.log Actual results: step 2 will hang and there are lots of error info continue writing on libvirtd.log Expected results: It may works well Additional info:
The programming will enter into a infinite loop due to 'i != ndevices' is true forever in this case: static int qemuDomainSetBlkioParameters(virDomainPtr dom, virTypedParameterPtr params, int nparams, unsigned int flags) { ...... for (i = 0; i < nparams; i++) { int rc; virTypedParameterPtr param = ¶ms[i]; ...... for (i = 0; i < ndevices; i++) { rc = virCgroupSetBlkioDeviceWeight(group, devices[i].path, devices[i].weight); if (rc < 0) { virReportSystemError(-rc, _("Unable to set io device weight " "for path %s"), devices[i].path); break; } } if (i != ndevices) { ret = -1; continue; } ......
commit 1a3f6608aa4c945bb3f392c25ff06b13f1dc5f30 Author: Eric Blake <eblake@redhat.com> Date: Wed Dec 28 06:53:27 2011 -0700 qemu: fix inf-loop in blkio parameters https://bugzilla.redhat.com/show_bug.cgi?id=770520 We had two nested loops both trying to use 'i' as the iteration variable, which can result in an infinite loop when the inner loop interferes with the outer loop. Introduced in commit 93ab585. * src/qemu/qemu_driver.c (qemuDomainSetBlkioParameters): Don't reuse iteration variable across two loops.
I test it with: libvirt-0.9.9-0rc1.el6.x86_64 qemu-kvm-0.12.1.2-2.209.el6_2.2.x86_64 Steps: 1. virsh blkiotune guest --device-weights /dev/sda,100 --weight 500 Results: it hangs forever. Check /var/log/libvirt/libvirtd.log: 2012-01-06 09:21:37.693+0000: 2294: warning : virKeepAliveTimer:182 : No response from client 0x234d270 after 5 keepalive messages in 30 seconds 2012-01-06 09:23:50.529+0000: 2294: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error (*When press Ctrl+C to terminate Step 1*)
Sorry, the PKGs are libvirt-0.9.9-0rc1.el6.x86_64 qemu-kvm-0.12.1.2-2.213.el6.x86_64
The issue still exists on libvirt-0.9.9-1.el6.x86_64, the root reason is a copy-paste error on previous patch, and I have committed a patch to fix the issue: https://www.redhat.com/archives/libvir-list/2012-January/msg00309.html
Upstream fix for the typo in the previous patch: commit d8d9b0e05844802d9f659f35e1c8a4653f32d5f2 Author: Alex Jia <ajia@redhat.com> Date: Tue Jan 10 17:55:01 2012 +0800 qemu: fix a typo on qemuDomainSetBlkioParameters It should be a copy-paste error, the result is programming will result in an infinite loop again due to without iterating 'j' variable. * src/qemu/qemu_driver.c: fix a typo on qemuDomainSetBlkioParameters.
verify with: libvirt-0.9.10-1.el6.x86_64 qemu-kvm-0.12.1.2-2.229.el6.x86_64 step: 1.start the guest #virsh start v1-clone 2.run command #virsh blkiotune v1-clone --device-weights /dev/sda,100 --weight 500 no hang,the command finished w/o error 2. virsh dumpxml v1-clone ......... <blkiotune> <device> <path>/dev/sda</path> <weight>100</weight> </device> </blkiotune> ......... 3. #virsh blkiotune v1-clone weight : 500 device_weight : /dev/sda,100 verification passed. move to verified.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: No Documentation needed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0748.html