Hide Forgot
Description of problem: Version-Release number of selected component (if applicable): libvirt-0.8.7-3.el6.x86_64 How reproducible: at the beginning , the top 100 times loop , didn't meet this error , but from the 150 times, the error occurred a little more frequently , but not 100% . When error occurred, libvirtd was still running Steps to Reproduce: 1. install a RHEL guest on the lv over iscsi lun 2. make a tight save for this guest to a lv over iscsi lun (mkfs.ext3 on the lun , and mount it to a directory, the save target is a file) # for i in `seq 1000`; do virsh save --domain test --file /opt/test.save; virsh restore /opt/test.save; done Actual results: Domain restored from /opt/test.save Domain test saved to /opt/test.save error: Failed to restore domain from /opt/test.save error: cannot close file: Bad file descriptor error: Failed to save domain test to /opt/test.save error: Requested operation is not valid: domain is not running Expected results: Additional info:
In my testing this appears to be related to the load of the machine. If i have an idle machine and just run save/restore in a loop it works fine. If i launch a major compile job on the machine then save/restore will fail reasonably frequently.
This is looking like a possible QEMU issue. Migration appears to process pretty much all the RAM, and the final 'info migrate' command then returns 'failed'. Perhaps some kind of race condition in the cleanup code for exec based migration.
Confirmed that this is not a libvirt bug. QEMU will randomly fail exec based migration even when used standalone. It is a particular problem when the machine is under high load. The problem is a race condition in QEMU waitpid() calls. See bug 678524 for full info. Will leave this libvirt bug open to track QA re-testing of libvirt once QEMU is fixed.
I've also opened bug 678548 as an RFE to stop using exec: migration completely, since QEMU has proved buggy as hell with exec: ever since we started using it.
In one regards, bug 678548 is an extension of bug 620363, and I've already started playing with the conversion process to avoid exec: for qemu 0.12.0 and newer. For qemu 0.10.x (think RHEL 5), however, there is no fd: support, so we are stuck with exec: there.
Setting the TestOnly keyword per comment 8.
I've reproduced the bad fd error; libvirt has a double close bug: qemudDomainRestore calls qemudDomainSaveImageOpen to open an fd, then calls qemudDomainSaveImageStartVM to start a VM from that fd, then calls qemudDomainSaveImageClose to close the fd. But qemudDomainSaveImageStartVM _also_ calls qemudDomainSaveImageClose, but only on some paths, leading to double closure of the fd and possible corruption of an unintended fd opened by another thread at the same time. I'm removing the TestOnly keyword; this bug should be considered a blocker for 6.1.
upstream patch posted, still awaiting review and further testing: https://www.redhat.com/archives/libvir-list/2011-March/msg00010.html
*** Bug 672492 has been marked as a duplicate of this bug. ***
In POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-March/msg00045.html
(In reply to comment #18) > tested with following rpms , and FAILED > > # rpm -q libvirt qemu-kvm kernel > libvirt-0.8.7-10.el6.x86_64 > qemu-kvm-0.12.1.2-2.149.el6.x86_64 > kernel-2.6.32-118.el6.x86_64 > > > Steps: > 1. for i in `seq 1 1000`; do virsh save --domain cdrom_test --file > /opt/test.save; virsh restore /opt/test.save; done > > rolling message of > > error: Failed to save domain cdrom_test to /opt/test.save > error: operation failed: domain save job: unexpectedly failed The particular patch in libvirt 0.8.7-10.el6 only fixes the double-close bug; it does not fix the fact that qemu's exec: migration still has a race. You therefore cannot reliably verify this fix until either qemu bug 678524 is also fixed (to make exec: non-racy), or until libvirt bug 678548 is also fixed (to avoid exec: altogether). Moving back to MODIFIED, as the double-close patch is in. You can verify that no double-closes are happening if you no longer get any 'bad file descriptor' messages in your tight loop.
tested with following rpms , and Passed # rpm -q libvirt qemu-kvm kernel libvirt-0.8.7-10.el6.x86_64 qemu-kvm-0.12.1.2-2.149.el6.x86_64 kernel-2.6.32-118.el6.x86_64 Steps: 1. for i in `seq 1 1000`; do virsh save --domain cdrom_test --file /opt/test.save; virsh restore /opt/test.save; done NO error for 'bad file descriptor' displayed So set bug status to VERIFIED according to comment 19
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-0596.html