Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Version-Release number of selected component (if applicable):
libvirt-0.8.7-3.el6.x86_64
How reproducible:
at the beginning , the top 100 times loop , didn't meet this error , but from the 150 times, the error occurred a little more frequently , but not 100% . When error occurred, libvirtd was still running
Steps to Reproduce:
1. install a RHEL guest on the lv over iscsi lun
2. make a tight save for this guest to a lv over iscsi lun (mkfs.ext3 on the lun , and mount it to a directory, the save target is a file)
# for i in `seq 1000`; do virsh save --domain test --file /opt/test.save; virsh restore /opt/test.save; done
Actual results:
Domain restored from /opt/test.save
Domain test saved to /opt/test.save
error: Failed to restore domain from /opt/test.save
error: cannot close file: Bad file descriptor
error: Failed to save domain test to /opt/test.save
error: Requested operation is not valid: domain is not running
Expected results:
Additional info:
In my testing this appears to be related to the load of the machine. If i have an idle machine and just run save/restore in a loop it works fine. If i launch a major compile job on the machine then save/restore will fail reasonably frequently.
This is looking like a possible QEMU issue. Migration appears to process pretty much all the RAM, and the final 'info migrate' command then returns 'failed'. Perhaps some kind of race condition in the cleanup code for exec based migration.
Confirmed that this is not a libvirt bug. QEMU will randomly fail exec based migration even when used standalone. It is a particular problem when the machine is under high load. The problem is a race condition in QEMU waitpid() calls. See bug 678524 for full info.
Will leave this libvirt bug open to track QA re-testing of libvirt once QEMU is fixed.
I've also opened bug 678548 as an RFE to stop using exec: migration completely, since QEMU has proved buggy as hell with exec: ever since we started using it.
In one regards, bug 678548 is an extension of bug 620363, and I've already started playing with the conversion process to avoid exec: for qemu 0.12.0 and newer. For qemu 0.10.x (think RHEL 5), however, there is no fd: support, so we are stuck with exec: there.
I've reproduced the bad fd error; libvirt has a double close bug:
qemudDomainRestore calls qemudDomainSaveImageOpen to open an fd, then calls qemudDomainSaveImageStartVM to start a VM from that fd, then calls qemudDomainSaveImageClose to close the fd. But qemudDomainSaveImageStartVM _also_ calls qemudDomainSaveImageClose, but only on some paths, leading to double closure of the fd and possible corruption of an unintended fd opened by another thread at the same time.
I'm removing the TestOnly keyword; this bug should be considered a blocker for 6.1.
(In reply to comment #18)
> tested with following rpms , and FAILED
>
> # rpm -q libvirt qemu-kvm kernel
> libvirt-0.8.7-10.el6.x86_64
> qemu-kvm-0.12.1.2-2.149.el6.x86_64
> kernel-2.6.32-118.el6.x86_64
>
>
> Steps:
> 1. for i in `seq 1 1000`; do virsh save --domain cdrom_test --file
> /opt/test.save; virsh restore /opt/test.save; done
>
> rolling message of
>
> error: Failed to save domain cdrom_test to /opt/test.save
> error: operation failed: domain save job: unexpectedly failed
The particular patch in libvirt 0.8.7-10.el6 only fixes the double-close bug; it does not fix the fact that qemu's exec: migration still has a race. You therefore cannot reliably verify this fix until either qemu bug 678524 is also fixed (to make exec: non-racy), or until libvirt bug 678548 is also fixed (to avoid exec: altogether).
Moving back to MODIFIED, as the double-close patch is in. You can verify that no double-closes are happening if you no longer get any 'bad file descriptor' messages in your tight loop.
tested with following rpms , and Passed
# rpm -q libvirt qemu-kvm kernel
libvirt-0.8.7-10.el6.x86_64
qemu-kvm-0.12.1.2-2.149.el6.x86_64
kernel-2.6.32-118.el6.x86_64
Steps:
1. for i in `seq 1 1000`; do virsh save --domain cdrom_test --file
/opt/test.save; virsh restore /opt/test.save; done
NO error for 'bad file descriptor' displayed
So set bug status to VERIFIED according to comment 19
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.
http://rhn.redhat.com/errata/RHBA-2011-0596.html