In my testing this appears to be related to the load of the machine. If i have an idle machine and just run save/restore in a loop it works fine. If i launch a major compile job on the machine then save/restore will fail reasonably frequently.

This is looking like a possible QEMU issue. Migration appears to process pretty much all the RAM, and the final 'info migrate' command then returns 'failed'. Perhaps some kind of race condition in the cleanup code for exec based migration.

Confirmed that this is not a libvirt bug. QEMU will randomly fail exec based migration even when used standalone. It is a particular problem when the machine  is under high load. The problem is a race condition in QEMU  waitpid() calls. See bug 678524 for full info.

Will leave this libvirt bug open to track QA re-testing of libvirt once QEMU is fixed.

I've also opened bug 678548 as an RFE to stop using exec: migration completely, since QEMU has proved buggy as hell with exec: ever since we started using it.

In one regards, bug 678548 is an extension of bug 620363, and I've already started playing with the conversion process to avoid exec: for qemu 0.12.0 and newer.  For qemu 0.10.x (think RHEL 5), however, there is no fd: support, so we are stuck with exec: there.

Setting the TestOnly keyword per comment 8.

I've reproduced the bad fd error; libvirt has a double close bug:

qemudDomainRestore calls qemudDomainSaveImageOpen to open an fd, then calls qemudDomainSaveImageStartVM to start a VM from that fd, then calls qemudDomainSaveImageClose to close the fd.  But qemudDomainSaveImageStartVM _also_ calls qemudDomainSaveImageClose, but only on some paths, leading to double closure of the fd and possible corruption of an unintended fd opened by another thread at the same time.

I'm removing the TestOnly keyword; this bug should be considered a blocker for 6.1.

upstream patch posted, still awaiting review and further testing:
https://www.redhat.com/archives/libvir-list/2011-March/msg00010.html

*** Bug 672492 has been marked as a duplicate of this bug. ***

In POST:
http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-March/msg00045.html

(In reply to comment #18)
> tested with following rpms , and FAILED 
> 
> # rpm -q libvirt qemu-kvm kernel
> libvirt-0.8.7-10.el6.x86_64
> qemu-kvm-0.12.1.2-2.149.el6.x86_64
> kernel-2.6.32-118.el6.x86_64
> 
> 
> Steps:
> 1. for  i in `seq 1 1000`; do  virsh save --domain cdrom_test --file
> /opt/test.save; virsh restore /opt/test.save; done
> 
> rolling message of 
> 
> error: Failed to save domain cdrom_test to /opt/test.save
> error: operation failed: domain save job: unexpectedly failed

The particular patch in libvirt 0.8.7-10.el6 only fixes the double-close bug; it does not fix the fact that qemu's exec: migration still has a race.  You therefore cannot reliably verify this fix until either qemu bug 678524 is also fixed (to make exec: non-racy), or until libvirt bug 678548 is also fixed (to avoid exec: altogether).

Moving back to MODIFIED, as the double-close patch is in.  You can verify that no double-closes are happening if you no longer get any 'bad file descriptor' messages in your tight loop.

tested with following rpms , and Passed

# rpm -q libvirt qemu-kvm kernel
libvirt-0.8.7-10.el6.x86_64
qemu-kvm-0.12.1.2-2.149.el6.x86_64
kernel-2.6.32-118.el6.x86_64


Steps:
1. for  i in `seq 1 1000`; do  virsh save --domain cdrom_test --file
/opt/test.save; virsh restore /opt/test.save; done

NO error for 'bad file descriptor' displayed 

So set bug status to VERIFIED  according to comment 19

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0596.html