Hide Forgot
I'm not sure if this is a qemu problem or a libvirt problem, and it may be both. Description of problem: Running migrate --copy-storage-all on a qemu image seems to ignore a disk full condition at the destination. The running qemu on the destination continues to receive data until the entire file is transferred, even though it's not able to store it anywhere. No errors are logged anywhere that I can see, for libvirt or for qemu, and the source assumes the copy has completed and shuts down the source VM even though the destination VM hasn't started. Version-Release number of selected component (if applicable): libvirt 0.8.7 qemu-kvm-0.12.1.2-2.113.el6_0.3 (as distributed w/ RH6) How reproducible: Every time Steps to Reproduce: 1. Fill up /var to mostly full with junk 2. migrate --p2p --live --tunnelled --persistent --direct --copy-storage-all testvm qemu+ssh://dest.system.com/system 3. Actual results: Migration appears to succeed, but no image is created on destination. Source VM is shut down. Expected results: An error should be displayed indicating that the image could not be transferred. Source VM should be left running. Additional info:
Actually, on further investigation, the problem doesn't appear to be related to the disk-full condition after all. I moved /var to a large enough partition so that it was nowhere near full. Instead, it appears that migrate is fully transferring the data, but isn't writing it to disk at all. strace on qemu shows a continuous stream of reads, but no writes. According to the domain XML the image should be written to /export/vm, but the image file there is zero length. "df" doesn't show any growth in any local partitions either, so I don't know where the data is actually going.
I just ran into this issue. Description of problem: virsh migrate --live --persistent --undefinesource --copy-storage-all --verbose --desturi qemu+ssh://DESTSYSTEM/system VMNAME On the destination I used qemu-img to create a raw file that had virtual sizes larger than the amount of free space. Migration will not show any errors and will succeed, but will leave a corrupted disk on the destination server. Version-Release number of selected component (if applicable): libvirtd (libvirt) 1.2.9 QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11), Copyright (c) 2003-2008 Fabrice Bellard How reproducible: Every time Steps to Reproduce: 1. Create disk image with a virtual size greater than the amount of free storage on the destination server 2. Attempt to migrate VM with disk larger than the destination's free space 3. virsh migrate --live --persistent --undefinesource --copy-storage-all --verbose --desturi qemu+ssh://DESTSYSTEM/system VMNAME Actual results: Migration will complete successfully. The disk image will grow to fill all the free space, but since the destination is full the disk is left corrupted. Expected results: Migration should abort.
Fixed upstream as: commit 80c5f10e865cda0302519492f197cb020bd14a07 Author: Michal Privoznik <mprivozn> AuthorDate: Tue Feb 10 16:25:27 2015 +0100 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Feb 19 14:12:38 2015 +0100 qemuMigrationDriveMirror: Listen to events https://bugzilla.redhat.com/show_bug.cgi?id=1179678 When migrating with storage, libvirt iterates over domain disks and instruct qemu to migrate the ones we are interested in (shared, RO and source-less disks are skipped). The disks are migrated in series. No new disk is transferred until the previous one hasn't been quiesced. This is checked on the qemu monitor via 'query-jobs' command. If the disk has been quiesced, it practically went from copying its content to mirroring state, where all disk writes are mirrored to the other side of migration too. Having said that, there's one inherent error in the design. The monitor command we use reports only active jobs. So if the job fails for whatever reason, we will not see it anymore in the command output. And this can happen fairly simply: just try to migrate a domain with storage. If the storage migration fails (e.g. due to ENOSPC on the destination) we resume the host on the destination and let it run on partly copied disk. The proper fix is what even the comment in the code says: listen for qemu events instead of polling. If storage migration changes state an event is emitted and we can act accordingly: either consider disk copied and continue the process, or consider disk mangled and abort the migration. Signed-off-by: Michal Privoznik <mprivozn> commit 76c61cdca20c106960af033e5d0f5da70177af0f Author: Michal Privoznik <mprivozn> AuthorDate: Tue Feb 10 16:24:45 2015 +0100 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Feb 19 14:12:38 2015 +0100 qemuProcessHandleBlockJob: Take status into account Upon BLOCK_JOB_COMPLETED event delivery, we check if the job has completed (in qemuMonitorJSONHandleBlockJobImpl()). For better image, the event looks something like this: "timestamp": {"seconds": 1423582694, "microseconds": 372666}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio-disk0", "len": 8412790784, "offset": 409993216, "speed": 8796093022207, "type": "mirror", "error": "No space left on device"}} If "len" does not equal "offset" it's considered an error, and we can clearly see "error" field filled in. However, later in the event processing this case was handled no differently to case of job being aborted via separate API. It's time that we start differentiate these two because of the future work. Signed-off-by: Michal Privoznik <mprivozn> commit c37943a0687a8fdb08e6eda8ae4b9f4f43f4f2ed Author: Michal Privoznik <mprivozn> AuthorDate: Tue Feb 10 15:32:59 2015 +0100 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Feb 19 14:12:38 2015 +0100 qemuProcessHandleBlockJob: Set disk->mirrorState more often Currently, upon BLOCK_JOB_* event, disk->mirrorState is not updated each time. The callback code handling the events checks if a blockjob was started via our public APIs prior to setting the mirrorState. However, some block jobs may be started internally (e.g. during storage migration), in which case we don't bother with setting disk->mirror (there's nothing we can set it to anyway), or other fields. But it will come handy if we update the mirrorState in these cases too. The event wasn't delivered just for fun - we've started the job after all. So, in this commit, the mirrorState is set to whatever job status we've obtained. Of course, there are some actions on some statuses that we want to perform. But instead of if {} else if {} else {} ... enumeration, let's move to switch(). Signed-off-by: Michal Privoznik <mprivozn> v1.2.12-155-g80c5f10 It's fixed in v1.2.13.