Bug 1531888

This is a bug in the qemu locking code, so I'm reassigning this to qemu-kvm. 

Also this might be impossible to solve since qemu can't know whether the image is locked by the original source or another VM. If qemu can't solve this please close this bug as CANTFIX since libvirt has it's own locking.

Looking at the stderr of QEMU, there _is_ an error like:

"Failed to get "write" lock
Is another process using the image?"

However, it is not reported to the monitor because the point where this happens is at the end of migration and we currently don't have a channel to report it to libvirt correctly, neither does the migration process abort/cancel due to the locking failure.

The fix would be changing destination QEMU to fail the migration if image locking failed.

Well, libvirt only ever looks at the stderr output if the qemu process exits before the monitor is up. After that the error should be reported on the monitor. Either by failing the migration, which is properly wired up in libvirt's migration code or by reporting an event along with obviously stopping the cpus.

May I have your libvirt logs, please?

Created attachment 1524938 [details]
Libvirtd log for migration step

Alright, let's simplify the reproducer.

0. Pick a raw test image.  Any image should do.  Mine is called test.img.

1. Run a qemu-kvm using that image:

   $ qemu-kvm -nodefaults -S -display none -drive if=none,id=disk0,format=raw,file=test.img -device virtio-blk-pci,drive=disk0

   Starts successfully.

2. In a second terminal, run another qemu-kvm using the same image:

   $ qemu-kvm -nodefaults -S -display none -drive if=none,id=disk0,format=raw,file=test.img -device virtio-blk-pci,drive=disk0
   qemu-kvm: -device virtio-blk-pci,drive=disk0: Failed to get "write" lock
   Is another process using the image?

   Fails as expected.

3. Same, but add -incoming defer:

   $ qemu-kvm -nodefaults -S -display none -drive if=none,id=disk0,format=raw,file=test.img -device virtio-blk-pci,drive=disk0 -incoming defer

   Starts successfully.

   So, -incoming defer somehow suppresses the error, and it's how libvirt starts qemu-kvm on the destination end of the migration, as the logs from comment#10 show.

   I tracked this down to blk_do_attach_dev():

    /* While migration is still incoming, we don't need to apply the
     * permissions of guest device BlockBackends. We might still have a block
     * job or NBD server writing to the image for storage migration. */
    if (runstate_check(RUN_STATE_INMIGRATE)) {
        blk->disable_perm = true;
    }

   This is from commit d35ff5e6b3aa3a706b0aa3bcf11400fac945b67a "block: Ignore guest dev permissions during incoming migration".

4. Kill the second qemu-kvm, restart it with a monitor, then make it await incoming migration

   $ qemu-kvm -nodefaults -S -display none -drive if=none,id=disk0,format=raw,file=test.img -device virtio-blk-pci,drive=disk0 -incoming defer -monitor stdio
   QEMU 2.12.0 monitor - type 'help' for more information
   (qemu) migrate_incoming unix:test-mig
   (qemu) 

5. In a third terminal, copy the image, start a third qemu-kvm, then migrate it to the second one

   $ cp test.img copy-of-test.img
   $ qemu-kvm -nodefaults -S -display none -drive if=none,id=disk0,format=raw,file=copy-of-test.img -device virtio-blk-pci,drive=disk0 -monitor stdio
   QEMU 2.12.0 monitor - type 'help' for more information
   (qemu) migrate unix:test-mig
   (qemu) 

   Apparently succeeds, but the second qemu-kvm (in the second terminal) gripes:

   rhev7-qemu-kvm: Failed to get "write" lock
   Is another process using the image?

   Matches what Fam observed in comment#5.

Slightly refined simplified reproducer:

= Terminal 1 =

    $ qemu-kvm -nodefaults -display none -drive if=none,id=disk0,format=raw,file=test.img -device virtio-blk-pci,drive=disk0 -monitor stdio
    QEMU 2.12.0 monitor - type 'help' for more information
    (qemu) info block
    disk0 (#block122): test.img (raw)
	Attached to:      /machine/peripheral-anon/device[0]/virtio-backend
	Cache mode:       writethrough
    (qemu) info status
    VM status: running
    (qemu) 

Nothing interesting's going to happen in this terminal from now on.  Its
sole purpose is holding a lock on test.img.

= Terminal 2 =

    $ qemu-kvm -nodefaults -display none -drive if=none,id=disk0,format=raw,file=test.img -device virtio-blk-pci,drive=disk0 -incoming unix:test-mig -monitor stdio
    QEMU 2.12.0 monitor - type 'help' for more information
    (qemu) info status
    VM status: paused (inmigrate)
    (qemu) 

= Terminal 3 =

    $ qemu-kvm -nodefaults -display none -drive if=none,id=disk0,format=raw,file=copy-of-test.img -device virtio-blk-pci,drive=disk0 -monitor stdio
    QEMU 2.12.0 monitor - type 'help' for more information
    (qemu) info status
    VM status: running
    (qemu) migrate unix:test-mig

The command completes after a small delay:

    (qemu) 

= Terminal 2 =

Locking fails:

    qemu-kvm: Failed to get "write" lock
    Is another process using the image [test.img]?

Checking status:

    info status
    VM status: paused
   (qemu) 
= Terminal 3 =

    info status
    VM status: paused (postmigrate)
   (qemu) 

= Terminal 2 =

Continue the stopped migrated guest, and check its status:

    cont
    (qemu) info status
    VM status: running
    (qemu) info block
    disk0 (#block130): test.img (raw)
	Attached to:      /machine/peripheral-anon/device[0]/virtio-backend
	Cache mode:       writethrough
    (qemu) 

I now have two guests running, and both use test.img.  Oops.

Upstream behaves the same.

I enlisted Kevin Wolf's and Dave Gilbert's assistance.  They pointed out that with QCOW2 rather than raw images, the "cont" fails, and the guest remains stopped.

Armed with that information, Kevin went looking for the reason, and found a bug on an error path that isn't taken with QCOW2.  He volunteered to take this bug.

Thanks, guys!

Fix included in qemu-kvm-rhev-2.12.0-22.el7

Reproduced on qemu-kvm-rhev-2.12.0-18.el7_6.1.x86_64 with steps in comment 12:
Terminal 1:
(qemu) info status
VM status: running
Terminal 2:
(qemu) c
Failed to get "write" lock
Is another process using the image?
**after migration**
(qemu) info status
VM status: running
Terminal 3:
(qemu) info status
VM status: paused (postmigrate)

Verified this bug on qemu-kvm-rhev-2.12.0-23.el7.x86_64:
Terminal 1:
(qemu) info status
VM status: running
Terminal 2:
(qemu) c
Failed to get "write" lock
Is another process using the image [/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.raw]?
**after migration**
(qemu) qemu-kvm: Failed to get "write" lock
Is another process using the image [/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.raw]?
(qemu) info status
VM status: paused
(qemu) c
Failed to get "write" lock
Is another process using the image [/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.raw]?
Terminal 3:
(qemu) info status
VM status: paused (postmigrate)

Thanks.

Verified this bug per comment 16.

Thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553