Created attachment 1195195 [details]
libvirtd.log on target host

Not sure this is going to be possible with the current code. Migration of disks isn't something I'm overly familiar with, but in general what I see happening is libvirt will create an NBD connection and use the 'drive-mirror' command to transfer disks (qemuMonitorJSONDriveMirror called eventually from qemuMigrationDriveMirror).

Looking at the QEMU code for drive-mirror, I'm not sure creation of a luks volume was added. I can see in the qemu-img code a call to bdrv_img_create with "options" in the 5th parameter.  The options are how qemu-img handles the "--object secret,id=xxx,..." and "-o key-secret=xxx" for the "-f luks" volume.  Compared to the qmp_drive_mirror code the two calls to bdrv_img_create pass NULL to the bdrv_img_create command.

I've placed a needsinfo on Dan Berrange who handled the qemu luks code to verify what I've seen or perhaps point me in the right direction to get this to work.

John's correct - no support was added for creating LUKS volumes during drive-mirror.

It turns out that QEMU 2.6 added a new "blockdev-mirror" command which is intended to replace "drive-mirror". This new command lets us provide image options, mostly particularly the key-secret required by LUKS, when mirroring. It is slightly different in that it can only operate on pre-existing images, so we would need to use qemu-img to create the new image upfront if needed.

Anyway, use of this new command should let libvirt fix the luks disk problem.

See also https://lists.gnu.org/archive/html/qemu-block/2016-09/msg00834.html

This bug is going to be addressed in next major release.

blockdev-mirror is used for the copy job if blockdev is enabled since:

commit ce7229a3b0d28479e0f123efce3fa73617889a50
Author: Peter Krempa <pkrempa>
Date:   Mon Jul 22 13:59:01 2019 +0200

    qemu: Add blockdev support for the block copy job
    
    Implement job handling for the block copy job (drive/blockdev-mirror)
    when using -blockdev. In contrast to the previously implemented
    blockjobs the block copy job introduces new images to the running qemu
    instance, thus requires a bit more handling.
    
    When copying to new images the code now makes use of blockdev-create to
    format the images explicitly rather than depending on automagic qemu
    behaviour.

The blockdev feature was enabled since:

commit c6a9e54ce3252196f1fc6aa9e57537a659646d18
Author: Peter Krempa <pkrempa>
Date:   Mon Jan 7 11:45:19 2019 +0100

    qemu: enable blockdev support

    Now that all pieces are in place (hopefully) let's enable -blockdev.

    We base the capability on presence of the fix for 'auto-read-only' on
    files so that blockdev works properly, mandate that qemu supports
    explicit SCSI id strings to avoid ABI regression and that the fix for
    'savevm' is present so that internal snapshots work.

v5.9.0-390-gc6a9e54ce3

and requires upstream qemu-4.2 or appropriate downstream.

Verified on libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64 qemu-kvm-4.2.0-6.module+el8.2.0+5453+31b2b136.x86_64:
1. Start an VM
2. Prepare a luks disk
# cat sec.xml                                                                                                                                                                                               
<secret ephemeral='no' private='yes'>
   <usage type='volume'>
      <volume>/var/lib/libvirt/images/luks-dest</volume>
   </usage>
</secret>

# qemu-img create -f luks --object secret,data=12345,id=sec0 -o key-secret=sec0 /var/lib/libvirt/images/luks-dest 10G

# virsh secret-define sec.xml

# virsh secret-set-value 73ad8135-0e18-4cb2-b98d-4e6daaf7c0ce --base64 $(printf 12345 | base64)

# cat luks.xml                                                                                                       
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/luks-dest'/>
      <target dev='vda' bus='virtio'/>
      <encryption format='luks'>
        <secret type='passphrase' usage='/var/lib/libvirt/images/luks-dest'/>
      </encryption>
    </disk>

3. Start blockcopy job:
# virsh blockcopy pc vda --reuse-external --xml luks.xml --transient-job --wait --verbose --pivot                 
Block Copy: [100 %]

4. Do copy storage migration:
Prepare luks disk and secret on destination host
# virsh migrate pc qemu+ssh://rhel82-nest2.usersys.redhat.com/system --verbose --copy-storage-all                                                                                                              
Migration: [100 %]

Actually I don't think this works properly. For the image to actually use LUKS the <encryption> element must be a subelement of <source>. You've created a non-encrypted image.

Additionally while testing it I found a bug in the code where we actually wouldn't format the LUKS image due to a logic bug.

Moving back to ASSIGNED. The fix for creation of the LUKS disk is trivial but I'll also need to improve documentation.

Formatting of raw images fixed upstream:

f4e7c792d5 qemu: block: Don't skip creation of 'luks' formatted images

Known issues:
for luks usage: https://bugzilla.redhat.com/show_bug.cgi?id=1814923
for nvme disk:  https://bugzilla.redhat.com/show_bug.cgi?id=1814947
for scsi pr:    https://bugzilla.redhat.com/show_bug.cgi?id=1814962

Version:
libvirt-6.0.0-13.virtcov.el8.x86_64
qemu-kvm-4.2.0-15.module+el8.2.0+6029+618ef2ec.x86_64

Created attachment 1671356 [details]
disk xml destinations

Test on libvirt-6.0.0-13.virtcov.el8.x86_64 qemu-kvm-4.2.0-15.module+el8.2.0+6029+618ef2ec.x86_64
1. Prepare block, gluster cluster, nbd server, rbd cluster, iscsi server
2. Prepare luks file on these backends above
3. Prepare secrets for luks, iscsi, rbd
4. Start VM do blockcopy with --reuse-external
for i in *dest.xml;do
        echo $i:                                                                                                                                                                       
        virsh blockcopy new sda --xml $i --pivot --transient-job --verbose --wait --reuse-external
        virsh detach-disk new sda
        sleep 5
        virsh attach-disk new /tmp/disk sda
done

Results:
block-dest.xml:
Block Copy: [100 %]
Successfully pivoted
Disk detached successfully

Disk attached successfully

file-dest.xml:
Block Copy: [100 %]
Successfully pivoted
Disk detached successfully

Disk attached successfully

gluster-dest.xml:
Block Copy: [100 %]
Successfully pivoted
Disk detached successfully

Disk attached successfully

iscsi-dest.xml:
Block Copy: [100 %]
Successfully pivoted
Disk detached successfully

Disk attached successfully

nbd-dest.xml:
Block Copy: [100 %]
Successfully pivoted
Disk detached successfully

Disk attached successfully

rbd-dest.xml:
Block Copy: [100 %]
Successfully pivoted
Disk detached successfully

Disk attached successfully


For migration with --copy-storage-all, I will test it after https://bugzilla.redhat.com/show_bug.cgi?id=1814923 fixed

For blockcopy with slice&luks in dest xml, I will test it after https://bugzilla.redhat.com/show_bug.cgi?id=1814975 fixed

Test for block-mirror of copy storage migration on libvirt-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64 qemu-kvm-4.2.0-17.module+el8.2.0+6141+0f540f16.x86_64

1. To make the testing scenarios more complex, setup env for the tls migration tunnel:
Prepare tls CA&client certs on the src host, CA&server certs on the dest host
Enable tls verify on qemu.conf:
default_tls_x509_cert_dir = "/etc/pki/qemu"
default_tls_x509_verify = 1


2. Prepare file, block, iscsi, nbd, nvme, gluster, rbd environments
To make the tests more complex
   prepare iscsi with chap auth, rbd with cephx auth, nbd with tls creds
   prepare rbd and gluster with multi-nodes


3. Create a qcow2 luks file
# qemu-img create --object secret,data=redhat,id=sec0 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /tmp/new 100M

Convert it to these block backends above:
# qemu-img convert [--object,...] /tmp/new 'json:{"file": BLOCKDEV_SCHEMA }' -O raw -f raw


4. Prepare secrets and domain xml with the disk backends:
Block:
<disk type="block" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source dev="/dev/vg/new">
    <encryption format="luks">
      <secret type="passphrase" usage="luks"/>
    </encryption>
  </source>
  <target dev="sda" bus="scsi"/>
</disk>

File:
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source file="/var/lib/libvirt/images/luks-dest">
    <encryption format="luks">
      <secret type="passphrase" usage="luks"/>
    </encryption>
  </source>
  <target dev="sda" bus="scsi"/>
</disk>

Gluster:
<disk type="network" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source protocol='gluster' name='gv/luks-dest'>
    <encryption format="luks">
      <secret type="passphrase" usage="luks"/>
    </encryption>
    <host name='NODE1'/>
    <host name='NODE2'/>
  </source>
  <target dev="sda" bus="scsi"/>
</disk>

Iscsi:
<disk type="network" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source protocol='iscsi' name='iqn.2020-03.com.libvirt:iscsi-chap/1'>
    <encryption format="luks">
      <secret type="passphrase" usage="luks"/>
    </encryption>
    <auth username='redhat'>
      <secret type='iscsi' usage='iscsi'/>
    </auth>
    <host name='HOST'/>
  </source>
  <target dev="sda" bus="scsi"/>
</disk>

Nbd:
<disk type="network" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source protocol='nbd' name='nbd-tls' tls='yes'>
    <encryption format="luks">
      <secret type="passphrase" usage="luks"/>
    </encryption>
    <host name='HOST' port='10811'/>
  </source>
  <target dev="sda" bus="scsi"/>
</disk>

Nvme:
<disk type="nvme" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source type="pci" managed="yes" namespace="1">
        <address domain="0x0000" bus="0x44" slot="0x00" function="0x0"/>
	    <encryption format="luks">
	      <secret type="passphrase" usage="luks"/>
	    </encryption>
      </source>
      <target dev="sda" bus="scsi"/>
    </disk>

Rbd:
<disk type="network" device="disk">
  <driver name="qemu" type="raw"/>
  <source protocol='rbd' name='rbd/luks-dest'>
    <encryption format="luks">
      <secret type="passphrase" usage="luks"/>
    </encryption>
    <auth username='admin'>
      <secret type='ceph' usage='ceph_example'/>
    </auth>
    <host name='NODE1'/>
    <host name='NODE2'/>
    <host name='NODE3'/>
  </source>
  <target dev="sda" bus="scsi"/>
</disk>


4. Prepare migration env(hostname, add ports on firewall for migration)

5. Migrate with --tls --copy-storage-all --xml
# virsh migrate fedora31 qemu+ssh://DEST_HOST/system --copy-storage-all --tls --xml file-dest.xml-dom.xml --verbose


Test results:
File: pass
Block: pass
Iscsi: pass
Rbd: pass
Gluster: pass
But if gv/luks-dest is not pre-created, it will report a Pemission Denied error and then trigger segment fault of https://bugzilla.redhat.com/show_bug.cgi?id=1783187

Nvme:
# virsh migrate fedora31 qemu+ssh://hp-dl385g10-15.lab.eng.pek2.redhat.com/system --copy-storage-all --tls --xml nvme-dest.xml-dom.xml --verbose
error: internal error: cannot precreate storage for disk type 'nvme'

Nbd:
# virsh migrate fedora31 qemu+ssh://hp-dl385g10-15.lab.eng.pek2.redhat.com/system --copy-storage-all --xml nbd-dest.xml-dom.xml
error: operation failed: migration of disk sda failed: Invalid argument
And it could be reproduced without --tls

NBD passed when use the qcow2 luks file as nbd backend

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017