Bug 2001327

Summary: Wrong backing-fmt in QMP command when do push mode incremental backup for luks encrypted qcow2 disk
Product: Red Hat Enterprise Linux 9 Reporter: John Ferlan <jferlan>
Component: libvirtAssignee: Peter Krempa <pkrempa>
libvirt sub component: General QA Contact: yisun
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: fjin, jdenemar, lmen, pkrempa, virt-maint, xuzhang, yisun
Version: 9.0Keywords: AutomationTriaged, Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-7.7.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1995865 Environment:
Last Closed: 2022-05-17 12:45:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 7.7.0
Embargoed:
Bug Depends On: 1995865    
Bug Blocks:    

Description John Ferlan 2021-09-05 14:24:14 UTC 
+++ This bug was initially created as a clone of Bug #1995865 +++

Description of problem:
Wrong backing-fmt in QMP command when do push mode incremental backup for luks encrypted qcow2 disk

Version-Release number of selected component:
libvirt-7.6.0-2.module+el8.5.0+12219+a5ea13d2.x86_64
qemu-kvm-6.0.0-27.module+el8.5.0+12121+c40c8708.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a vm with luks encrypted qcow2 image.
vm xml snippet:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/luks.full.backup.qcow2' index='1'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb'/>
        </encryption>
      </source>
      <backingStore/>
      <target dev='vdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </disk>

# qemu-img info -U /var/lib/libvirt/images/luks.full.backup.qcow2
image: /var/lib/libvirt/images/luks.full.backup.qcow2
file format: qcow2
virtual size: 100 MiB (104857600 bytes)
disk size: 123 MiB
encrypted: yes
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    encrypt:
        ivgen alg: plain64
        hash alg: sha256
        cipher alg: aes-256
        uuid: 259a3bae-82fc-47cd-bf64-a9edd571f854
        format: luks
        cipher mode: xts
        slots:
            [0]:
                active: true
                iters: 1305498
                key offset: 4096
                stripes: 4000
            [1]:
                active: false
                key offset: 262144
            [2]:
                active: false
                key offset: 520192
            [3]:
                active: false
                key offset: 778240
            [4]:
                active: false
                key offset: 1036288
            [5]:
                active: false
                key offset: 1294336
            [6]:
                active: false
                key offset: 1552384
            [7]:
                active: false
                key offset: 1810432
        payload offset: 2068480
        master key iters: 324727
    corrupt: false
    extended l2: false

2. Create full backup and checkpoint for disk vdd
# cat backup-push-full-blk.xml 
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdc' backup='no' />
    <disk name='vdd' backup='yes' type='block' backupmode='full'>
      <driver type='qcow2'/>
      <target dev='/dev/vdb2'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/var/lib/libvirt/images/luks.backup.qcow2'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>

# cat checkpoint.xml 
<domaincheckpoint>
  <name>check10</name>
  <disks>
    <disk checkpoint='no' name='vda'/>
    <disk checkpoint='no' name='vdc'/>
    <disk checkpoint='bitmap' name='vdd'/>
  </disks>
</domaincheckpoint>

# virsh backup-begin backup-push-full-blk.xml

3. Do incremental backup for disk vdd
# cat backup.xml
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdc' backup='no' />
    <disk name='vdd' backup='yes' type='block' backupmode='incremental' incremental='check10'>
     <driver type='qcow2'/>
      <target dev='/dev/vdb3'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/var/lib/libvirt/images/luks.backup.qcow2'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>

# virsh backup-begin vm1 backup.xml
Backup started

# virsh domjobinfo vm1 --completed
Job type:         Completed  
Operation:        Backup      
Time elapsed:     6343         ms
File processed:   128.000 KiB
File remaining:   0.000 B
File total:       128.000 KiB

4. Check backup image info:
# qemu-img info --backing-chain /dev/vdb3
qemu-img: Could not open '/var/lib/libvirt/images/luks.full.backup.qcow2': Volume is not in LUKS format

# qemu-img info /dev/vdb3
image: /dev/vdb3
file format: qcow2
virtual size: 100 MiB (104857600 bytes)
disk size: 0 B
encrypted: yes
cluster_size: 65536
backing file: /var/lib/libvirt/images/luks.full.backup.qcow2
backing file format: luks
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    encrypt:
        ivgen alg: plain64
        hash alg: sha256
        cipher alg: aes-256
        uuid: 42edd449-2e13-4f9f-b241-03001c2a7b88
        format: luks
        cipher mode: xts
        slots:
            [0]:
                active: true
                iters: 1110778
                key offset: 4096
                stripes: 4000
            [1]:
                active: false
                key offset: 262144
            [2]:
                active: false
                key offset: 520192
            [3]:
                active: false
                key offset: 778240
            [4]:
                active: false
                key offset: 1036288
            [5]:
                active: false
                key offset: 1294336
            [6]:
                active: false
                key offset: 1552384
            [7]:
                active: false
                key offset: 1810432
        payload offset: 2068480
        master key iters: 323408
    corrupt: false
    extended l2: false

5. Try to rebase the backup image:
# qemu-img rebase --object secret,id=sec0,data=redhat1 -b 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "host_device", "filename": "/dev/vdb2"}}'  -f qcow2 -F qcow2 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "host_device", "filename": "/dev/vdb3"}}'
qemu-img: Could not open 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "host_device", "filename": "/dev/vdb3"}}': Could not open backing file: Parameter 'key-secret' is required for cipher


6. Check QMP commands during incremental backup:
{"execute":"query-named-block-nodes","arguments":{"flat":true},"id":"libvirt-482"}
{"execute":"blockdev-add","arguments":{"driver":"host_device","filename":"/dev/vdb3","node-name":"libvirt-11-storage","auto-read-only":true,"discard":"unmap"},"id":"libvirt-483"}
{"execute":"object-add","arguments":{"qom-type":"secret","id":"libvirt-11-format-encryption-secret0","data":"WNgF+Sx+HVyEIu7YfAC8jQ==","keyid":"masterKey0","iv":"oC66OwWrXfe3jyP1kXbVHw==","format":"base64"},"id":"libvirt-484"}
{"execute":"blockdev-create","arguments":{"job-id":"create-libvirt-11-format","options":{"driver":"qcow2","file":"libvirt-11-storage","size":104857600,"cluster-size":65536,"backing-file":"/var/lib/libvirt/images/luks.full.backup.qcow2","backing-fmt":"luks","encrypt":{"key-secret":"libvirt-11-format-encryption-secret0","format":"luks"}}},"id":"libvirt-485"}
{"execute":"query-jobs","id":"libvirt-486"}
{"execute":"job-dismiss","arguments":{"id":"create-libvirt-11-format"},"id":"libvirt-487"}
{"execute":"blockdev-add","arguments":{"node-name":"libvirt-11-format","read-only":false,"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-11-format-encryption-secret0"},"file":"libvirt-11-storage","backing":"libvirt-1-format"},"id":"libvirt-488"}
{"execute":"transaction","arguments":{"actions":[{"type":"block-dirty-bitmap-add","data":{"node":"libvirt-1-format","name":"backup-vdd","persistent":false,"disabled":true,"granularity":65536}},{"type":"block-dirty-bitmap-merge","data":{"node":"libvirt-1-format","target":"backup-vdd","bitmaps":[{"node":"libvirt-1-format","name":"check10"}]}},{"type":"blockdev-backup","data":{"device":"libvirt-1-format","job-id":"backup-vdd-libvirt-1-format","target":"libvirt-11-format","sync":"incremental","bitmap":"backup
{"execute":"query-jobs","id":"libvirt-490"}
{"execute":"job-dismiss","arguments":{"id":"backup-vdd-libvirt-1-format"},"id":"libvirt-491"}
{"execute":"blockdev-del","arguments":{"node-name":"libvirt-11-format"},"id":"libvirt-492"}
{"execute":"blockdev-del","arguments":{"node-name":"libvirt-11-storage"},"id":"libvirt-493"}
{"execute":"object-del","arguments":{"id":"libvirt-11-format-encryption-secret0"},"id":"libvirt-494"}
{"execute":"block-dirty-bitmap-remove","arguments":{"node":"libvirt-1-format","name":"backup-vdd"},"id":"libvirt-495"}


Actual results:
backing-fmt is wrong for incremental backup image, it leads to failure of image rebase.

Expected results:
backing-fmt should be qcow2


Additional info:

--- Additional comment from Peter Krempa on 2021-08-20 13:45:50 UTC ---

Fixed upstream:

commit ca444a2eb27996b8940217d045c31a98882092e1
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 20 13:25:19 2021 +0200

    qemublocktest: Add test for creating a qcow2 on top of an luks-encrypted qcow2
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 15ab65583588702eb15b9f323fa57b8b5441a8ea
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 20 13:26:13 2021 +0200

    qemu: block: Use correct format name when formatting overlay of qcow2+luks
    
    A logic bug in the code creating overlays on existing images resulted
    into wrongly using "luks" instead of "qcow2" for the backing format if
    the backing image is an luks-encrypted qcow2. The special format munging
    is needed only for raw luks images.
    
    In practice the impact is not as critical as to use encrypted images in
    the backing chain the user must fully describe the backing chain
    including backing images to provide encryption keys, which overrides the
    metadata recorded in the qcow2 header.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>
Comment 1 yisun 2021-09-30 09:20:09 UTC 
verified with: libvirt-7.7.0-3.el9.x86_64
1. prepare x partitions /dev/sdb1 to /dev/sdb3 (Can use iscsi or any other block device)
[root@dell-per740xd-25 ~]# ll /dev/sdb?
brw-rw----. 1 root disk 8, 17 Sep 30 03:58 /dev/sdb1
brw-rw----. 1 root disk 8, 18 Sep 30 03:58 /dev/sdb2
brw-rw----. 1 root disk 8, 19 Sep 30 03:58 /dev/sdb3
brw-rw----. 1 root disk 8, 20 Sep 30 03:58 /dev/sdb4

2. prepare a qcow2 native luks-encrypted image and use it in vm as vdb
[root@dell-per740xd-25 ~]# qemu-img create --object secret,id=sec0,data=redhat -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/vdb.qcow2 200M
Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 extended_l2=off compression_type=zlib size=209715200 lazy_refcounts=off refcount_bits=16


[root@dell-per740xd-25 ~]# cat secret.xml 
<secret ephemeral='no' private='yes'>
<description>LUKS Secret</description>
<usage type='volume'>
	<volume>/just/a/path</volume>
</usage>
</secret>

[root@dell-per740xd-25 ~]# virsh secret-define secret.xml 
Secret cba89c46-bae8-4e17-8b0b-2326fcd25d73 created

[root@dell-per740xd-25 ~]# MYSECRET=`printf %s "redhat" | base64`

[root@dell-per740xd-25 ~]# virsh secret-set-value cba89c46-bae8-4e17-8b0b-2326fcd25d73 $MYSECRET
error: Passing secret value as command-line argument is insecure!
Secret value set

[root@dell-per740xd-25 ~]# virsh edit vm1
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vdb.qcow2' index='1'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='cba89c46-bae8-4e17-8b0b-2326fcd25d73'/>
        </encryption>
      </source>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@dell-per740xd-25 ~]# virsh start vm1
Domain 'vm1' started

3. prepare round 1 backup xmls
[root@dell-per740xd-25 ~]# cat backup_full.xml 
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdb' backup='yes' type='block' backupmode='full'>
      <driver type='qcow2'/>
      <target dev='/dev/sdb1'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/just/a/path'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>
[root@dell-per740xd-25 ~]# cat ck1.xml 
<domaincheckpoint>
  <name>ck1</name>
  <disks>
    <disk checkpoint='no' name='vda'/>
    <disk checkpoint='bitmap' name='vdb'/>
  </disks>
</domaincheckpoint>

4. start round 1 - full backup
[root@dell-per740xd-25 ~]# virsh backup-begin vm1 backup_full.xml ck1.xml 
Backup started

[root@dell-per740xd-25 ~]# virsh domjobinfo vm1 --completed
Job type:         Completed   
Operation:        Backup      
Time elapsed:     7328         ms
File processed:   200.000 MiB
File remaining:   0.000 B
File total:       200.000 MiB

5. prepare round 2 backup xml
[root@dell-per740xd-25 ~]# cat backup_inc.xml 
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdb' backup='yes' type='block' backupmode='incremental' incremental='ck1'>
     <driver type='qcow2'/>
      <target dev='/dev/sdb2'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/just/a/path'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>

6. start round 2 - incremental backup
[root@dell-per740xd-25 ~]# virsh backup-begin vm1 backup_inc.xml 
Backup started

[root@dell-per740xd-25 ~]# virsh domjobinfo vm1 --completed
Job type:         Completed   
Operation:        Backup      
Time elapsed:     7319         ms

7. (optional step) check the libvirtd debug log. When create incremental backup image, the base image's format is 'qcow2' but not 'luks' ("backing-file":"/var/lib/libvirt/images/vdb.qcow2","backing-fmt":"qcow2")
2021-09-30 09:07:09.686+0000: 163978: info : qemuMonitorSend:956 : QEMU_MONITOR_SEND_MSG: mon=0x7fc240005090 msg={"execute":"blockdev-create","arguments":{"job-id":"create-libvirt-4-format","options":{"driver":"qcow2","file":"libvirt-4-storage","size":209715200,"cluster-size":65536,"backing-file":"/var/lib/libvirt/images/vdb.qcow2","backing-fmt":"qcow2","encrypt":{"key-secret":"libvirt-4-format-encryption-secret0","format":"luks"}}},"id":"libvirt-29"}


8. check the incremtnal backup target file's backing chain info. 
[root@dell-per740xd-25 ~]# qemu-img info --backing-chain /dev/sdb2 -U 
image: /dev/sdb2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
...
encrypted: yes
cluster_size: 65536
backing file: /var/lib/libvirt/images/vdb.qcow2
backing file format: qcow2
...
image: /var/lib/libvirt/images/vdb.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 3.25 MiB
encrypted: yes
...
Comment 4 errata-xmlrpc 2022-05-17 12:45:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390