RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2001327 - Wrong backing-fmt in QMP command when do push mode incremental backup for luks encrypted qcow2 disk
Summary: Wrong backing-fmt in QMP command when do push mode incremental backup for luk...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
Depends On: 1995865
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-05 14:24 UTC by John Ferlan
Modified: 2022-05-17 13:03 UTC (History)
7 users (show)

Fixed In Version: libvirt-7.7.0-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1995865
Environment:
Last Closed: 2022-05-17 12:45:08 UTC
Type: Bug
Target Upstream Version: 7.7.0
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-96255 0 None None None 2021-09-05 14:24:47 UTC
Red Hat Product Errata RHBA-2022:2390 0 None None None 2022-05-17 12:45:33 UTC

Description John Ferlan 2021-09-05 14:24:14 UTC
+++ This bug was initially created as a clone of Bug #1995865 +++

Description of problem:
Wrong backing-fmt in QMP command when do push mode incremental backup for luks encrypted qcow2 disk

Version-Release number of selected component:
libvirt-7.6.0-2.module+el8.5.0+12219+a5ea13d2.x86_64
qemu-kvm-6.0.0-27.module+el8.5.0+12121+c40c8708.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a vm with luks encrypted qcow2 image.
vm xml snippet:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/luks.full.backup.qcow2' index='1'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb'/>
        </encryption>
      </source>
      <backingStore/>
      <target dev='vdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </disk>

# qemu-img info -U /var/lib/libvirt/images/luks.full.backup.qcow2
image: /var/lib/libvirt/images/luks.full.backup.qcow2
file format: qcow2
virtual size: 100 MiB (104857600 bytes)
disk size: 123 MiB
encrypted: yes
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    encrypt:
        ivgen alg: plain64
        hash alg: sha256
        cipher alg: aes-256
        uuid: 259a3bae-82fc-47cd-bf64-a9edd571f854
        format: luks
        cipher mode: xts
        slots:
            [0]:
                active: true
                iters: 1305498
                key offset: 4096
                stripes: 4000
            [1]:
                active: false
                key offset: 262144
            [2]:
                active: false
                key offset: 520192
            [3]:
                active: false
                key offset: 778240
            [4]:
                active: false
                key offset: 1036288
            [5]:
                active: false
                key offset: 1294336
            [6]:
                active: false
                key offset: 1552384
            [7]:
                active: false
                key offset: 1810432
        payload offset: 2068480
        master key iters: 324727
    corrupt: false
    extended l2: false

2. Create full backup and checkpoint for disk vdd
# cat backup-push-full-blk.xml 
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdc' backup='no' />
    <disk name='vdd' backup='yes' type='block' backupmode='full'>
      <driver type='qcow2'/>
      <target dev='/dev/vdb2'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/var/lib/libvirt/images/luks.backup.qcow2'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>

# cat checkpoint.xml 
<domaincheckpoint>
  <name>check10</name>
  <disks>
    <disk checkpoint='no' name='vda'/>
    <disk checkpoint='no' name='vdc'/>
    <disk checkpoint='bitmap' name='vdd'/>
  </disks>
</domaincheckpoint>

# virsh backup-begin backup-push-full-blk.xml

3. Do incremental backup for disk vdd
# cat backup.xml
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdc' backup='no' />
    <disk name='vdd' backup='yes' type='block' backupmode='incremental' incremental='check10'>
     <driver type='qcow2'/>
      <target dev='/dev/vdb3'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/var/lib/libvirt/images/luks.backup.qcow2'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>

# virsh backup-begin vm1 backup.xml
Backup started

# virsh domjobinfo vm1 --completed
Job type:         Completed  
Operation:        Backup      
Time elapsed:     6343         ms
File processed:   128.000 KiB
File remaining:   0.000 B
File total:       128.000 KiB

4. Check backup image info:
# qemu-img info --backing-chain /dev/vdb3
qemu-img: Could not open '/var/lib/libvirt/images/luks.full.backup.qcow2': Volume is not in LUKS format

# qemu-img info /dev/vdb3
image: /dev/vdb3
file format: qcow2
virtual size: 100 MiB (104857600 bytes)
disk size: 0 B
encrypted: yes
cluster_size: 65536
backing file: /var/lib/libvirt/images/luks.full.backup.qcow2
backing file format: luks
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    encrypt:
        ivgen alg: plain64
        hash alg: sha256
        cipher alg: aes-256
        uuid: 42edd449-2e13-4f9f-b241-03001c2a7b88
        format: luks
        cipher mode: xts
        slots:
            [0]:
                active: true
                iters: 1110778
                key offset: 4096
                stripes: 4000
            [1]:
                active: false
                key offset: 262144
            [2]:
                active: false
                key offset: 520192
            [3]:
                active: false
                key offset: 778240
            [4]:
                active: false
                key offset: 1036288
            [5]:
                active: false
                key offset: 1294336
            [6]:
                active: false
                key offset: 1552384
            [7]:
                active: false
                key offset: 1810432
        payload offset: 2068480
        master key iters: 323408
    corrupt: false
    extended l2: false

5. Try to rebase the backup image:
# qemu-img rebase --object secret,id=sec0,data=redhat1 -b 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "host_device", "filename": "/dev/vdb2"}}'  -f qcow2 -F qcow2 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "host_device", "filename": "/dev/vdb3"}}'
qemu-img: Could not open 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "host_device", "filename": "/dev/vdb3"}}': Could not open backing file: Parameter 'key-secret' is required for cipher


6. Check QMP commands during incremental backup:
{"execute":"query-named-block-nodes","arguments":{"flat":true},"id":"libvirt-482"}
{"execute":"blockdev-add","arguments":{"driver":"host_device","filename":"/dev/vdb3","node-name":"libvirt-11-storage","auto-read-only":true,"discard":"unmap"},"id":"libvirt-483"}
{"execute":"object-add","arguments":{"qom-type":"secret","id":"libvirt-11-format-encryption-secret0","data":"WNgF+Sx+HVyEIu7YfAC8jQ==","keyid":"masterKey0","iv":"oC66OwWrXfe3jyP1kXbVHw==","format":"base64"},"id":"libvirt-484"}
{"execute":"blockdev-create","arguments":{"job-id":"create-libvirt-11-format","options":{"driver":"qcow2","file":"libvirt-11-storage","size":104857600,"cluster-size":65536,"backing-file":"/var/lib/libvirt/images/luks.full.backup.qcow2","backing-fmt":"luks","encrypt":{"key-secret":"libvirt-11-format-encryption-secret0","format":"luks"}}},"id":"libvirt-485"}
{"execute":"query-jobs","id":"libvirt-486"}
{"execute":"job-dismiss","arguments":{"id":"create-libvirt-11-format"},"id":"libvirt-487"}
{"execute":"blockdev-add","arguments":{"node-name":"libvirt-11-format","read-only":false,"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-11-format-encryption-secret0"},"file":"libvirt-11-storage","backing":"libvirt-1-format"},"id":"libvirt-488"}
{"execute":"transaction","arguments":{"actions":[{"type":"block-dirty-bitmap-add","data":{"node":"libvirt-1-format","name":"backup-vdd","persistent":false,"disabled":true,"granularity":65536}},{"type":"block-dirty-bitmap-merge","data":{"node":"libvirt-1-format","target":"backup-vdd","bitmaps":[{"node":"libvirt-1-format","name":"check10"}]}},{"type":"blockdev-backup","data":{"device":"libvirt-1-format","job-id":"backup-vdd-libvirt-1-format","target":"libvirt-11-format","sync":"incremental","bitmap":"backup
{"execute":"query-jobs","id":"libvirt-490"}
{"execute":"job-dismiss","arguments":{"id":"backup-vdd-libvirt-1-format"},"id":"libvirt-491"}
{"execute":"blockdev-del","arguments":{"node-name":"libvirt-11-format"},"id":"libvirt-492"}
{"execute":"blockdev-del","arguments":{"node-name":"libvirt-11-storage"},"id":"libvirt-493"}
{"execute":"object-del","arguments":{"id":"libvirt-11-format-encryption-secret0"},"id":"libvirt-494"}
{"execute":"block-dirty-bitmap-remove","arguments":{"node":"libvirt-1-format","name":"backup-vdd"},"id":"libvirt-495"}


Actual results:
backing-fmt is wrong for incremental backup image, it leads to failure of image rebase.

Expected results:
backing-fmt should be qcow2


Additional info:

--- Additional comment from Peter Krempa on 2021-08-20 13:45:50 UTC ---

Fixed upstream:

commit ca444a2eb27996b8940217d045c31a98882092e1
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 20 13:25:19 2021 +0200

    qemublocktest: Add test for creating a qcow2 on top of an luks-encrypted qcow2
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 15ab65583588702eb15b9f323fa57b8b5441a8ea
Author: Peter Krempa <pkrempa>
Date:   Fri Aug 20 13:26:13 2021 +0200

    qemu: block: Use correct format name when formatting overlay of qcow2+luks
    
    A logic bug in the code creating overlays on existing images resulted
    into wrongly using "luks" instead of "qcow2" for the backing format if
    the backing image is an luks-encrypted qcow2. The special format munging
    is needed only for raw luks images.
    
    In practice the impact is not as critical as to use encrypted images in
    the backing chain the user must fully describe the backing chain
    including backing images to provide encryption keys, which overrides the
    metadata recorded in the qcow2 header.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

Comment 1 yisun 2021-09-30 09:20:09 UTC
verified with: libvirt-7.7.0-3.el9.x86_64
1. prepare x partitions /dev/sdb1 to /dev/sdb3 (Can use iscsi or any other block device)
[root@dell-per740xd-25 ~]# ll /dev/sdb?
brw-rw----. 1 root disk 8, 17 Sep 30 03:58 /dev/sdb1
brw-rw----. 1 root disk 8, 18 Sep 30 03:58 /dev/sdb2
brw-rw----. 1 root disk 8, 19 Sep 30 03:58 /dev/sdb3
brw-rw----. 1 root disk 8, 20 Sep 30 03:58 /dev/sdb4

2. prepare a qcow2 native luks-encrypted image and use it in vm as vdb
[root@dell-per740xd-25 ~]# qemu-img create --object secret,id=sec0,data=redhat -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/vdb.qcow2 200M
Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 extended_l2=off compression_type=zlib size=209715200 lazy_refcounts=off refcount_bits=16


[root@dell-per740xd-25 ~]# cat secret.xml 
<secret ephemeral='no' private='yes'>
<description>LUKS Secret</description>
<usage type='volume'>
	<volume>/just/a/path</volume>
</usage>
</secret>

[root@dell-per740xd-25 ~]# virsh secret-define secret.xml 
Secret cba89c46-bae8-4e17-8b0b-2326fcd25d73 created

[root@dell-per740xd-25 ~]# MYSECRET=`printf %s "redhat" | base64`

[root@dell-per740xd-25 ~]# virsh secret-set-value cba89c46-bae8-4e17-8b0b-2326fcd25d73 $MYSECRET
error: Passing secret value as command-line argument is insecure!
Secret value set

[root@dell-per740xd-25 ~]# virsh edit vm1
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vdb.qcow2' index='1'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='cba89c46-bae8-4e17-8b0b-2326fcd25d73'/>
        </encryption>
      </source>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@dell-per740xd-25 ~]# virsh start vm1
Domain 'vm1' started

3. prepare round 1 backup xmls
[root@dell-per740xd-25 ~]# cat backup_full.xml 
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdb' backup='yes' type='block' backupmode='full'>
      <driver type='qcow2'/>
      <target dev='/dev/sdb1'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/just/a/path'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>
[root@dell-per740xd-25 ~]# cat ck1.xml 
<domaincheckpoint>
  <name>ck1</name>
  <disks>
    <disk checkpoint='no' name='vda'/>
    <disk checkpoint='bitmap' name='vdb'/>
  </disks>
</domaincheckpoint>

4. start round 1 - full backup
[root@dell-per740xd-25 ~]# virsh backup-begin vm1 backup_full.xml ck1.xml 
Backup started

[root@dell-per740xd-25 ~]# virsh domjobinfo vm1 --completed
Job type:         Completed   
Operation:        Backup      
Time elapsed:     7328         ms
File processed:   200.000 MiB
File remaining:   0.000 B
File total:       200.000 MiB

5. prepare round 2 backup xml
[root@dell-per740xd-25 ~]# cat backup_inc.xml 
<domainbackup mode='push'>
  <disks>
    <disk name='vda' backup='no' />
    <disk name='vdb' backup='yes' type='block' backupmode='incremental' incremental='ck1'>
     <driver type='qcow2'/>
      <target dev='/dev/sdb2'>
        <encryption format='luks'>
          <secret type='passphrase' usage='/just/a/path'/>
        </encryption>
      </target>
    </disk>
  </disks>
</domainbackup>

6. start round 2 - incremental backup
[root@dell-per740xd-25 ~]# virsh backup-begin vm1 backup_inc.xml 
Backup started

[root@dell-per740xd-25 ~]# virsh domjobinfo vm1 --completed
Job type:         Completed   
Operation:        Backup      
Time elapsed:     7319         ms

7. (optional step) check the libvirtd debug log. When create incremental backup image, the base image's format is 'qcow2' but not 'luks' ("backing-file":"/var/lib/libvirt/images/vdb.qcow2","backing-fmt":"qcow2")
2021-09-30 09:07:09.686+0000: 163978: info : qemuMonitorSend:956 : QEMU_MONITOR_SEND_MSG: mon=0x7fc240005090 msg={"execute":"blockdev-create","arguments":{"job-id":"create-libvirt-4-format","options":{"driver":"qcow2","file":"libvirt-4-storage","size":209715200,"cluster-size":65536,"backing-file":"/var/lib/libvirt/images/vdb.qcow2","backing-fmt":"qcow2","encrypt":{"key-secret":"libvirt-4-format-encryption-secret0","format":"luks"}}},"id":"libvirt-29"}


8. check the incremtnal backup target file's backing chain info. 
[root@dell-per740xd-25 ~]# qemu-img info --backing-chain /dev/sdb2 -U 
image: /dev/sdb2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
...
encrypted: yes
cluster_size: 65536
backing file: /var/lib/libvirt/images/vdb.qcow2
backing file format: qcow2
...
image: /var/lib/libvirt/images/vdb.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 3.25 MiB
encrypted: yes
...

Comment 4 errata-xmlrpc 2022-05-17 12:45:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390


Note You need to log in before you can comment on or make changes to this bug.