Bug 1788898 - virtual disk's backing file encryption info lost when start/restart vm
Summary: virtual disk's backing file encryption info lost when start/restart vm
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
Depends On:
Blocks: 1375855
TreeView+ depends on / blocked
 
Reported: 2020-01-08 10:48 UTC by yisun
Modified: 2020-11-06 04:48 UTC (History)
7 users (show)

Fixed In Version: libvirt-6.0.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-05 09:55:17 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2017 None None None 2020-05-05 09:56:58 UTC

Internal Links: 1789310

Description yisun 2020-01-08 10:48:38 UTC
Description:
virtual disk's backing file encryption info lost when start/restart vm

Versions:
qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64
libvirt-5.10.0-1.module+el8.2.0+5135+ed3b2489.x86_64

How reproducible:
100%

Steps:
1. making a disk chain: top.qcow2 ---> base.qcow2, in which base.qcow2 is luks encrypted qcow2 file
# qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 1G
Formatting 'base.qcow2', fmt=qcow2 size=1073741824 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16

# qemu-img create -f qcow2 top.qcow2 100M
Formatting 'top.qcow2', fmt=qcow2 size=104857600 cluster_size=65536 lazy_refcounts=off refcount_bits=16

# qemu-img rebase --object secret,id=sec0,data=123456 --image-opts driver=qcow2,file.filename=top.qcow2 -b 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "file", "filename": "/home/images/base.qcow2"}}'

2. add the disk chain into a vm's xml
# virsh secret-list
 UUID                                   Usage
--------------------------------------------------------------------------
 0a5fa242-64a8-48dd-932e-340f56b5ad48   volume /home/images/base.qcow2


# MYSECRET=`printf %s "123456" | base64`


# virsh secret-set-value 8d17b267-7d6d-475d-8ba0-2549acc0d4ce $MYSECRET
Secret value set


# virsh edit vm1
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/images/top.qcow2'/>
      <backingStore type='file'>
        <format type='qcow2'/>
        <source file='/home/images/base.qcow2'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='0a5fa242-64a8-48dd-932e-340f56b5ad48'/>
        </encryption>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
    </disk>

3. start the vm, the vm can be started normally
# virsh start vm1
Domain vm1 started


qemu process has correct info about base.qcow2's encryption
# ps -ef | grep vm1 
qemu     23898     1 99 05:22 ?        00:00:37 /usr/libexec/qemu-kvm -name guest=vm1...
-device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=libvirt-3-format,id=virtio-disk0,bootindex=1 -object secret,id=libvirt-2-format-luks-secret0,data=6OiOyd7roAIby5YbQ2QSrQ==,keyid=masterKey0,iv=/gWthTSNaeNBP0KHY4m8Gg==,format=base64 -blockdev {"driver":"file","filename":"/home/images/base.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-2-format-luks-secret0"},"file":"libvirt-2-storage","backing":null} ...


4. check the vm's xml now, <encryption> part gone
# virsh dumpxml vm1| awk '/<disk/,/<\/disk/'
setlocale: No such file or directory
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/images/top.qcow2' index='1'/>
      <backingStore type='file' index='2'>
        <format type='qcow2'/>
        <source file='/home/images/base.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

5. since the encryption part gone, vm cannot be restarted due to missing cipher
# virsh destroy vm1; virsh start vm1
Domain vm1 destroyed

error: Failed to start domain vm1
error: internal error: process exited while connecting to monitor: 2020-01-08T10:25:59.327051Z qemu-kvm: -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"qcow2","file":"libvirt-2-storage","backing":null}: Parameter 'encrypt.key-secret' is required for cipher


Actual result:
In step4, the encryption part lost

Expected resutl:
The backing file's encryption info should be kept so vm can be resatrted successfully

Comment 1 Peter Krempa 2020-01-13 12:00:33 UTC
Fixed upstream:

commit 3f2d167d9c733f588e693d44d7aa9b21dcb415c7 (HEAD -> master, origin/master, origin/HEAD)
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Fri Jan 10 17:25:16 2020 +0100

    conf: Always format storage source auth and encryption under <source> for backing files
    
    Historically there are two places where we format authentication and
    encryption for a disk. The logich which formats it for backing files was
    flawed though and didn't format it at all. This worked if the image
    became a backing file through the means of a snapshot but not directly.
    
    Force formatting of the source and encryption for any non-disk case to
    fix the issue.
    
    This caused problems in many places as we use the formatter to copy the
    definition. Effectively any copy lost the secret definition.

Comment 3 yisun 2020-02-20 08:37:59 UTC
Test result: PASS
[root@hp-dl320eg8-05 bz1788898]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-4"
qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64
libvirt-6.0.0-5.module+el8.2.0+5765+64816f89.x86_64


1. prepare a secret for luks image
[root@hp-dl320eg8-05 bz1788898]# virsh secret-define sec.xml
Secret f981dd17-143f-45bc-88e6-ed1fe20ce9da created


[root@hp-dl320eg8-05 bz1788898]# MYSECRET=`printf %s "123456" | base64`
[root@hp-dl320eg8-05 bz1788898]#
[root@hp-dl320eg8-05 bz1788898]# virsh secret-set-value f981dd17-143f-45bc-88e6-ed1fe20ce9da $MYSECRET
Secret value set

2. prepare a luks image 'luks.qcow2'
[root@hp-dl320eg8-05 bz1788898]# qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G
Formatting '/var/lib/libvirt/images/luks.qcow2', fmt=qcow2 size=1073741824 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16

3. create a new image 'top.qcow2' based on luks.qcow2
[root@hp-dl320eg8-05 bz1788898]# qemu-img create -f qcow2 /var/lib/libvirt/images/top.qcow2 100M
Formatting '/var/lib/libvirt/images/top.qcow2', fmt=qcow2 size=104857600 cluster_size=65536 lazy_refcounts=off refcount_bits=16


[root@hp-dl320eg8-05 bz1788898]# qemu-img rebase --object secret,id=sec0,data=123456 --image-opts driver=qcow2,file.filename=/var/lib/libvirt/images/top.qcow2 -b 'json:{"encrypt.key-secret": "sec0", "driver": "qcow2", "file": {"driver": "file", "filename": "/var/lib/libvirt/images/luks.qcow2"}}'

4. prepare correct disk xml in vm as follow
[root@hp-dl320eg8-05 bz1788898]# virsh dumpxml vm1 --inactive| awk '/<disk/,/<\/disk/'
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/top.qcow2'/>
      <backingStore type='file'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/luks.qcow2'>
          <encryption format='luks'>
            <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
          </encryption>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
...

5. start vm
[root@hp-dl320eg8-05 bz1788898]# virsh start vm1
Domain vm1 started



6. restart vm, now everything is ok: vm can be started; disk xml is still correct; qemu process has correct secret info
[root@hp-dl320eg8-05 bz1788898]# virsh destroy vm1; sleep 2; virsh start vm1
Domain vm1 destroyed
Domain vm1 started

[root@hp-dl320eg8-05 bz1788898]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/top.qcow2' index='1'/>
      <backingStore type='file' index='2'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/luks.qcow2'>
          <encryption format='luks'>
            <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
          </encryption>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@hp-dl320eg8-05 bz1788898]# ps -ef | grep vm1 | grep luks
... -object secret,id=libvirt-2-format-luks-secret0,data=WQeoTGCWco2A3XA0LvgmiQ==,keyid=masterKey0,iv=xpit3ZeGEYjnf7DUn0ewkg==,format=base64 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/luks.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-2-format-luks-secret0"},"file":"libvirt-2-storage","backing":null}

(ps: auth part is found and covered by bz1789310)

Comment 5 errata-xmlrpc 2020-05-05 09:55:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.