Description of problem: Start a vm using a rbd disk specifying the <auth../> element providing the authentication credentials. The <auth../> element disappears after creating external disk snapshot. The issue is also reproduced when using iscsi as backing file. It will cause committing to base image fails. Version-Release number of selected component (if applicable): libvirt-1.2.8-10.el7.x86_64 qemu-kvm-rhev-2.1.2-14.el7.x86_64 kernel-3.10.0-212.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. start vm with rbd disk specifying <auth> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='libvirt'> <secret type='ceph' usage='client.libvirt secret'/> </auth> <source protocol='rbd' name='libvirt-pool/rbd1.img'> <config file='/etc/ceph/ceph.conf'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> # virsh list --all Id Name State ---------------------------------------------------- 10 rbd running 2. create external disk snapshot # virsh snapshot-create-as rbd s1 --disk-only --diskspec vda,file=/tmp/rbd.s1 Domain snapshot s1 created [root@rhel7_test yy]# virsh snapshot-list rbd Name Creation Time State ------------------------------------------------------------ s1 2014-12-08 13:30:30 +0800 disk-snapshot 3.check the domain xml # virsh dumpxml rbd | grep disk -a6 <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/tmp/rbd.s1'/> <backingStore type='network' index='1'> <format type='raw'/> <source protocol='rbd' name='libvirt-pool/rbd1.img'> <config file='/etc/ceph/ceph.conf'/> </source> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> 4. check image backing chain # qemu-img info /tmp/rbd.s1 --backing-chain image: /tmp/rbd.s1 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 580K cluster_size: 65536 backing file: rbd:libvirt-pool/rbd1.img:id=libvirt:key=AQCI335UkAgXHhAA90By4w5NR6zb63LbbM0MGg==:auth_supported=cephx\;none:conf=/etc/ceph/ceph.conf backing file format: raw Format specific information: compat: 1.1 lazy refcounts: false image: rbd:libvirt-pool/rbd1.img:id=libvirt:key=AQCI335UkAgXHhAA90By4w5NR6zb63LbbM0MGg==:auth_supported=cephx\;none:conf=/etc/ceph/ceph.conf file format: raw virtual size: 8.0G (8589934592 bytes) disk size: unavailable cluster_size: 4194304 Actual results: In step 3, <auth../> element disappeared in <backingStore> element. Expected results: Keep <auth../> element in <backingStore> element Additional info: The issue is also reproduced when using iscsi as backing file. The issue will cause that committing to base image fails
Libvirt really needs to track the complete backing chain internally in such cases as otherwise the backing chain is re-loaded from the disk state and thus the <auth> information are lost (unless qemu would record them in the backing file name).
Note that this happens for every image in the backing chain which originally had the auth element. While it won't happen currently for snapshots during the lifetime of the VM it will happen if libvirtd is restarted or if the VM is restarted. It will be fixed with full backing chain tracking in libvirt.
*** Bug 1467287 has been marked as a duplicate of this bug. ***
*** Bug 1467271 has been marked as a duplicate of this bug. ***
The latest progress for <auth> element: Starting with libvirt 3.9.0 the auth element is preferred to be a sub-element of the source element. The element is still read and managed as a disk sub-element. Test problems in backing chain( external snapshot) when the auth element is a sub-element of the source element: 1. <auth> sub-element is missing after restart guest. 2. <auth> sub-element is missing after blockcommit/blockpull/blockcopy. Test version: libvirt-3.9.0-7.el7.x86_64 One of the test scenario: 1. Prepare a guest with iscsi auth disk. # virsh dumpxml rhel7 | grep disk -a6 ... <disk type='network' device='disk'> <driver name='qemu' type='qcow2'/> <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'> <host name='**IP**' port='3260'/> <auth username='redhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk>... 2. Create external disk and check xml. # for i in 1 2 3 4;do virsh snapshot-create-as rhel7 s$i --disk-only --diskspec vda,file=/tmp/rhel7.s$i;done # virsh dumpxml rhel7 | grep disk -a12 ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/rhel7.s4'/> <backingStore type='file' index='1'> <format type='qcow2'/> <source file='/tmp/rhel7.s3'/> <backingStore type='file' index='2'> <format type='qcow2'/> <source file='/tmp/rhel7.s2'/> <backingStore type='file' index='3'> <format type='qcow2'/> <source file='/tmp/rhel7.s1'/> <backingStore type='network' index='4'> <format type='qcow2'/> <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'> <host name='**IP**' port='3260'/> <auth username='redhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore>... 3. Do blockcommit from middle to middle. # virsh blockcommit rhel7 vda --top vda[1] --base vda[3] --wait --verbose --pivot # virsh dumpxml rhel7 | grep disk -a12 ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/rhel7.s4'/> <backingStore type='file' index='1'> <format type='qcow2'/> <source file='/tmp/rhel7.s1'/> <backingStore type='network' index='2'> <format type='qcow2'/> <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'> <host name='**IP**' port='3260'/> </source> --<auth> element is missing </backingStore>... Actual results: As above step 3. Expected results: # virsh dumpxml rhel7 | grep disk -a12 … <backingStore type='network' index='2'> <format type='qcow2'/> <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'> <host name='**IP**' port='3260'/> <auth username='redhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore>...
This bug is going to be addressed in next major release.
The following commits prepared for proper backing chain manipulation which does not drop the <auth> element and other configuration: commit 1bf3808207fdac0a90a91399d1d1f19e73d35676 Author: Peter Krempa <pkrempa> Date: Mon Jul 22 13:39:24 2019 +0200 qemu: Add -blockdev support for block pull job Introduce the handler for finalizing a block pull job which will allow to use it with blockdev. This patch also contains some additional machinery which is required to store all the relevant job data in the status XML which will also be reused with other block job types. commit 3f93884a4d047a012b968c62b94ea07dadd1759b Author: Peter Krempa <pkrempa> Date: Mon Jul 22 13:39:24 2019 +0200 qemu: Add -blockdev support for block commit job Introduce the handler for finalizing a block commit and active bloc commit job which will allow to use it with blockdev. commit ce7229a3b0d28479e0f123efce3fa73617889a50 Author: Peter Krempa <pkrempa> Date: Mon Jul 22 13:59:01 2019 +0200 qemu: Add blockdev support for the block copy job Implement job handling for the block copy job (drive/blockdev-mirror) when using -blockdev. In contrast to the previously implemented blockjobs the block copy job introduces new images to the running qemu instance, thus requires a bit more handling. When copying to new images the code now makes use of blockdev-create to format the images explicitly rather than depending on automagic qemu behaviour. The blockdev feature was enabled since: commit c6a9e54ce3252196f1fc6aa9e57537a659646d18 Author: Peter Krempa <pkrempa> Date: Mon Jan 7 11:45:19 2019 +0100 qemu: enable blockdev support Now that all pieces are in place (hopefully) let's enable -blockdev. We base the capability on presence of the fix for 'auto-read-only' on files so that blockdev works properly, mandate that qemu supports explicit SCSI id strings to avoid ABI regression and that the fix for 'savevm' is present so that internal snapshots work. v5.9.0-390-gc6a9e54ce3 and requires upstream qemu-4.2 or appropriate downstream.
Test result: PASS [root@hp-dl320eg8-05 bz1171569]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-4" qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64 libvirt-6.0.0-5.module+el8.2.0+5765+64816f89.x86_64 1. having a vm with authenticated iscsi device [root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' … <disk type='network' device='disk'> <driver name='qemu' type='raw'/> <auth username='rhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> <source protocol='iscsi' name='iqn.1992-01.com.example/0' index='1'> <host name='127.0.0.1' port='3260'/> </source> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> [root@hp-dl320eg8-05 bz1171569]# virsh domstate vm1 running [root@hp-dl320eg8-05 bz1171569]# virsh domblklist vm1 Target Source --------------------------------------------- … vdb iqn.1992-01.com.example/0 2. create 1 snapshot and see if the authentication part works fine in vm's xml [root@hp-dl320eg8-05 bz1171569]# virsh snapshot-create-as vm1 snap1 --disk-only --diskspec vdb,file=/tmp/vdb.s1 Domain snapshot snap1 created [root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' … <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/vdb.s1' index='4'/> <backingStore type='network' index='1'> <format type='raw'/> <source protocol='iscsi' name='iqn.1992-01.com.example/0'> <host name='127.0.0.1' port='3260'/> <auth username='rhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> [root@hp-dl320eg8-05 bz1171569]# virsh destroy vm1; virsh start vm1 Domain vm1 destroyed Domain vm1 started [root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' … <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/vdb.s1' index='1'/> <backingStore type='network' index='2'> <format type='raw'/> <source protocol='iscsi' name='iqn.1992-01.com.example/0'> <host name='127.0.0.1' port='3260'/> <auth username='rhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> 3. create snapshot 2 to 4 [root@hp-dl320eg8-05 bz1171569]# for i in {2..4}; do virsh snapshot-create-as vm1 snap$i --disk-only --diskspec vdb,file=/tmp/vdb.s$i; done Domain snapshot snap2 created Domain snapshot snap3 created Domain snapshot snap4 created [root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' … <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/vdb.s4' index='10'/> <backingStore type='file' index='8'> <format type='qcow2'/> <source file='/tmp/vdb.s3'/> <backingStore type='file' index='6'> <format type='qcow2'/> <source file='/tmp/vdb.s2'/> <backingStore type='file' index='1'> <format type='qcow2'/> <source file='/tmp/vdb.s1'/> <backingStore type='network' index='2'> <format type='raw'/> <source protocol='iscsi' name='iqn.1992-01.com.example/0'> <host name='127.0.0.1' port='3260'/> <auth username='rhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore> </backingStore> </backingStore> </backingStore> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> 5. do a blockcommit/blockpull to see if the auth part still exists [root@hp-dl320eg8-05 bz1171569]# virsh blockcommit vm1 vdb --top vdb[1] --base vdb[4] --wait --verbose --pivot Block commit: [100 %] Successfully pivoted [root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' … <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/vdb.s1' index='4'/> <backingStore type='network' index='5'> <format type='raw'/> <source protocol='iscsi' name='iqn.1992-01.com.example/0'> <host name='127.0.0.1' port='3260'/> <auth username='rhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> [root@hp-dl320eg8-05 bz1171569]# virsh destroy vm1; virsh start vm1 Domain vm1 destroyed Domain vm1 started 6. do a shallow blockcopy to see if the auth part still exists [root@hp-dl320eg8-05 bz1171569]# virsh blockcopy vm1 vdb /tmp/vdb --pivot --transient-job --shallow Successfully pivoted [root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' … <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/vdb' index='8'/> <backingStore type='network' index='2'> <format type='raw'/> <source protocol='iscsi' name='iqn.1992-01.com.example/0'> <host name='127.0.0.1' port='3260'/> <auth username='rhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> </backingStore> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </disk> [root@hp-dl320eg8-05 bz1171569]# virsh destroy vm1; virsh start vm1 Domain vm1 destroyed Domain vm1 started
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017