Bug 1171569 - <auth>..</auth> element is gone after block jobs
Summary: <auth>..</auth> element is gone after block jobs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: rc
: 8.1
Assignee: Peter Krempa
QA Contact: yisun
URL:
Whiteboard:
: 1467271 1467287 (view as bug list)
Depends On: 760547
Blocks: 1467287 1473046
TreeView+ depends on / blocked
 
Reported: 2014-12-08 06:11 UTC by Yang Yang
Modified: 2020-05-05 09:45 UTC (History)
8 users (show)

Fixed In Version: libvirt-5.10.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-05 09:43:16 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2017 0 None None None 2020-05-05 09:45:46 UTC

Description Yang Yang 2014-12-08 06:11:26 UTC
Description of problem:
Start a vm using a rbd disk specifying the <auth../> element providing the authentication credentials. The <auth../> element disappears after creating external disk snapshot. The issue is also reproduced when using iscsi as backing file. It will cause committing to base image fails.

Version-Release number of selected component (if applicable):
libvirt-1.2.8-10.el7.x86_64
qemu-kvm-rhev-2.1.2-14.el7.x86_64
kernel-3.10.0-212.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. start vm with rbd disk specifying <auth>
<disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='libvirt'>
        <secret type='ceph' usage='client.libvirt secret'/>
      </auth>
      <source protocol='rbd' name='libvirt-pool/rbd1.img'>
        <config file='/etc/ceph/ceph.conf'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 10    rbd                            running

2. create external disk snapshot
# virsh snapshot-create-as rbd s1 --disk-only --diskspec vda,file=/tmp/rbd.s1
Domain snapshot s1 created
[root@rhel7_test yy]# virsh snapshot-list rbd
 Name                 Creation Time             State
------------------------------------------------------------
 s1                   2014-12-08 13:30:30 +0800 disk-snapshot

3.check the domain xml
# virsh dumpxml rbd | grep disk -a6
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/tmp/rbd.s1'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='rbd' name='libvirt-pool/rbd1.img'>
          <config file='/etc/ceph/ceph.conf'/>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>

4. check image backing chain
# qemu-img info /tmp/rbd.s1 --backing-chain
image: /tmp/rbd.s1
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 580K
cluster_size: 65536
backing file: rbd:libvirt-pool/rbd1.img:id=libvirt:key=AQCI335UkAgXHhAA90By4w5NR6zb63LbbM0MGg==:auth_supported=cephx\;none:conf=/etc/ceph/ceph.conf
backing file format: raw
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: rbd:libvirt-pool/rbd1.img:id=libvirt:key=AQCI335UkAgXHhAA90By4w5NR6zb63LbbM0MGg==:auth_supported=cephx\;none:conf=/etc/ceph/ceph.conf
file format: raw
virtual size: 8.0G (8589934592 bytes)
disk size: unavailable
cluster_size: 4194304

Actual results:
In step 3, <auth../> element disappeared in <backingStore> element.

Expected results:
Keep <auth../> element in <backingStore> element

Additional info:
The issue is also reproduced when using iscsi as backing file. The issue will cause that committing to base image fails

Comment 2 Peter Krempa 2015-01-05 12:21:25 UTC
Libvirt really needs to track the complete backing chain internally in such cases as otherwise the backing chain is re-loaded from the disk state and thus the <auth> information are lost (unless qemu would record them in the backing file name).

Comment 7 Peter Krempa 2017-07-03 10:53:55 UTC
Note that this happens for every image in the backing chain which originally had the auth element. While it won't happen currently for snapshots during the lifetime of the VM it will happen if libvirtd is restarted or if the VM is restarted.

It will be fixed with full backing chain tracking in libvirt.

Comment 8 Peter Krempa 2017-07-03 10:55:15 UTC
*** Bug 1467287 has been marked as a duplicate of this bug. ***

Comment 9 Peter Krempa 2017-07-03 10:55:17 UTC
*** Bug 1467271 has been marked as a duplicate of this bug. ***

Comment 11 Meina Li 2018-01-15 03:12:21 UTC
The latest progress for <auth> element:
Starting with libvirt 3.9.0 the auth element is preferred to be a sub-element of the source element. The element is still read and managed as a disk sub-element. 

Test problems in backing chain( external snapshot) when the auth element is a sub-element of the source element:
1.  <auth> sub-element is missing after restart guest.
2.  <auth> sub-element is missing after blockcommit/blockpull/blockcopy.

Test version:
libvirt-3.9.0-7.el7.x86_64

One of the test scenario:
1. Prepare a guest with iscsi auth disk.
# virsh dumpxml rhel7 | grep disk -a6
... <disk type='network' device='disk'>
     <driver name='qemu' type='qcow2'/>
     <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'>
       <host name='**IP**' port='3260'/>
       <auth username='redhat'>
         <secret type='iscsi' usage='libvirtiscsi'/>
       </auth>
     </source>
     <target dev='vda' bus='virtio'/>
     <alias name='virtio-disk0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
   </disk>...

2. Create external disk and check xml.
# for i in 1 2 3 4;do virsh snapshot-create-as rhel7 s$i --disk-only --diskspec vda,file=/tmp/rhel7.s$i;done
# virsh dumpxml rhel7 | grep disk -a12 
... <disk type='file' device='disk'>
     <driver name='qemu' type='qcow2'/>
     <source file='/tmp/rhel7.s4'/>
     <backingStore type='file' index='1'>
       <format type='qcow2'/>
       <source file='/tmp/rhel7.s3'/>
       <backingStore type='file' index='2'>
         <format type='qcow2'/>
         <source file='/tmp/rhel7.s2'/>
         <backingStore type='file' index='3'>
          <format type='qcow2'/>
           <source file='/tmp/rhel7.s1'/>
           <backingStore type='network' index='4'>
            <format type='qcow2'/>
             <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'>
               <host name='**IP**' port='3260'/>
               <auth username='redhat'>
                 <secret type='iscsi' usage='libvirtiscsi'/>
               </auth>
             </source>
           </backingStore>...

3.  Do blockcommit from middle to middle.
# virsh blockcommit rhel7 vda --top vda[1] --base vda[3] --wait --verbose --pivot
# virsh dumpxml rhel7 | grep disk -a12
... <disk type='file' device='disk'>
     <driver name='qemu' type='qcow2'/>
     <source file='/tmp/rhel7.s4'/>
         <backingStore type='file' index='1'>
          <format type='qcow2'/>
           <source file='/tmp/rhel7.s1'/>
           <backingStore type='network' index='2'>
            <format type='qcow2'/>
             <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'>
               <host name='**IP**' port='3260'/>
             </source>                                                                 --<auth> element is missing
           </backingStore>...

Actual results:
As above step 3.

Expected results:
# virsh dumpxml rhel7 | grep disk -a12
… <backingStore type='network' index='2'>
            <format type='qcow2'/>
             <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.9cba196611e6/0'>
               <host name='**IP**' port='3260'/>
              <auth username='redhat'>
                 <secret type='iscsi' usage='libvirtiscsi'/>
               </auth>
             </source>
           </backingStore>...

Comment 12 Jaroslav Suchanek 2019-04-24 12:26:48 UTC
This bug is going to be addressed in next major release.

Comment 13 Peter Krempa 2019-11-27 08:37:49 UTC
The following commits prepared for proper backing chain manipulation which does not drop the <auth> element and other configuration:

commit 1bf3808207fdac0a90a91399d1d1f19e73d35676
Author: Peter Krempa <pkrempa>
Date:   Mon Jul 22 13:39:24 2019 +0200

    qemu: Add -blockdev support for block pull job
    
    Introduce the handler for finalizing a block pull job which will allow
    to use it with blockdev.
    
    This patch also contains some additional machinery which is required to
    store all the relevant job data in the status XML which will also be
    reused with other block job types.

commit 3f93884a4d047a012b968c62b94ea07dadd1759b
Author: Peter Krempa <pkrempa>
Date:   Mon Jul 22 13:39:24 2019 +0200

    qemu: Add -blockdev support for block commit job
    
    Introduce the handler for finalizing a block commit and active bloc
    commit job which will allow to use it with blockdev.

commit ce7229a3b0d28479e0f123efce3fa73617889a50
Author: Peter Krempa <pkrempa>
Date:   Mon Jul 22 13:59:01 2019 +0200

    qemu: Add blockdev support for the block copy job
    
    Implement job handling for the block copy job (drive/blockdev-mirror)
    when using -blockdev. In contrast to the previously implemented
    blockjobs the block copy job introduces new images to the running qemu
    instance, thus requires a bit more handling.
    
    When copying to new images the code now makes use of blockdev-create to
    format the images explicitly rather than depending on automagic qemu
    behaviour.

The blockdev feature was enabled since:

commit c6a9e54ce3252196f1fc6aa9e57537a659646d18
Author: Peter Krempa <pkrempa>
Date:   Mon Jan 7 11:45:19 2019 +0100

    qemu: enable blockdev support

    Now that all pieces are in place (hopefully) let's enable -blockdev.

    We base the capability on presence of the fix for 'auto-read-only' on
    files so that blockdev works properly, mandate that qemu supports
    explicit SCSI id strings to avoid ABI regression and that the fix for
    'savevm' is present so that internal snapshots work.

v5.9.0-390-gc6a9e54ce3

and requires upstream qemu-4.2 or appropriate downstream.

Comment 15 yisun 2020-02-24 08:28:43 UTC
Test result: PASS

[root@hp-dl320eg8-05 bz1171569]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-4"
qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64
libvirt-6.0.0-5.module+el8.2.0+5765+64816f89.x86_64

1. having a vm with authenticated iscsi device
[root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
…
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth username='rhat'>
        <secret type='iscsi' usage='libvirtiscsi'/>
      </auth>
      <source protocol='iscsi' name='iqn.1992-01.com.example/0' index='1'>
        <host name='127.0.0.1' port='3260'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@hp-dl320eg8-05 bz1171569]# virsh domstate vm1
running

[root@hp-dl320eg8-05 bz1171569]# virsh domblklist vm1
 Target   Source
---------------------------------------------
…
 vdb      iqn.1992-01.com.example/0


2. create 1 snapshot and see if the authentication part works fine in vm's xml
[root@hp-dl320eg8-05 bz1171569]# virsh snapshot-create-as vm1 snap1 --disk-only --diskspec vdb,file=/tmp/vdb.s1
Domain snapshot snap1 created

[root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/vdb.s1' index='4'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='iscsi' name='iqn.1992-01.com.example/0'>
          <host name='127.0.0.1' port='3260'/>
          <auth username='rhat'>
            <secret type='iscsi' usage='libvirtiscsi'/>
          </auth>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@hp-dl320eg8-05 bz1171569]# virsh destroy vm1; virsh start vm1
Domain vm1 destroyed
Domain vm1 started

[root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/vdb.s1' index='1'/>
      <backingStore type='network' index='2'>
        <format type='raw'/>
        <source protocol='iscsi' name='iqn.1992-01.com.example/0'>
          <host name='127.0.0.1' port='3260'/>
          <auth username='rhat'>
            <secret type='iscsi' usage='libvirtiscsi'/>
          </auth>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

3. create snapshot 2 to 4
[root@hp-dl320eg8-05 bz1171569]# for i in {2..4}; do virsh snapshot-create-as vm1 snap$i --disk-only --diskspec vdb,file=/tmp/vdb.s$i; done
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created


[root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/vdb.s4' index='10'/>
      <backingStore type='file' index='8'>
        <format type='qcow2'/>
        <source file='/tmp/vdb.s3'/>
        <backingStore type='file' index='6'>
          <format type='qcow2'/>
          <source file='/tmp/vdb.s2'/>
          <backingStore type='file' index='1'>
            <format type='qcow2'/>
            <source file='/tmp/vdb.s1'/>
            <backingStore type='network' index='2'>
              <format type='raw'/>
              <source protocol='iscsi' name='iqn.1992-01.com.example/0'>
                <host name='127.0.0.1' port='3260'/>
                <auth username='rhat'>
                  <secret type='iscsi' usage='libvirtiscsi'/>
                </auth>
              </source>
            </backingStore>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

5. do a blockcommit/blockpull to see if the auth part still exists
[root@hp-dl320eg8-05 bz1171569]# virsh blockcommit vm1 vdb --top vdb[1] --base vdb[4] --wait --verbose --pivot
Block commit: [100 %]
Successfully pivoted

[root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
 …
   <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/vdb.s1' index='4'/>
      <backingStore type='network' index='5'>
        <format type='raw'/>
        <source protocol='iscsi' name='iqn.1992-01.com.example/0'>
          <host name='127.0.0.1' port='3260'/>
          <auth username='rhat'>
            <secret type='iscsi' usage='libvirtiscsi'/>
          </auth>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@hp-dl320eg8-05 bz1171569]# virsh destroy vm1; virsh start vm1
Domain vm1 destroyed
Domain vm1 started

6. do a shallow blockcopy to see if the auth part still exists
[root@hp-dl320eg8-05 bz1171569]# virsh blockcopy vm1 vdb /tmp/vdb --pivot --transient-job --shallow
Successfully pivoted

[root@hp-dl320eg8-05 bz1171569]# virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
…
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/vdb' index='8'/>
      <backingStore type='network' index='2'>
        <format type='raw'/>
        <source protocol='iscsi' name='iqn.1992-01.com.example/0'>
          <host name='127.0.0.1' port='3260'/>
          <auth username='rhat'>
            <secret type='iscsi' usage='libvirtiscsi'/>
          </auth>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[root@hp-dl320eg8-05 bz1171569]# virsh destroy vm1; virsh start vm1
Domain vm1 destroyed
Domain vm1 started

Comment 17 errata-xmlrpc 2020-05-05 09:43:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.