Bug 1544659

Could you please attach the live domain XML after you attach the disk in step 2?

Also a debug log would be nice.

the live domain XML is:

[root@lmen1 ~]# virsh dumpxml test1
...
  <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='rbd' name='lmen/lmen.img'>
          <host name='x.x.x.x'/>
          <auth username='libvirt'>
            <secret type='ceph'/>
          </auth>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
...

and will upload the debug log

Created attachment 1395733 [details]
libvirtd-bug1544659.log

So the problem is that libvirt started reporting the <auth> element for <backingStore> but the data which was detected from the RBD image did not contain all the necessary information so the subelement was invalid.

This is a problem with virsh detach-disk since it feeds back a XML snippet extracted from the domain XML rather than constructing a new XML file with the necessary data. Since the <auth> element is malformed parsing failed fails.

As the <backingStore> data is not necessary for detaching the disk the fix is to remove the detected authentication data in case of the backing store and also to fix virsh so that it does not report back the <backingStore> subelement.

Note that this is possible only with the manual configuration and the image would not contain such backing store by using libvirt APIs mostly as qemu has a bug which prevents parsing the backing store data by itself as reported by:

https://bugzilla.redhat.com/show_bug.cgi?id=1464837

commit f2f4e960601bd8cdbd8e082815caf6940feb599d
Author: Peter Krempa <pkrempa>
Date:   Wed Feb 14 15:17:51 2018 +0100

    virsh: Remove <backingStore> sub-element in virshFindDisk
    
    Previously we've removed the data only in virshUpdateDiskXML when
    changing the disk source for the CDROM since the backing store would be
    invalid. Move the code into a separate function and callit from
    virshFindDisk which is also used when detaching disk.
    
    The detaching code does not necessarily need to get the full backing
    chain since it will need to act on the one managed by libvirt anyways
    and this also takes care of problems when parts of the backing store
    were invalid due to buggy RBD detection code.
    
    Signed-off-by: Peter Krempa <pkrempa>

commit 28a36fe2ce2e88b45e6d960f113706e1c2732586
Author: Peter Krempa <pkrempa>
Date:   Wed Feb 14 14:12:23 2018 +0100

    util: storage: Remove detected authentication data for backing chains
    
    We can't really detect all the authentication data in a sane manner for
    disk backing chains. Since the old RBD parser parses it in some cases as
    the argv->XML convertor requires it, we can't just drop it.
    
    Instead clear any detected authentication data in the code paths related
    to disk backing chain lookup and fix the tests to cope with the change.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1544659
    
    Signed-off-by: Peter Krempa <pkrempa>

Because 'filename' is dropped since qemu-kvm-rhev-2.12.0-5.el7.x86_64, verified it on qemu-kvm-rhev-2.12.0-4.el7.x86_64 libvirt-4.5.0-6.virtcov.el7.x86_64

Preparation:
1. Prepare a running VM
2. Prepare a ceph server with cephx
3. Prepare a iscsi with chap authorization  

Scenarios 1: Attach and detach backing image with iscsi: filename:
1. Create backing file
# qemu-img create -f qcow2 -b 'json:{"file.driver":"iscsi","file.filename":"iscsi://redhat:redhat.XX.XX/iqn.2003-01.org.linux-iscsi.lab.x8664:sn.e51e19211db6/0"}' /var/lib/libvirt/images/iscsi.qcow2
qemu-img: warning: 'filename' option specified. This is an unsupported option, and may be deprecated in the future
Formatting '/var/lib/libvirt/images/iscsi.qcow2', fmt=qcow2 size=1073741824 backing_file=json:{"file.driver":"iscsi",,"file.filename":"iscsi://redhat:redhat.XX.XX/iqn.2003-01.org.linux-iscsi.lab.x8664:sn.e51e19211db6/0"} cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Attach the disk:
# virsh attach-disk nest7.6-1 /var/lib/libvirt/images/iscsi.qcow2 vdb --subdriver qcow2 
Disk attached successfully

# virsh dumpxml nest7.6-1|awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/iscsi.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.lab.x8664:sn.e51e19211db6/0'>
          <host name='XX.XX.XX.XX' port='3260'/>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>

3. Detach the disk
# virsh detach-disk nest7.6-1 vdb                
Disk detached successfully


Scenarios 2: Attach and detach backing image with rbd: filename:
1. Create backing file
# qemu-img create -f qcow2 -b 'json:{"file.driver":"rbd","file.filename":"rbd:hhan/xx:mon_host=XX.XX.XX.XX:id=admin:key=AQAwfVlbpH5/DBAAL+9LGWGTl9CkleSEMltiag=="}' /var/lib/libvirt/images/ceph.qcow2 
qemu-img: warning: 'filename' option specified. This is an unsupported option, and may be deprecated in the future
Formatting '/var/lib/libvirt/images/ceph.qcow2', fmt=qcow2 size=1073741824 backing_file=json:{"file.driver":"rbd",,"file.filename":"rbd:hhan/xx:mon_host=XX.XX.XX.XX:id=admin:key=AQAwfVlbpH5/DBAAL+9LGWGTl9CkleSEMltiag=="} cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Attach the disk:
# virsh attach-disk nest7.6-1 /var/lib/libvirt/images/ceph.qcow2 vdb --subdriver qcow2 
Disk attached successfully

# virsh dumpxml nest7.6-1|awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ceph.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='rbd' name='hhan/xx'>
          <host name='XX.XX.XX.XX'/>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>

3. Detach the disk
# virsh detach-disk nest7.6-1 vdb                
Disk detached successfully

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113