Bug 1544659 - can't detach disk with ceph auth backing file
Summary: can't detach disk with ceph auth backing file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Han Han
URL:
Whiteboard:
Depends On:
Blocks: 1558351
TreeView+ depends on / blocked
 
Reported: 2018-02-13 08:13 UTC by lijuan men
Modified: 2018-10-30 09:54 UTC (History)
8 users (show)

Fixed In Version: libvirt-4.3.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 09:52:39 UTC
Target Upstream Version:


Attachments (Terms of Use)
libvirtd-bug1544659.log (117.32 KB, text/plain)
2018-02-14 01:57 UTC, lijuan men
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 None None None 2018-10-30 09:54:42 UTC

Description lijuan men 2018-02-13 08:13:55 UTC
Description of problem:
can't detach disk with ceph auth backing file

Version-Release number of selected component (if applicable):
libvirt-3.9.0-12.el7.x86_64
qemu-kvm-rhev-2.10.0-20.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.create a disk with a ceph auth backing file
[root@lmen1 ~]# qemu-img create -f qcow2 -b 'json:{"file.driver":"rbd","file.filename":"rbd:lmen/lmen.img:mon_host=x.x.x.x:id=libvirt:key=AQCWJZ1Zh052AxAAHFa9E7QSh0vYLbv/Erc7uw=="}' /var/lib/libvirt/images/test.qcow2

[root@lmen1 ~]# qemu-img info /var/lib/libvirt/images/test.qcow2
image: /var/lib/libvirt/images/test.qcow2
file format: qcow2
virtual size: 100M (104857600 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"file.driver":"rbd","file.filename":"rbd:lmen/lmen.img:mon_host=x.x.x.x:id=libvirt:key=AQCWJZ1Zh052AxAAHFa9E7QSh0vYLbv/Erc7uw=="}
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false


2.attach the disk
[root@lmen1 ~]# virsh attach-disk test1 /var/lib/libvirt/images/test.qcow2 vdb --subdriver qcow2  
Disk attached successfully

[root@lmen1 ~]# virsh domblklist test1
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/V-GUI.qcow2
vdb        /var/lib/libvirt/images/test.qcow2

3.detach the disk
[root@lmen1 ~]# virsh detach-disk test1 vdb
error: Failed to detach disk
error: XML error: missing secret uuid or usage attribute

Actual results:
detach failed

Expected results:
detach successfully

Additional info:
can't reproduce in 7.4

Comment 3 Peter Krempa 2018-02-13 15:03:51 UTC
Could you please attach the live domain XML after you attach the disk in step 2?

Also a debug log would be nice.

Comment 4 lijuan men 2018-02-14 01:54:52 UTC
the live domain XML is:

[root@lmen1 ~]# virsh dumpxml test1
...
  <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='rbd' name='lmen/lmen.img'>
          <host name='x.x.x.x'/>
          <auth username='libvirt'>
            <secret type='ceph'/>
          </auth>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
...

and will upload the debug log

Comment 5 lijuan men 2018-02-14 01:57:59 UTC
Created attachment 1395733 [details]
libvirtd-bug1544659.log

Comment 6 Peter Krempa 2018-02-14 16:00:45 UTC
So the problem is that libvirt started reporting the <auth> element for <backingStore> but the data which was detected from the RBD image did not contain all the necessary information so the subelement was invalid.

This is a problem with virsh detach-disk since it feeds back a XML snippet extracted from the domain XML rather than constructing a new XML file with the necessary data. Since the <auth> element is malformed parsing failed fails.

As the <backingStore> data is not necessary for detaching the disk the fix is to remove the detected authentication data in case of the backing store and also to fix virsh so that it does not report back the <backingStore> subelement.

Note that this is possible only with the manual configuration and the image would not contain such backing store by using libvirt APIs mostly as qemu has a bug which prevents parsing the backing store data by itself as reported by:

https://bugzilla.redhat.com/show_bug.cgi?id=1464837

Comment 8 Peter Krempa 2018-02-16 12:32:33 UTC
commit f2f4e960601bd8cdbd8e082815caf6940feb599d
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Feb 14 15:17:51 2018 +0100

    virsh: Remove <backingStore> sub-element in virshFindDisk
    
    Previously we've removed the data only in virshUpdateDiskXML when
    changing the disk source for the CDROM since the backing store would be
    invalid. Move the code into a separate function and callit from
    virshFindDisk which is also used when detaching disk.
    
    The detaching code does not necessarily need to get the full backing
    chain since it will need to act on the one managed by libvirt anyways
    and this also takes care of problems when parts of the backing store
    were invalid due to buggy RBD detection code.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>

commit 28a36fe2ce2e88b45e6d960f113706e1c2732586
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Feb 14 14:12:23 2018 +0100

    util: storage: Remove detected authentication data for backing chains
    
    We can't really detect all the authentication data in a sane manner for
    disk backing chains. Since the old RBD parser parses it in some cases as
    the argv->XML convertor requires it, we can't just drop it.
    
    Instead clear any detected authentication data in the code paths related
    to disk backing chain lookup and fix the tests to cope with the change.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1544659
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>

Comment 10 Han Han 2018-08-03 09:19:50 UTC
Because 'filename' is dropped since qemu-kvm-rhev-2.12.0-5.el7.x86_64, verified it on qemu-kvm-rhev-2.12.0-4.el7.x86_64 libvirt-4.5.0-6.virtcov.el7.x86_64

Preparation:
1. Prepare a running VM
2. Prepare a ceph server with cephx
3. Prepare a iscsi with chap authorization  

Scenarios 1: Attach and detach backing image with iscsi: filename:
1. Create backing file
# qemu-img create -f qcow2 -b 'json:{"file.driver":"iscsi","file.filename":"iscsi://redhat:redhat@XX.XX.XX.XX/iqn.2003-01.org.linux-iscsi.lab.x8664:sn.e51e19211db6/0"}' /var/lib/libvirt/images/iscsi.qcow2
qemu-img: warning: 'filename' option specified. This is an unsupported option, and may be deprecated in the future
Formatting '/var/lib/libvirt/images/iscsi.qcow2', fmt=qcow2 size=1073741824 backing_file=json:{"file.driver":"iscsi",,"file.filename":"iscsi://redhat:redhat@XX.XX.XX.XX/iqn.2003-01.org.linux-iscsi.lab.x8664:sn.e51e19211db6/0"} cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Attach the disk:
# virsh attach-disk nest7.6-1 /var/lib/libvirt/images/iscsi.qcow2 vdb --subdriver qcow2 
Disk attached successfully

# virsh dumpxml nest7.6-1|awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/iscsi.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='iscsi' name='iqn.2003-01.org.linux-iscsi.lab.x8664:sn.e51e19211db6/0'>
          <host name='XX.XX.XX.XX' port='3260'/>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>

3. Detach the disk
# virsh detach-disk nest7.6-1 vdb                
Disk detached successfully


Scenarios 2: Attach and detach backing image with rbd: filename:
1. Create backing file
# qemu-img create -f qcow2 -b 'json:{"file.driver":"rbd","file.filename":"rbd:hhan/xx:mon_host=XX.XX.XX.XX:id=admin:key=AQAwfVlbpH5/DBAAL+9LGWGTl9CkleSEMltiag=="}' /var/lib/libvirt/images/ceph.qcow2 
qemu-img: warning: 'filename' option specified. This is an unsupported option, and may be deprecated in the future
Formatting '/var/lib/libvirt/images/ceph.qcow2', fmt=qcow2 size=1073741824 backing_file=json:{"file.driver":"rbd",,"file.filename":"rbd:hhan/xx:mon_host=XX.XX.XX.XX:id=admin:key=AQAwfVlbpH5/DBAAL+9LGWGTl9CkleSEMltiag=="} cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Attach the disk:
# virsh attach-disk nest7.6-1 /var/lib/libvirt/images/ceph.qcow2 vdb --subdriver qcow2 
Disk attached successfully

# virsh dumpxml nest7.6-1|awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ceph.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='rbd' name='hhan/xx'>
          <host name='XX.XX.XX.XX'/>
        </source>
      </backingStore>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>

3. Detach the disk
# virsh detach-disk nest7.6-1 vdb                
Disk detached successfully

Comment 12 errata-xmlrpc 2018-10-30 09:52:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113


Note You need to log in before you can comment on or make changes to this bug.