Bug 1406442

Summary: libvirt no longer generates the cephx auth secret when hot plugging virtio-scsi devices
Product: Red Hat Enterprise Linux 7 Reporter: Mike Lowe <j.michael.lowe>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED ERRATA QA Contact: lijuan men <lmen>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.3CC: adhingra, cpatters, dmaley, dyuan, jbiao, jdenemar, jdillama, jferlan, jsuchane, knoel, mschuppe, rbalakri, skinjo, snagar, wrichter, xuzhang
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-3.0.0-1.el7 Doc Type: Bug Fix
Doc Text:
Cause: Missed code path to add secret for a virtio-scsi during alterations to support domain master secret for encrypted secret passing to qemu. Consequence: Cannot hot plug a virtio-scsi device using cephx authentication. Fix: Adjusted the algorithm to add the authentication object with the encrypted secret for qemu to parse. Result: Capability restored to authenticate hotplug'd virtio-scsi device.
Story Points: ---
Clone Of:
: 1411398 (view as bug list) Environment:
Last Closed: 2017-08-01 17:19:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1411398    

Description Mike Lowe 2016-12-20 14:26:38 UTC
Description of problem:
When attaching a ceph rbd block device with ceph authx enabled to a running vm using the virtio-scsi driver the cephx auth secret is no longer added causing the attachment to fail with "No secret with id 'scsi0-0-0-1-secret0'" or similar and if using openstack "libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-scsi0-0-0-1' could not be initialized" in the nova-compute.log.  This problem is discussed in this thread on the libvirt mailing list https://www.redhat.com/archives/libvir-list/2016-October/msg00396.html The libvirt version that shipped with 7.2 (1.2.17) did not have this problem.  

Version-Release number of selected component (if applicable):
2.0.0-10.el7_3.2

How reproducible:
Very, will likely affect every openstack deployment using ceph on rhel 7.3

Steps to Reproduce:
1.  Create a vm with virtio-scsi and ceph rbd root disk with cephx auth enabled
2.  use libvirt to attach a second virtio-scsi rbd device 
3.

Actual results:
Device attachment fails with the /var/log/libvirt/qemu log containing "No secret with id 'scsi0-0-0-1-secret0'" and if using openstack nova-compute log contains "libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-scsi0-0-0-1' could not be initialized"


Expected results:
device attachment to complete successfully without error

Additional info:

Comment 1 Jiri Denemark 2016-12-20 19:19:54 UTC
*** Bug 1406463 has been marked as a duplicate of this bug. ***

Comment 3 Mike Lowe 2016-12-20 19:24:53 UTC
I was able to build and install rpms from this srpm https://libvirt.org/sources/libvirt-2.5.0-1.fc24.src.rpm and it seems to resolve this issue as it includes this commit from upstream https://libvirt.org/git/?p=libvirt.git;a=commit;h=0701abcb3ba78ba27cf1f47e01b3d9607ad37b72

Comment 4 John Ferlan 2016-12-21 20:56:42 UTC
Referenced commit '0701abcb3' seems to resolve the issue. That commit is already upstream as of libvirt-2.4.0 which will be in the the next RHEL 7.4

POSTed patch upstream using this bz for 7.3.z:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-December/msg00231.html

Comment 7 lijuan men 2017-01-05 09:23:43 UTC
I will update our test status in libvirt:

1.I can reproduce the bug in  libvirt-2.0.0-10.el7_3.2

2.I have tested the patch in comment 4,not found new bugs.

Comment 11 lijuan men 2017-02-15 07:00:03 UTC
verify the bug

version:
libvirt-3.0.0-1.el7.x86_64
qemu-kvm-rhev-2.8.0-3.el7.x86_64

steps:

scenario 1:

1.prepare an auth enabled  ceph environment

2.start a guest with the following xml:
...
  <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/beifen.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
...
 <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
...

3.attach a device with the xml:
# cat ceph-disk.xml 
 <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='libvirt'>
        <secret type='ceph' usage='client.ceph'/>
      </auth>
      <source protocol='rbd' name='lmen/vol.img'>
        <host name='10.73.75.52'/>
      </source>
      <target dev='sdb' bus='scsi'/>
    </disk>

# virsh attach-device qq ceph-disk.xml 
Device attached successfully

[root@localhost ~]# virsh domblklist qq
Target     Source
------------------------------------------------
sda        /var/lib/libvirt/images/beifen.qcow2
sdb        lmen/vol.img

4.in the guest,
the second disk can be found and used.


scenario 2:
1.prepare an auth enabled  ceph environment

2.start a guest with the following xml:
...
  <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/beifen.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
...
*** <controller type='scsi' index='0'>***
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
...

3.attach a device with the xml:
# cat ceph-disk.xml 
 <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='libvirt'>
        <secret type='ceph' usage='client.ceph'/>
      </auth>
      <source protocol='rbd' name='lmen/vol.img'>
        <host name='10.73.75.52'/>
      </source>
      <target dev='sdb' bus='scsi'/>
    </disk>

# virsh attach-device qq ceph-disk.xml 
Device attached successfully

# virsh domblklist qq
Target     Source
------------------------------------------------
sda        /var/lib/libvirt/images/beifen.qcow2
sdb        lmen/vol.img


4.in the guest,
the second disk can be found and used.

Comment 17 errata-xmlrpc 2017-08-01 17:19:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 18 errata-xmlrpc 2017-08-01 23:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846