Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1700534

Summary: [OSP10] nova rbd auth fallback uses cinder user with libvirt secret
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: Matthew Booth <mbooth>
Severity: high Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: akaiser, alink, bbowen, dasmith, eglynn, jhakimra, jthomas, kchamart, lyarwood, sbauza, sgordon, smooney, ssigwald, vromanso
Target Milestone: z12Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-14.1.0-49.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-10 09:19:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Hill 2019-04-16 20:02:24 UTC 
Description of problem:
Backport commit [1] to fix rbd bug in nova when 2 different rbd clusters are used

[1] https://github.com/openstack/nova/commit/b89efa3ef611a1932df0c2d6e6f30315b5111a57#diff-21d379ea8d12438ade2a82cfb3db3dfb

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create another ceph cluster
2. Create a volume on that cluster
3. Try attaching it to a volume where nova.conf is configured to use first cluster

Actual results:
Fails to attach

Expected results:
Succeeds

Additional info:
Comment 12 David Hill 2019-04-25 16:20:53 UTC 
It looks like we might be hitting this bug now [1].

[1]  https://bugs.launchpad.net/nova/+bug/1809454
Comment 17 Brian 2019-05-07 16:34:54 UTC 
Note from Lee email:

The openstack-nova-14.1.0-49.el7ost build is approved as a hotfix for
the customer however my last conversation with dhill suggested there
were other issues in the env.
Comment 26 Matthew Booth 2019-06-19 14:47:08 UTC 
Validation steps:

1. Create a new 'openstack2' ceph user, and configure cinder to use the new credentials.
2. Create a volume and attach it to an instance.

Validation 1: Confirm using dumpxml on the compute host that we attached the volume using the openstack2 credentials.

3. Comment out rbd_secret_uuid in cinder.conf, but leave rbd_user set to openstack2.
4. Create and attach a new volume to an instance

Validation 2: Confirm using dumpxml on the compute host that attached the volume using both user and secret from nova.conf.
Comment 27 Matthew Booth 2019-06-20 12:55:47 UTC 
[root@controller-0 ~]# ceph auth add client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes'
added key for client.cinder

[root@controller-0 ~]# ceph auth get client.cinder
exported keyring for client.cinder
[client.cinder]
        key = AQDGZgtd4mF6KhAApy0u9fUeFDGVg412TaBcsg==
        caps mon = "allow r"
        caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes"

[root@controller-0 ~]# ceph auth get client.cinder > /etc/ceph/ceph.client.cinder.keyring
exported keyring for client.cinder

[root@compute-0 ~]# cat client.cinder.xml
<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
</secret>

[root@compute-0 ~]# virsh secret-define client.cinder.xml
Secret 7d625999-c295-4edf-8052-12d18dd99fe8 created

[root@compute-0 ~]# virsh secret-set-value 7d625999-c295-4edf-8052-12d18dd99fe8 AQDGZgtd4mF6KhAApy0u9fUeFDGVg412TaBcsg==
Secret value set

Repeat above on compute-1, except with explicit uuid in client.cinder.xml

[root@controller-0 ~]# chmod 640 /etc/ceph/ceph.client.cinder.keyring
[root@controller-0 ~]# chown ceph.ceph /etc/ceph/ceph.client.cinder.keyring
[root@controller-0 ~]# setfacl -m "user:glance:r--" /etc/ceph/ceph.client.cinder.keyring
[root@controller-0 ~]# setfacl -m "user:cinder:r--" /etc/ceph/ceph.client.cinder.keyring
[root@controller-0 ~]# setfacl -m "user:gnocchi:r--" /etc/ceph/ceph.client.cinder.keyring


Copy /etc/ceph/ceph.client.cinder.keyring to controller-1 and controller-2

Edit cinder.conf on controller-{0,1,2}, set:
rbd_secret_uuid=7d625999-c295-4edf-8052-12d18dd99fe8
rbd_user=cinder

Restart openstack-cinder-volume on controller-0 (it is not running on 1,2)

[stack@undercloud-0 ~]$ openstack volume create --size 1 credstest
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2019-06-20T12:35:26.886926           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 752cabb9-1ed6-4897-8dff-831d8e1cc6e9 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | credstest                            |
| properties          |                                      |
| replication_status  | disabled                             |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | None                                 |
| updated_at          | None                                 |
| user_id             | 837fb6d1122e410fb9c93a21002207c9     |
+---------------------+--------------------------------------+

[stack@undercloud-0 ~]$ openstack server create --image cirros-0.4.0-x86_64 --flavor m1.tiny --nic net-id=private testinstance1
+--------------------------------------+------------------------------------------------------------+
| Field                                | Value                                                      |
+--------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                     |
| OS-EXT-AZ:availability_zone          |                                                            |
| OS-EXT-SRV-ATTR:host                 | None                                                       |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | None                                                       |
| OS-EXT-SRV-ATTR:instance_name        |                                                            |
| OS-EXT-STS:power_state               | NOSTATE                                                    |
| OS-EXT-STS:task_state                | scheduling                                                 |
| OS-EXT-STS:vm_state                  | building                                                   |
| OS-SRV-USG:launched_at               | None                                                       |
| OS-SRV-USG:terminated_at             | None                                                       |
| accessIPv4                           |                                                            |
| accessIPv6                           |                                                            |
| addresses                            |                                                            |
| adminPass                            | 38cbjEUQWkge                                               |
| config_drive                         |                                                            |
| created                              | 2019-06-20T12:36:51Z                                       |
| flavor                               | m1.tiny (3f0877a2-03b8-4bd5-8e8b-4ec96421b0f3)             |
| hostId                               |                                                            |
| id                                   | d028e6dd-e291-4c7e-9103-ab87b79aba19                       |
| image                                | cirros-0.4.0-x86_64 (daeb330e-9d4d-4aee-8765-8fd6d3651ce8) |
| key_name                             | None                                                       |
| name                                 | testinstance1                                              |
| os-extended-volumes:volumes_attached | []                                                         |
| progress                             | 0                                                          |
| project_id                           | 9f8895b7cb374eae8531bd7f04276447                           |
| properties                           |                                                            |
| security_groups                      | [{u'name': u'default'}]                                    |
| status                               | BUILD                                                      |
| updated                              | 2019-06-20T12:36:51Z                                       |
| user_id                              | 837fb6d1122e410fb9c93a21002207c9                           |
+--------------------------------------+------------------------------------------------------------+

[stack@undercloud-0 ~]$ openstack server add volume d028e6dd-e291-4c7e-9103-ab87b79aba19 752cabb9-1ed6-4897-8dff-831d8e1cc6e9

[root@compute-1 ~]# virsh dumpxml instance-00000015
 ...
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='openstack'>
        <secret type='ceph' uuid='24d4f218-8ea0-11e9-a0ad-525400af756a'/>
      </auth>
      <source protocol='rbd' name='vms/d028e6dd-e291-4c7e-9103-ab87b79aba19_disk'>
        <host name='172.17.3.19' port='6789'/>
        <host name='172.17.3.21' port='6789'/>
        <host name='172.17.3.27' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='7d625999-c295-4edf-8052-12d18dd99fe8'/>
      </auth>
      <source protocol='rbd' name='volumes/volume-752cabb9-1ed6-4897-8dff-831d8e1cc6e9'>
        <host name='172.17.3.19' port='6789'/>
        <host name='172.17.3.21' port='6789'/>
        <host name='172.17.3.27' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <serial>752cabb9-1ed6-4897-8dff-831d8e1cc6e9</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
...

VERIFIED: volume is attached using cinder-provided username and secret

[stack@undercloud-0 ~]$ openstack server remove volume testinstance1 credstest

On controller-0, edit cinder.conf, comment out:
rbd_secret_uuid=7d625999-c295-4edf-8052-12d18dd99fe8

Restart openstack-cinder-volume

[stack@undercloud-0 ~]$ openstack volume create --size 1 credstest2
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2019-06-20T12:51:21.996058           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | bda266e0-5b31-4461-ba34-0fc8683bc50f |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | credstest2                           |
| properties          |                                      |
| replication_status  | disabled                             |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | None                                 |
| updated_at          | None                                 |
| user_id             | 837fb6d1122e410fb9c93a21002207c9     |
+---------------------+--------------------------------------+

[stack@undercloud-0 ~]$ openstack server add volume testinstance1 credstest2

# virsh dumpxml instance-00000015
...
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='openstack'>
        <secret type='ceph' uuid='24d4f218-8ea0-11e9-a0ad-525400af756a'/>
      </auth>
      <source protocol='rbd' name='vms/d028e6dd-e291-4c7e-9103-ab87b79aba19_disk'>
        <host name='172.17.3.19' port='6789'/>
        <host name='172.17.3.21' port='6789'/>
        <host name='172.17.3.27' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='openstack'>
        <secret type='ceph' uuid='24d4f218-8ea0-11e9-a0ad-525400af756a'/>
      </auth>
      <source protocol='rbd' name='volumes/volume-bda266e0-5b31-4461-ba34-0fc8683bc50f'>
        <host name='172.17.3.19' port='6789'/>
        <host name='172.17.3.21' port='6789'/>
        <host name='172.17.3.27' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <serial>bda266e0-5b31-4461-ba34-0fc8683bc50f</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
...

VERIFIED: volume is attached using nova provided user and secret
Comment 29 errata-xmlrpc 2019-07-10 09:19:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1715