Bug 1588093

Summary: openstack_keys are not copied over to all MONs
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Giulio Fidente <gfidente>
Component: Ceph-AnsibleAssignee: Guillaume Abrioux <gabrioux>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: urgent Docs Contact: John Brier <jbrier>
Priority: urgent    
Version: 3.1CC: agunn, aschoen, ceph-eng-bugs, gabrioux, gfidente, gmeno, jbrier, nthomas, sankarshan, tserlin, vakulkar, yrabl
Target Milestone: rcKeywords: Triaged
Target Release: 3.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-ansible-3.1.0-0.1.rc8.el7cp Ubuntu: ceph-ansible_3.1.0~rc8-2redhat1 Doc Type: Bug Fix
Doc Text:
.The OpenStack keys are copied to all Ceph Monitors When {product} was configured with `run_once: true` and `inventory_hostname == groups.get(client_group_name) | first` it can cause a bug when the only node being run is not the first node in the group. In a deployment with a single client node the keyrings will not be created since the task can be skipped. With this release this situation no longer occurs and all the OpenStack keys are copied to the monitor nodes.
Story Points: ---
Clone Of: 1588092 Environment:
Last Closed: 2018-09-26 18:21:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1584264, 1588092    
Attachments:
Description Flags
ansible-playbook log
none
ansible-mistral-actionwCk7RF.tar none

Description Giulio Fidente 2018-06-06 15:41:38 UTC
openstack_keys are not copied over to all MONs

ceph-0.localdomain
total 20
drwxr-xr-x.   2 ceph ceph   37 Jun  6 13:14 .
drwxr-xr-x. 151 root root 8192 Jun  6 13:31 ..
-rw-r--r--.   1 root root  895 Jun  6 13:14 ceph.conf
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap
ceph-1.localdomain                                                                                                                                                                                            [0/461]
total 20
drwxr-xr-x.   2 ceph ceph   37 Jun  6 13:14 .
drwxr-xr-x. 151 root root 8192 Jun  6 13:31 ..
-rw-r--r--.   1 root root  895 Jun  6 13:14 ceph.conf
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap
ceph-2.localdomain
total 20
drwxr-xr-x.   2 ceph ceph   37 Jun  6 13:14 .
drwxr-xr-x. 151 root root 8192 Jun  6 13:31 ..
-rw-r--r--.   1 root root  895 Jun  6 13:14 ceph.conf
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap
compute-0.localdomain
total 32
drwxr-xr-x.   2 ceph ceph  143 Jun  6 13:15 .
drwxr-xr-x. 151 root root 8192 Jun  6 13:31 ..
-rw-------.   1 root root  284 Jun  6 13:15 ceph.client.manila.keyring
-rw-------.   1 root root  307 Jun  6 13:15 ceph.client.openstack.keyring
-rw-------.   1 root root  157 Jun  6 13:15 ceph.client.radosgw.keyring
-rw-r--r--.   1 root root 1002 Jun  6 13:15 ceph.conf
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap
controller-0.localdomain
total 40
drwxr-xr-x.   2 ceph ceph  205 Jun  6 13:12 .
drwxr-xr-x. 152 root root 8192 Jun  6 13:31 ..
-rw-------.   1 ceph ceph  159 Jun  6 13:12 ceph.client.admin.keyring
-rw-r--r--.   1 root root  811 Jun  6 13:12 ceph.conf
-rw-r--r--.   1 root root   67 Jun  6 13:12 ceph.mgr.controller-0.keyring
-rw-r--r--.   1 root root   67 Jun  6 13:12 ceph.mgr.controller-1.keyring
-rw-r--r--.   1 root root   67 Jun  6 13:12 ceph.mgr.controller-2.keyring
-rw-------.   1 ceph ceph  688 Jun  6 13:12 ceph.mon.keyring
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap
controller-1.localdomain
total 40
drwxr-xr-x.   2 ceph ceph  200 Jun  6 13:14 .
drwxr-xr-x. 152 root root 8192 Jun  6 13:31 ..
-rw-------.   1 root root  159 Jun  6 13:10 ceph.client.admin.keyring
-rw-------.   1 ceph ceph  276 Jun  6 13:14 ceph.client.manila.keyring
-rw-------.   1 ceph ceph  299 Jun  6 13:14 ceph.client.openstack.keyring
-rw-------.   1 ceph ceph  149 Jun  6 13:14 ceph.client.radosgw.keyring
-rw-r--r--.   1 root root  811 Jun  6 13:10 ceph.conf
-rw-------.   1 ceph ceph  688 Jun  6 13:11 ceph.mon.keyring
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap
controller-2.localdomain
total 40
drwxr-xr-x.   2 ceph ceph  200 Jun  6 13:14 .
drwxr-xr-x. 152 root root 8192 Jun  6 13:31 ..
-rw-------.   1 ceph ceph  159 Jun  6 13:11 ceph.client.admin.keyring
-rw-------.   1 ceph ceph  276 Jun  6 13:14 ceph.client.manila.keyring
-rw-------.   1 ceph ceph  299 Jun  6 13:14 ceph.client.openstack.keyring
-rw-------.   1 ceph ceph  149 Jun  6 13:14 ceph.client.radosgw.keyring
-rw-r--r--.   1 root root  811 Jun  6 13:11 ceph.conf
-rw-------.   1 ceph ceph  688 Jun  6 13:12 ceph.mon.keyring
-rw-r--r--.   1 root root   92 May  2 22:22 rbdmap

Comment 3 Giulio Fidente 2018-06-06 15:52:40 UTC
Created attachment 1448371 [details]
ansible-playbook log

Comment 4 Giulio Fidente 2018-06-06 15:56:35 UTC
Created attachment 1448373 [details]
ansible-mistral-actionwCk7RF.tar

Contains the inventory and the actual ansible-playbook cmdline.

Comment 5 Giulio Fidente 2018-06-08 05:38:15 UTC
Looks like we're better but still not completely done. The keyrings passed with "keys" are not provisioned on the ceph-client nodes (overcloud-novacompute-0).

Also, on the ceph-mon nodes the keyrings owner is forcibly set to ceph_uid, instead of root, not sure why; it might not be a blocking issue.

overcloud-cephstorage-0
total 16
drwxr-xr-x.   2  167  167   23 Jun  7 23:01 .
drwxr-xr-x. 119 root root 8192 Jun  7 22:59 ..
-rw-r--r--.   1 root root  906 Jun  7 23:01 ceph.conf
overcloud-novacompute-0
total 16
drwxr-xr-x.   2  167  167   23 Jun  7 23:08 .
drwxr-xr-x. 119 root root 8192 Jun  7 22:36 ..
-rw-r--r--.   1 root root 1013 Jun  7 23:08 ceph.conf
overcloud-controller-0
total 52
drwxr-xr-x.   2  167  167 4096 Jun  7 23:03 .
drwxr-xr-x. 121 root root 8192 Jun  7 22:37 ..
-rw-------.   1  167  167  159 Jun  7 22:55 ceph.client.admin.keyring
-rw-------.   1  167  167  276 Jun  7 23:03 ceph.client.manila.keyring
-rw-------.   1  167  167  299 Jun  7 23:03 ceph.client.openstack.keyring
-rw-------.   1  167  167  149 Jun  7 23:03 ceph.client.radosgw.keyring
-rw-r--r--.   1 root root 1826 Jun  7 22:55 ceph.conf
-rw-r--r--.   1 root root   77 Jun  7 22:57 ceph.mgr.overcloud-controller-0.keyring
-rw-r--r--.   1 root root   77 Jun  7 22:57 ceph.mgr.overcloud-controller-1.keyring
-rw-r--r--.   1 root root   77 Jun  7 22:56 ceph.mgr.overcloud-controller-2.keyring
-rw-------.   1  167  167  688 Jun  7 22:56 ceph.mon.keyring
overcloud-controller-1
total 36
drwxr-xr-x.   2  167  167  186 Jun  7 23:03 .
drwxr-xr-x. 121 root root 8192 Jun  7 22:37 ..
-rw-------.   1  167  167  159 Jun  7 22:52 ceph.client.admin.keyring
-rw-------.   1  167  167  276 Jun  7 23:03 ceph.client.manila.keyring
-rw-------.   1  167  167  299 Jun  7 23:02 ceph.client.openstack.keyring
-rw-------.   1  167  167  149 Jun  7 23:03 ceph.client.radosgw.keyring
-rw-r--r--.   1 root root 1826 Jun  7 22:52 ceph.conf
-rw-------.   1  167  167  688 Jun  7 22:53 ceph.mon.keyring
overcloud-controller-2
total 36
drwxr-xr-x.   2  167  167  186 Jun  7 23:02 .
drwxr-xr-x. 121 root root 8192 Jun  7 22:37 ..
-rw-------.   1 root root  159 Jun  7 22:49 ceph.client.admin.keyring
-rw-------.   1  167  167  284 Jun  7 23:08 ceph.client.manila.keyring
-rw-------.   1  167  167  307 Jun  7 23:08 ceph.client.openstack.keyring
-rw-------.   1  167  167  157 Jun  7 23:09 ceph.client.radosgw.keyring
-rw-r--r--.   1 root root 1826 Jun  7 22:49 ceph.conf
-rw-------.   1  167  167  688 Jun  7 22:49 ceph.mon.keyring

Comment 11 Yogev Rabl 2018-06-21 15:01:29 UTC
verified on ceph-ansible-3.1.0-0.1.rc9.el7cp.noarch

Comment 16 errata-xmlrpc 2018-09-26 18:21:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2819