Bug 1569290

Created attachment 1423819 [details]
ceph-install-workflow.log

Description of problem:
Ceph upgrade fails during OSP10 to OSP13 fast forward upgrade while running  TASK [ceph-client : chmod cephx key(s)]

Snippet from /var/log/mistral/ceph-install-workflow.log:

2018-04-18 20:01:41,306 p=26738 u=mistral |  TASK [ceph-client : chmod cephx key(s)] ****************************************
2018-04-18 20:01:41,307 p=26738 u=mistral |  task path: /usr/share/ceph-ansible/roles/ceph-client/tasks/create_users_keys.yml:99
2018-04-18 20:01:41,307 p=26738 u=mistral |  Wednesday 18 April 2018  20:01:41 -0400 (0:00:00.072)       0:11:19.185 ******* 
2018-04-18 20:01:41,573 p=26738 u=mistral |  changed: [192.168.24.8] => (item={'caps': {'mds': u"''", 'osd': u"'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx poo
l=images, allow rwx pool=metrics'", 'mon': u"'allow r'", 'mgr': u"'allow *'"}, 'mode': u'0600', 'key': u'AQDroddaAAAAABAA64fr7tW0dXZue2Pl9Wi8Qg==', 'name': u'client.openstack'}) => {"changed": true, "gid": 0, "group": "root", "item": {"ca
ps": {"mds": "''", "mgr": "'allow *'", "mon": "'allow r'", "osd": "'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics'"}, "key": 
"AQDroddaAAAAABAA64fr7tW0dXZue2Pl9Wi8Qg==", "mode": "0600", "name": "client.openstack"}, "mode": "0600", "owner": "root", "path": "/etc/ceph/ceph.client.openstack.keyring", "secontext": "system_u:object_r:container_file_t:s0", "size": 262
, "state": "file", "uid": 0}
2018-04-18 20:01:41,818 p=26738 u=mistral |  failed: [192.168.24.8] (item={'caps': {'mds': u"'allow *'", 'osd': u"'allow rw'", 'mon': u'\'allow r, allow command \\\\\\"auth del\\\\\\", allow command \\\\\\"auth caps\\\\\\", allow command 
\\\\\\"auth get\\\\\\", allow command \\\\\\"auth get-or-create\\\\\\"\'', 'mgr': u"'allow *'"}, 'name': u'client.manila', 'key': u'AQC+t9daAAAAABAA0zzoLz1NwZbBxrHNk14I4g==', 'mode': u'0600'}) => {"changed": false, "item": {"caps": {"mds"
: "'allow *'", "mgr": "'allow *'", "mon": "'allow r, allow command \\\\\\\"auth del\\\\\\\", allow command \\\\\\\"auth caps\\\\\\\", allow command \\\\\\\"auth get\\\\\\\", allow command \\\\\\\"auth get-or-create\\\\\\\"'", "osd": "'all
ow rw'"}, "key": "AQC+t9daAAAAABAA0zzoLz1NwZbBxrHNk14I4g==", "mode": "0600", "name": "client.manila"}, "msg": "file (/etc/ceph/ceph.client.manila.keyring) is absent, cannot continue", "path": "/etc/ceph/ceph.client.manila.keyring", "state
": "absent"}
2018-04-18 20:01:42,066 p=26738 u=mistral |  failed: [192.168.24.8] (item={'caps': {'mds': u"''", 'osd': u"'allow rwx'", 'mon': u"'allow rw'", 'mgr': u"'allow *'"}, 'mode': u'0600', 'key': u'AQDroddaAAAAABAAtrsj06ioGk1GRO2T4XUgOw==', 'nam
e': u'client.radosgw'}) => {"changed": false, "item": {"caps": {"mds": "''", "mgr": "'allow *'", "mon": "'allow rw'", "osd": "'allow rwx'"}, "key": "AQDroddaAAAAABAAtrsj06ioGk1GRO2T4XUgOw==", "mode": "0600", "name": "client.radosgw"}, "ms
g": "file (/etc/ceph/ceph.client.radosgw.keyring) is absent, cannot continue", "path": "/etc/ceph/ceph.client.radosgw.keyring", "state": "absent"}

/etc/ceph/ceph.client.radosgw.keyring and /etc/ceph/ceph.client.manila.keyring do not exist on the compute node(192.168.24.8)as these services were not enabled for the initial OSP10 deployment

Version-Release number of selected component (if applicable):
ceph-ansible-3.1.0-0.1.beta6.el7cp.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 with 3 controllers + 2 computes + 3 ceph osd nodes
2. Upgrade to OSP13 via the fast forward procedure

Actual results:
Ceph upgrade stage fails while running ceph-ansible.

Expected results:
Ceph upgrade succeeds fine.

Additional info:
Attaching /var/log/mistral/ceph-install-workflow.log and sosreports.