Bug 1697860

Description of problem:

This is a hyperconverged deployment rhosp11 upgraded to 12 then 13(august), and now to 13 latest.

When trying to update to the latest version we got the following error:

2019-04-07 13:03:58,839 p=29384 u=mistral |  failed: [192.168.24.10] (item=[u'/etc/ceph/ceph.mon.keyring', {'_ansible_parsed': True, u'stat': {u'exists': False}, '_ansible_item_result': True, '_ansible_no_log': 
False, '_ansible_delegated_vars': {'ansible_delegated_host': u'localhost', 'ansible_host': u'localhost'}, u'changed': False, 'failed': False, 'item': u'/etc/ceph/ceph.mon.keyring', u'invocation': {u'module_args'
: {u'checksum_algorithm': u'sha1', u'get_checksum': True, u'follow': False, u'path': u'/tmp/file-mistral-action40pAWK/eb2bb192-b1c9-11e6-9205-525400330666//etc/ceph/ceph.mon.keyring', u'get_md5': None, u'get_mime': True, u'get_attributes': True}}, 'failed_when_result': False, '_ansible_ignore_errors': None, '_ansible_item_label': u'/etc/ceph/ceph.mon.keyring'}]) => {"changed": false, "item": ["/etc/ceph/ceph.mon.keyring", {"_ansible_delegated_vars": {"ansible_delegated_host": "localhost", "ansible_host": "localhost"}, "_ansible_ignore_errors": null, "_ansible_item_label": "/etc/ceph/ceph.mon.keyring", "_ansible_item_result": true, "_ansible_no_log": false, "_ansible_parsed": true, "changed": false, "failed": false, "failed_when_result": false, "invocation": {"module_args": {"checksum_algorithm": "sha1", "follow": false, "get_attributes": true, "get_checksum": true, "get_md5": null, "get_mime": true, "path": "/tmp/file-mistral-action40pAWK/eb2bb192-b1c9-11e6-9205-525400330666//etc/ceph/ceph.mon.keyring"}}, "item": "/etc/ceph/ceph.mon.keyring", "stat": {"exists": false}}], "msg": "file not found: /etc/ceph/ceph.mon.keyring"}
---



but if we look a few hours before the file was there:

2019-04-07 11:31:39,678 p=3577 u=mistral |  TASK [ceph-mon : stat for ceph config and keys] ********************************
2019-04-07 11:31:39,678 p=3577 u=mistral |  Sunday 07 April 2019  11:31:39 +0300 (0:00:00.237)       0:04:29.449 ********** 
2019-04-07 11:31:39,897 p=3577 u=mistral |  ok: [192.168.24.10 -> localhost] => (item=/etc/ceph/ceph.client.admin.keyring)
2019-04-07 11:31:40,072 p=3577 u=mistral |  ok: [192.168.24.10 -> localhost] => (item=/etc/ceph/ceph.mon.keyring)
2019-04-07 11:31:40,241 p=3577 u=mistral |  ok: [192.168.24.10 -> localhost] => (item=/var/lib/ceph/bootstrap-osd/ceph.keyring)
2019-04-07 11:31:40,423 p=3577 u=mistral |  ok: [192.168.24.10 -> localhost] => (item=/var/lib/ceph/bootstrap-rgw/ceph.keyring)
2019-04-07 11:31:40,640 p=3577 u=mistral |  ok: [192.168.24.10 -> localhost] => (item=/var/lib/ceph/bootstrap-mds/ceph.keyring)
2019-04-07 11:31:40,829 p=3577 u=mistral |  ok: [192.168.24.10 -> localhost] => (item=/var/lib/ceph/bootstrap-rbd/ceph.keyring)

Version-Release number of selected component (if applicable):
OSP13

How reproducible:
unsure

Steps to Reproduce:
1. This is a hyperconverged deployment rhosp11 upgraded to 12 then 13(august), and now to 13 latest.
2. check produre on comments below
3.

Actual results:
Update fail "msg": "file not found: /etc/ceph/ceph.mon.keyring"}

Expected results:
Update finish successfully 

Additional info: