Bug 1655182 - ceph.keyring file not found when upgrading jewel containerized to luminous
Summary: ceph.keyring file not found when upgrading jewel containerized to luminous
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Ceph-Ansible
Version: 3.2
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 3.2
Assignee: leseb
QA Contact: Coady LaCroix
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-30 21:30 UTC by Coady LaCroix
Modified: 2019-01-03 19:02 UTC (History)
9 users (show)

Fixed In Version: RHEL: ceph-ansible-3.2.0-0.1.rc7.el7cp Ubuntu: ceph-ansible_3.2.0~rc7-2redhat1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-03 19:02:28 UTC
Target Upstream Version:
vakulkar: automate_bug+


Attachments (Terms of Use)
container upgrade failure logs (457.59 KB, application/zip)
2018-11-30 21:30 UTC, Coady LaCroix
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0020 None None None 2019-01-03 19:02:36 UTC

Description Coady LaCroix 2018-11-30 21:30:31 UTC
Created attachment 1510274 [details]
container upgrade failure logs

Description of problem: 

During execution of the rolling update playbook to upgrade a containerized jewel installation to luminous(3.2), the playbook is failing with the following message:

failed: [ceph-clacroix-1543605572132-node1-mon] (item=[u'/var/lib/ceph/bootstrap-rbd/ceph.keyring', {'_ansible_parsed': True, u'stat': {u'exists': False}, '_ansible_it
em_result': True, '_ansible_no_log': False, '_ansible_delegated_vars': {'ansible_delegated_host': u'localhost', 'ansible_host': u'localhost'}, u'changed': False, 'failed': False, 'item': u'/var/lib/ceph/bootstrap
-rbd/ceph.keyring', u'invocation': {u'module_args': {u'checksum_algorithm': u'sha1', u'get_checksum': True, u'follow': False, u'path': u'/home/cephuser/fetch//efed8992-780b-4d69-8acb-b2619a7868f7//var/lib/ceph/bo
otstrap-rbd/ceph.keyring', u'get_md5': None, u'get_mime': True, u'get_attributes': True}}, 'failed_when_result': False, '_ansible_ignore_errors': None, '_ansible_item_label': u'/var/lib/ceph/bootstrap-rbd/ceph.ke
yring'}]) => {"changed": false, "item": ["/var/lib/ceph/bootstrap-rbd/ceph.keyring", {"_ansible_delegated_vars": {"ansible_delegated_host": "localhost", "ansible_host": "localhost"}, "_ansible_ignore_errors": nul
l, "_ansible_item_label": "/var/lib/ceph/bootstrap-rbd/ceph.keyring", "_ansible_item_result": true, "_ansible_no_log": false, "_ansible_parsed": true, "changed": false, "failed
": false, "failed_when_result": false, "invocation": {"module_args": {"checksum_algorithm": "sha1", "follow": false, "get_attributes": true, "get_checksum": true, "get_md5": null, "get_mime": true, "path": "/home
/cephuser/fetch//efed8992-780b-4d69-8acb-b2619a7868f7//var/lib/ceph/bootstrap-rbd/ceph.keyring"}}, "item": "/var/lib/ceph/bootstrap-rbd/ceph.keyring", "stat": {"exists": false}}], "msg": "file not found: /var/lib
/ceph/bootstrap-rbd/ceph.keyring"}


Version-Release number of selected component (if applicable):
ceph-ansible-3.2.0-0.1.rc5.el7cp.noarch
ansible-2.6.8-1.el7ae.noarch

How reproducible:
Every attempt to upgrade a containerized jewel installation to luminous 3.2.

Steps to Reproduce:
1. Install jewel containerized
2. Prepare cluster for upgrade (collocate mons and mgr, update ansible/ceph-ansible, configure insecure registry, etc.)
3. Run rolling update playbook

Actual results:
Failure (see above) during execution. Full logs attached.

Expected results:
Successful playbook execution and upgraded cluster.

Additional info:

Comment 7 Coady LaCroix 2018-12-03 18:50:14 UTC
I am still seeing the same error when upgrading using the fix version of ceph-ansible.

[cephuser@ceph-clacroix-1543856898046-node8-installer ~]$ rpm -qa | grep ansible
ceph-ansible-3.2.0-0.1.rc6.el7cp.noarch
ansible-2.6.8-1.el7ae.noarch

failed: [ceph-clacroix-1543856898046-node1-mon] (item=[u'/var/lib/ceph/bootstrap-rbd/ceph.keyring', {'_ansible_parsed': True, u'stat': {u'exists': False}, '_ansible_it
em_result': True, '_ansible_no_log': False, '_ansible_delegated_vars': {'ansible_delegated_host': u'localhost', 'ansible_host': u'localhost'}, u'changed': False, 'failed': False, 'item': u'/var/lib/ceph/bootstrap
-rbd/ceph.keyring', u'invocation': {u'module_args': {u'checksum_algorithm': u'sha1', u'get_checksum': True, u'follow': False, u'path': u'/home/cephuser/fetch//c9e61a0f-a975-48d7-a2ec-d1d0b1b55243//var/lib/ceph/bo
otstrap-rbd/ceph.keyring', u'get_md5': None, u'get_mime': True, u'get_attributes': True}}, 'failed_when_result': False, '_ansible_ignore_errors': None, '_ansible_item_label': u'/var/lib/ceph/bootstrap-rbd/ceph.ke
yring'}]) => {"changed": false, "item": ["/var/lib/ceph/bootstrap-rbd/ceph.keyring", {"_ansible_delegated_vars": {"ansible_delegated_host": "localhost", "ansible_host": "localhost"}, "_ansible_ignore_errors": nul
l, "_ansible_item_label": "/var/lib/ceph/bootstrap-rbd/ceph.keyring", "_ansible_item_result": true, "_ansible_no_log": false, "_ansible_parsed": true, "changed": false, "failed
": false, "failed_when_result": false, "invocation": {"module_args": {"checksum_algorithm": "sha1", "follow": false, "get_attributes": true, "get_checksum": true, "get_md5": null, "get_mime": true, "path": "/home
/cephuser/fetch//c9e61a0f-a975-48d7-a2ec-d1d0b1b55243//var/lib/ceph/bootstrap-rbd/ceph.keyring"}}, "item": "/var/lib/ceph/bootstrap-rbd/ceph.keyring", "stat": {"exists": false}}], "msg": "file not found: /var/lib
/ceph/bootstrap-rbd/ceph.keyring"}

Comment 8 seb 2018-12-03 21:30:11 UTC
Sorry Coady, the fix is actually in https://github.com/ceph/ceph-ansible/releases/tag/v3.2.0rc7

Comment 12 Coady LaCroix 2018-12-04 22:24:05 UTC
Verified using ceph-ansible-3.2.0-0.1.rc8.el7cp.noarch.

Comment 14 errata-xmlrpc 2019-01-03 19:02:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0020


Note You need to log in before you can comment on or make changes to this bug.