Bug 1511811 - [Ceph-Ansible]: Upgrading cluster from 2.4 to 3.0 fails for missing mgr keyring.
Summary: [Ceph-Ansible]: Upgrading cluster from 2.4 to 3.0 fails for missing mgr keyring.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Ceph-Ansible
Version: 3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 3.0
Assignee: Ken Dreyer (Red Hat)
QA Contact: Ramakrishnan Periyasamy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 08:34 UTC by Ramakrishnan Periyasamy
Modified: 2017-12-05 23:50 UTC (History)
9 users (show)

Fixed In Version: RHEL: ceph-ansible-3.0.10-2.el7cp Ubuntu: ceph-ansible_3.0.10-2redhat1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-05 23:50:29 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph-ansible pull 2163 0 None closed rpm: require ansible 2.4.1.0 2020-09-30 23:35:36 UTC
Red Hat Product Errata RHBA-2017:3387 0 normal SHIPPED_LIVE Red Hat Ceph Storage 3.0 bug fix and enhancement update 2017-12-06 03:03:45 UTC

Description Ramakrishnan Periyasamy 2017-11-10 08:34:36 UTC
Description of problem:

Tried upgrading cluster from 2.4 release to 3.0, ceph-ansible failed for missing mgr keyring. mgr and its keyring not there in 2.4 but ceph-ansible is expecting it to be there which will be good if we upgrade cluster from 3.0 to 3.x release but not from 2.4 to 3.0

Following is the ceph-ansible failure when upgrading the cluster with mgr.
=========================================

TASK [ceph-mgr : copy mgr key] ******************************************************
task path: /usr/share/ceph-ansible/roles/ceph-mgr/tasks/pre_requisite.yml:33
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/copy.py", line 457, in run
    source = self._find_needle('files', source)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 983, in _find_needle
    return self._loader.path_dwim_relative_stack(path_stack, dirname, needle)
  File "/usr/lib/python2.7/site-packages/ansible/parsing/dataloader.py", line 386, in path_dwim_relative_stack
    raise AnsibleFileNotFound(file_name=source, paths=[to_text(p) for p in search])
AnsibleFileNotFound: Could not find or access '~/ceph-ansible-keys/2e342296-a72e-47ab-a200-8cb49b79a2c6/etc/ceph/test.mgr.magna086.keyring'

fatal: [magna086]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "msg": "Could not find or access '~/ceph-ansible-keys/2e342296-a72e-47ab-a200-8cb49b79a2c6/etc/ceph/test.mgr.magna086.keyring'"
}
================================

When tried to upgrade the cluster without mgr observed below fatal message 
================================
TASK [fail if no mgr host is present in the inventory] **********************************************************************************************************************
task path: /usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:38
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "msg": "Please add a mgr host to your inventory."
}
================================

Note: Observed this bug while verifying bz:1494238


Version-Release number of selected component (if applicable):
ceph-ansible version:
[ubuntu@host006 ceph-ansible]$ sudo rpm -qa | grep ansible
ceph-ansible-3.0.9-1.el7cp.noarch
ansible-2.4.0.0-5.el7.noarch

How reproducible:
3/3

Steps to Reproduce:
1. Configure 2.4 cluster 
2. Upgrade 2.4 cluster to 3.0 using ceph-ansible, follow the upgrade procedure as per doc
3. Added mgr to the /etc/ansible/hosts before starting rolling_upgrade

Actual results:
mgr creation fails with expectation of keying exists in cluster.

Expected results:
before proceeding mgr check for mgr service if service available the proceed the normal path but if mgr is going to get created new then it should create the mgr keyring and proceed.

Additional info:
Observed this bug while verifying bz:1494238

Comment 5 Ramakrishnan Periyasamy 2017-11-10 09:54:06 UTC
After upgrading ansible with "yum update ansible" playbook proceeded with creating mgr without any issues.

upgraded ansible version:
[ubuntu@host006 ceph-ansible]$ sudo rpm -qa | grep ansible
ceph-ansible-3.0.9-1.el7cp.noarch
ansible-2.4.1.0-1.el7ae.noarch

When we upgrade ceph-ansible it should update ansible as dependency, is this a build/packaging issue ? or we need to update doc ?

Comment 6 Ken Dreyer (Red Hat) 2017-11-10 16:41:59 UTC
We can make the ceph-ansible package require ansible >= 2.4.1.0

Comment 12 Ramakrishnan Periyasamy 2017-11-13 15:45:50 UTC
Moving this bug to verified state. 

Verified in ceph version 12.2.1-39.el7cp (22e26be5a4920c95c43f647b31349484f663e4b9) luminous (stable) build.

Comment 16 errata-xmlrpc 2017-12-05 23:50:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387


Note You need to log in before you can comment on or make changes to this bug.