Bug 1663026 - [FFU] [Rhos-10->13] Ceph upgrade fails on ffu-converge step
Summary: [FFU] [Rhos-10->13] Ceph upgrade fails on ffu-converge step
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Ceph-Ansible
Version: 3.2
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z1
: 3.2
Assignee: Guillaume Abrioux
QA Contact: Yogev Rabl
URL:
Whiteboard:
: 1665664 (view as bug list)
Depends On:
Blocks: 1578730
TreeView+ depends on / blocked
 
Reported: 2019-01-02 18:20 UTC by Archit Modi
Modified: 2019-01-31 10:36 UTC (History)
19 users (show)

Fixed In Version: RHEL: ceph-ansible-3.2.3-1.el7cp Ubuntu: ceph-ansible_3.2.3-2redhat1
Doc Type: Bug Fix
Doc Text:
Cause: ceph-ansible has tasks which try to interpret some json output of the ceph CLI. In the playbook, those output are json representation in a string, therefore, we have to convert them in real json by using the `from_json` filter so we can manage with dict instead. When the output is empty, we need to provide a default value anyway to the `from_json` filter, so there was a default value that had an incorrect type. Consequence: `from_json` filter expect to receive a string while we were passing a json type causing the filter to throw an error. Fix: set the default value passed to `from_json` to the correct type (a string) Result: tasks containing filter `from_json` don't fail anymore.
Clone Of:
Environment:
Last Closed: 2019-01-31 10:36:36 UTC
Embargoed:

Attachments (Terms of Use)
ceph_ansible_playbook.log (430.16 KB, text/plain)
2019-01-03 09:27 UTC, Giulio Fidente 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph-ansible pull 3474 0 'None' closed [backport][stable-3.2] fix json data type 2020-12-21 09:38:55 UTC
Github ceph ceph-ansible pull 3500 0 'None' closed switch: do not fail on missing key 2020-12-21 09:38:22 UTC
Red Hat Bugzilla 1458024 0 high CLOSED [ceph-ansible] [ceph-container] : upgrade of containerized cluster fails 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2019:0223 0 None None None 2019-01-31 10:36:39 UTC

Comment 1 Giulio Fidente 2019-01-03 09:27:26 UTC 
Created attachment 1518101 [details]
ceph_ansible_playbook.log
Comment 2 Giulio Fidente 2019-01-03 09:28:46 UTC 
in job #46 looking at the undercloud logs it looks like this is happening with ceph-ansible 3.2

  Dec 19 14:28:57 Updated: ceph-ansible-3.2.0-1.el7cp.noarch

there was a recent change in ceph-ansible meant to fix this issue [1] and it seems the fix is included in the build tested by the job; probably the fix is not sufficient in the scenario where there is some output but it's not in json format (maybe spurious error messages)

Attaching the playbook logs and moving the bug to ceph/ceph-ansible

1. https://github.com/ceph/ceph-ansible/commit/2cea33f7fc4bf59eaa249ca26ba326105e392402
Comment 6 Ken Dreyer (Red Hat) 2019-01-07 21:54:43 UTC 
https://github.com/ceph/ceph-ansible/pull/3474 is now available in ceph-ansible v3.2.1 upstream.
Comment 13 Giulio Fidente 2019-01-11 16:31:21 UTC 
Sorry guys, moving back to ASSIGNED because there is another error affecting FFU visible in the logs.

The switch-from-non-containerized-to-containerized-ceph-daemons playbook hits the same issue fixed by BZ #1650572 ; maybe the condition added with https://github.com/ceph/ceph-ansible/pull/3389 needs to be applied for the "switch" playbook too?
Comment 14 Sébastien Han 2019-01-14 15:36:06 UTC 
Do you mind testing? https://github.com/ceph/ceph-ansible/pull/3500
Thanks.
Comment 16 Giulio Fidente 2019-01-20 21:14:28 UTC 
*** Bug 1665664 has been marked as a duplicate of this bug. ***
Comment 19 Lukas Bezdicka 2019-01-22 09:53:58 UTC 
Hi,
FFWD worked with https://github.com/ceph/ceph-ansible/pull/3512
If this patch is present in the last build than it's working.

Lukas
Comment 23 Eliad Cohen 2019-01-23 20:05:36 UTC 
Verified, converge step completes successfully.
Comment 34 errata-xmlrpc 2019-01-31 10:36:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0223
Note You need to log in before you can comment on or make changes to this bug.