Bug 1715471

Summary: overcloud deployment fails with : conditional check 'ceph_health' failed
Product: Red Hat OpenStack Reporter: pkomarov
Component: ceph-ansibleAssignee: Francesco Pantano <fpantano>
Status: CLOSED NOTABUG QA Contact: pkomarov
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15.0 (Stein)CC: fpantano, gfidente, johfulto, michele, pkomarov, yrabl
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-05 17:47:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1594251    

Description pkomarov 2019-05-30 12:49:54 UTC
Description of problem:
overcloud deplpyment fails with : conditional check 'ceph_health' failed

detailed message: 
        "included: /usr/share/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml for controller-0, controller-1, controller-2",
        "TASK [ceph-mon : waiting for the monitor(s) to form the quorum...] *************",
        "task path: /usr/share/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml:2",
        "Wednesday 29 May 2019  10:43:41 +0000 (0:00:00.407)       0:02:29.558 ********* ",
        "fatal: [controller-0]: FAILED! => ",
        "  msg: |-",
        "    The conditional check '(ceph_health_raw.stdout != \"\") and (ceph_health_raw.stdout | default('{}') | from_json)['state'] in ['leader', 'peon']",
        "    ' failed. The error was: Expecting value: line 1 column 1 (char 0)",


Version-Release number of selected component (if applicable):
osp15

How reproducible:
100%

Steps to Reproduce:
rerun : 
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/sanity/job/DFG-pidone-sanity-15_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-sanity/

Additional info:
sos reports / ovrercloud node logs and stack home are at : 
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/sanity/job/DFG-pidone-sanity-15_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-sanity/9/artifact/

Comment 2 pkomarov 2019-06-05 11:40:09 UTC
adding ablocker since most of our downstream production testing is relying on ceph for HA testing

Comment 9 Francesco Pantano 2019-08-08 08:57:43 UTC
*** Bug 1733754 has been marked as a duplicate of this bug. ***