Bug 1640029

Created attachment 1494763 [details]
File contans rolling update log snippet

Description of problem:
When it was tried to upgrade cepph cluster from one build of 3.2 to latest 3.2 build, rollling updated failed in task [ceph-osd : use ceph-volume to create bluestore osds] saying  raise ValueError(\"No JSON object could be decoded

Version-Release number of selected component (if applicable):
ansible-2.6.5-1.el7ae.noarch
ceph-ansible-3.2.0-0.1.beta5.el7cp.noarch

How reproducible:
Always (5/5)

Steps to Reproduce:
1. Bring up 3.2 cluster with n-1 build with osd_scenario lvm and providing devices for lvm_volumes.
2. Upgrade ceph-ansible and run upgrade to update cluster.

Actual results:
failed: [xxxxx] (item={u'data': u'/dev/sdd'}) => {
    "changed": false, 
    "item": {
        "data": "/dev/sdd"
    }, 
    "module_stderr": "OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 8: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 72956\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\nTraceback (most recent call last):\n  File \"/tmp/ansible_dbAugM/ansible_module_ceph_volume.py\", line 625, in <module>\n    main()\n  File \"/tmp/ansible_dbAugM/ansible_module_ceph_volume.py\", line 621, in main\n    run_module()\n  File \"/tmp/ansible_dbAugM/ansible_module_ceph_volume.py\", line 521, in run_module\n    out_dict = json.loads(out)\n  File \"/usr/lib64/python2.7/json/__init__.py\", line 338, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/lib64/python2.7/json/decoder.py\", line 366, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/lib64/python2.7/json/decoder.py\", line 384, in raw_decode\n    raise ValueError(\"No JSON object could be decoded\")\nValueError: No JSON object could be decoded\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}


Expected results:
CLuster must get updated without any issues

Additional info:

$ cat group_vars/all.yml | egrep -v ^# | grep -v ^$
---
dummy:
fetch_directory: ~/ceph-ansible-keys
ceph_origin: distro
ceph_repository: rhcs
ceph_rhcs_version: 3
monitor_interface: eno1
public_network: 10.8.128.0/21


$  cat group_vars/osds.yml | egrep -v ^# | grep -v ^$
---
dummy:
osd_scenario: lvm
lvm_volumes:
   - data: /dev/sdb
   - data: /dev/sdc
   - data: /dev/sdd