Bug 1655350

Summary: containerized deployment using osd_scenario lvm fails
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Heðin <hej>
Component: Ceph-AnsibleAssignee: Sébastien Han <shan>
Status: CLOSED WONTFIX QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.1CC: adeza, aschoen, ceph-eng-bugs, gmeno, nthomas, sankarshan, seb
Target Milestone: rc   
Target Release: 3.*   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-03 22:03:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Heðin 2018-12-03 00:35:19 UTC
Description of problem:
rhcs deployment fails on osd deployment, when running ansible-playbook -i hosts.yml site-docker.yml after configuring ceph-ansible according to the RHCS 3 container guide dated 2018-11-06

Version-Release number of selected component (if applicable):
rhcs-3.1
ceph-anisble-3.1.10 from downstream repository.
rhel-7.6

How reproducible:
100%

Steps to Reproduce:
1. Spin up 3 rhel-7.6 minimal instances with at least one osd disk.
2. Install ceph-ansible.
3. Follow the above mentioned documentation.
4. Run the playbook: ansible-playbook -i hosts.yml site-docker.yml

Actual results:
This result is from the ceph-ansible-3.2 playbook.
2018-12-03 00:08:43,379 p=28549 u=cephadmin |  TASK [ceph-osd : use ceph-volume to create bluestore osds] *********************************************************
2018-12-03 00:08:43,379 p=28549 u=cephadmin |  Monday 03 December 2018  00:08:43 +0000 (0:00:00.086)       0:08:00.784 *******
2018-12-03 00:08:49,605 p=28549 u=cephadmin |  failed: [satv-rheltest01] (item={u'data_vg': u'ceph_vg_sdc', u'data': u'ceph_lv_sdc'}) => {"changed": false, "failed": true, "item": {"data": "ceph_lv_sdc", "data_vg": "ceph_vg_sdc"}, "msg": "Could not decode json output:  stdout: /dev/sda2\";\"\";\"CeZ2Pn-QOOv-72tw-HlVr-5MTc-JAKi-f8dxpr\";\"rhel\n stdout: /dev/sdb\";\"\";\"iEqNzZ-p5jz-GtAR-9pA4-2nf7-1kqR-vFYlxC\";\"ceph_vg_nvme_b\n stdout: /dev/sdc\";\"\";\"XcdhJo-h9VA-Hgqw-B7yH-EUSp-Hdpz-2un7fW\";\"ceph_vg_sdc\n stdout: /dev/sdd\";\"\";\"v64kkf-M6Xe-g2Ou-QeeK-2nMK-1PD8-SreewW\";\"ceph_vg_sdd\n stdout: /dev/sde\";\"\";\"wI6Ay7-aU5W-0uFl-KOzF-Nk0O-7J1r-aNM12M\";\"ceph_vg_sde\n stderr: Operation prohibited while global/metadata_read_only is set.\n stderr: Recovery of standalone physical volumes failed.\n stderr: Cannot process standalone physical volumes\n stderr: Operation prohibited while global/metadata_read_only is set.\n stderr: Recovery of standalone physical volumes failed.\n stderr: Cannot process standalone physical volumes\n stderr: Operation prohibited while global/metadata_read_only is set.\n stderr: Recovery of standalone physical volumes failed.\n stderr: Cannot process standalone physical volumes\n{}\n from the command ['docker', 'run', '--rm', '--privileged', '--net=host', '-v', '/run/lock/lvm:/run/lock/lvm:z', '-v', '/dev:/dev', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/run/lvm/lvmetad.socket:/run/lvm/lvmetad.socket', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph-volume', 'registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest', '--cluster', 'ceph', 'lvm', 'list', 'ceph_vg_sdc/ceph_lv_sdc', '--format=json']", "rc": 1}
2018-12-03 00:08:49,609 p=28549 u=cephadmin |  PLAY RECAP *********************************************************


Expected results:
deployed ceph cluster with 1 osd.

Additional info:
I have shared the playbook for 2 weeks: https://cloud.sa.fo/s/88QCLpabt6YsWni

the *3.1.10 files in groupvars are the ones I used in the ceph-anisble-3.1.10 playbook. (I trimmed the disk count down to 1 when i tried cpeh-anisble-3.2)

Comment 3 seb 2018-12-03 16:05:31 UTC
The error looks a bit familiar, thanks for sharing the logs.
Alfredo do you know if the patch that was fixing the output of the list in json format made it in 3.0?

Thanks.

Comment 4 Alfredo Deza 2018-12-03 21:51:53 UTC
Ceph 3.0 did not have the --readonly flag.

Comment 5 seb 2018-12-03 22:03:39 UTC
Thanks, I'm closing this then. If you're looking at using ceph-volume please wait for the 3.2 release. Thanks.