Bug 1650306

@Jhon the message: 

>  Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

Means that ceph-volume looked at the devices and determined that the strategy would be to place data in the spinning drives and block.db on the NVMe devices.

However, *filtering* happened, which I am assuming it removed/filtered-out the NVMe devices, leaving just the spinning ones. This change is detected and it correctly refuses to continue since the end result would be to ignore the NVMe devices and consuming the spinning drives fully.

There are a few reasons why the NVMe devices were excluded, but could you run the report command to get more information? For example:

> ceph-volume --cluster ceph lvm batch --bluestore --yes /dev/sdc /dev/sdd /dev/nvme0n1 /dev/sdq /dev/sdr /dev/nvme1n1 --report


And just in case the json report is richer:

> ceph-volume --cluster ceph lvm batch --bluestore --yes /dev/sdc /dev/sdd /dev/nvme0n1 /dev/sdq /dev/sdr /dev/nvme1n1 --report --format=json

Created attachment 1506227 [details]
text 'ceph lvm batch' report

(In reply to John Harrigan from comment #4)
> Created attachment 1506227 [details]
> text 'ceph lvm batch' report

Both reports failed to provide any information
$ cat report.txt 
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

$ cat report.json
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

John, that sounds like something we need to improve, thanks for catching that. Can you add the /var/log/ceph/ceph-volume.log file to this ticket? The reasons for filtering the devices should all be logged there

the ceph-ansible run leaves this cluster, all HDD based.
How should I specify the configuration in osds.yml to get the intended
configuration of ceph-ansible deploys bluestore cluster with two HDDs paired to each of the two NVMe devices

# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME              STATUS REWEIGHT PRI-AFF 
 -1       87.30176 root default                                   
-17        7.27515     host c05-h29-6048r                         
  7   hdd  1.81879         osd.7              up  1.00000 1.00000 
 19   hdd  1.81879         osd.19             up  1.00000 1.00000 
 31   hdd  1.81879         osd.31             up  1.00000 1.00000 
 43   hdd  1.81879         osd.43             up  1.00000 1.00000 
-21        7.27515     host c06-h01-6048r                         
  8   hdd  1.81879         osd.8              up  1.00000 1.00000 
 20   hdd  1.81879         osd.20             up  1.00000 1.00000 
 32   hdd  1.81879         osd.32             up  1.00000 1.00000 
 45   hdd  1.81879         osd.45             up  1.00000 1.00000 
-19        7.27515     host c06-h05-6048r                         
  9   hdd  1.81879         osd.9              up  1.00000 1.00000 
 21   hdd  1.81879         osd.21             up  1.00000 1.00000 
 33   hdd  1.81879         osd.33             up  1.00000 1.00000 
 46   hdd  1.81879         osd.46             up  1.00000 1.00000 
-25        7.27515     host c06-h09-6048r                         
 10   hdd  1.81879         osd.10             up  1.00000 1.00000 
 22   hdd  1.81879         osd.22             up  1.00000 1.00000 
 34   hdd  1.81879         osd.34             up  1.00000 1.00000 
 44   hdd  1.81879         osd.44             up  1.00000 1.00000 
-23        7.27515     host c06-h13-6048r                         
 11   hdd  1.81879         osd.11             up  1.00000 1.00000 
 23   hdd  1.81879         osd.23             up  1.00000 1.00000 
 35   hdd  1.81879         osd.35             up  1.00000 1.00000 
 47   hdd  1.81879         osd.47             up  1.00000 1.00000 
 -3        7.27515     host c07-h01-6048r                         
  0   hdd  1.81879         osd.0              up  1.00000 1.00000 
 12   hdd  1.81879         osd.12             up  1.00000 1.00000 
 26   hdd  1.81879         osd.26             up  1.00000 1.00000 
 37   hdd  1.81879         osd.37             up  1.00000 1.00000 
 -7        7.27515     host c07-h05-6048r                         
  3   hdd  1.81879         osd.3              up  1.00000 1.00000 
 13   hdd  1.81879         osd.13             up  1.00000 1.00000 
 25   hdd  1.81879         osd.25             up  1.00000 1.00000 
 38   hdd  1.81879         osd.38             up  1.00000 1.00000 
 -5        7.27515     host c07-h09-6048r                         
  1   hdd  1.81879         osd.1              up  1.00000 1.00000 
 14   hdd  1.81879         osd.14             up  1.00000 1.00000 
 24   hdd  1.81879         osd.24             up  1.00000 1.00000 
 36   hdd  1.81879         osd.36             up  1.00000 1.00000 
-15        7.27515     host c07-h13-6048r                         
  2   hdd  1.81879         osd.2              up  1.00000 1.00000 
 15   hdd  1.81879         osd.15             up  1.00000 1.00000 
 30   hdd  1.81879         osd.30             up  1.00000 1.00000 
 40   hdd  1.81879         osd.40             up  1.00000 1.00000 
-11        7.27515     host c07-h17-6048r                         
  4   hdd  1.81879         osd.4              up  1.00000 1.00000 
 16   hdd  1.81879         osd.16             up  1.00000 1.00000 
 27   hdd  1.81879         osd.27             up  1.00000 1.00000 
 41   hdd  1.81879         osd.41             up  1.00000 1.00000 
 -9        7.27515     host c07-h21-6048r                         
  5   hdd  1.81879         osd.5              up  1.00000 1.00000 
 17   hdd  1.81879         osd.17             up  1.00000 1.00000 
 28   hdd  1.81879         osd.28             up  1.00000 1.00000 
 42   hdd  1.81879         osd.42             up  1.00000 1.00000 
-13        7.27515     host c07-h25-6048r                         
  6   hdd  1.81879         osd.6              up  1.00000 1.00000 
 18   hdd  1.81879         osd.18             up  1.00000 1.00000 
 29   hdd  1.81879         osd.29             up  1.00000 1.00000 
 39   hdd  1.81879         osd.39             up  1.00000 1.00000

Created attachment 1506268 [details]
ceph volume log

What configuration are you using so that cehp-ansible deploys only to the HDDs?

I extended the /root/wipefs_6048r.sh script to include this
  dd if=/dev/zero of=/dev/$device bs=1M count=1

Then ran this sequence of cmds and got the same result:
  # ansible-playbook purge-cluster.yml 
  # ansible osds -m script -a "/root/wipefs_6048.sh"
  # ansible-playbook site.yml 2>&1 | tee -a Deploy2nvme.Nov15zap

TASK [ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created] ***
Friday 16 November 2018  00:02:14 +0000 (0:00:02.430)       0:34:58.752 ******* 
fatal: [c07-h01-6048r]: FAILED! => {"changed": true, "cmd": ["ceph-volume", "--cluster", "ceph", "lvm", "batch", "--bluestore", "--yes", "/dev/sdc", "/dev/sdd", "/dev/nvme0n1", "/dev/sdq", "/dev/sdr", "/dev/nvme1n1", "--report", "--format=json"], "msg": "non-zero return code", "rc": 1, "stderr": "", "stderr_lines": [], "stdout": "--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering", "stdout_lines": ["--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering"]}

so no change there.

Next up I will update to the latest ceph-ansible since this commit
  purge-cluster: zap devices used with the lvm scenario
  https://github.com/ceph/ceph-ansible/commit
        /9747f3dbd5a2eada543a6f61e482e005b6660016
is in ceph-ansible-3.2.0-0.1.rc2.el7cp.noarch.rpm.

I upgraded to ceph-ansible-rc2, ran the purge-cluster and deploy.
Unfortunately I had the same result, an early
exit from ceph-ansible with this message

TASK [ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created] ***
Friday 16 November 2018  15:54:33 +0000 (0:00:01.615)       0:34:51.107 ******* 
fatal: [c07-h01-6048r]: FAILED! => {"changed": true, "cmd": ["ceph-volume", "--cluster", "ceph", "lvm", "batch", "--bluestore", "--yes", "/dev/sdc", "/dev/sdd", "/dev/nvme0n1", "/dev/sdq", "/dev/sdr", "/dev/nvme1n1", "--report", "--format=json"], "msg": "non-zero return code", "rc": 1, "stderr": "", "stderr_lines": [], "stdout": "--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering", "stdout_lines": ["--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering"]}

I have attached the logs from the purge and deploy, both performed using rc2 version of ceph-ansible.

Here are the cmds I used to perform this test
---------------------------------------------
# cd /root; wget http://download.eng.bos.redhat.com/composes/auto/ceph-3.2-rhel-7/RHCEPH-3.2-RHEL-7-20181115.ci.1/compose/Tools/x86_64/os/Packages/ceph-ansible-3.2.0-0.1.rc2.el7cp.noarch.rpm
# rpm -Uvh ceph-ansible-3.2.0-0.1.rc2.el7cp.noarch.rpm
# yum list ceph-ansible
  ceph-ansible.noarch                3.2.0-0.1.rc2.el7cp
Purge and redeploy using ceph-ansible.rc2
# ssh c07-h01-6048r ceph -s        ← 48 OSDs
# ansible-playbook purge-cluster.yml 2>&1 | tee -a PurgeRC2.Nov16
# ansible-playbook site.yml 2>&1 | tee -a DeployRC2.Nov16

Created attachment 1506461 [details]
purge runlog using ceph-ansibleRC2

Created attachment 1506462 [details]
deploy runlog using ceph-ansibleRC2

I decided to try this same sequence of cmds but this time only specify ONE NVMe device. 

Sure enough, the deploy work like a charm.

# yum list ceph-ansible
  ceph-ansible.noarch                3.2.0-0.1.rc2.el7cp
# ansible-playbook purge-cluster.yml
# ansible-playbook site.yml 2>&1 | tee -a DeployOneNVMe.Nov16
No Errors
TASK [show ceph status for cluster ceph] ***************************************
Friday 16 November 2018  19:37:58 +0000 (0:00:00.542)       0:55:45.555 ******* 
ok: [c05-h33-6018r -> c05-h33-6018r] => {
    "msg": [
        "  cluster:", 
        "    id:     2f9e9148-125e-4783-ab30-6fcd121aca01", 
        "    health: HEALTH_OK", 
        " ", 
        "  services:", 
        "    mon: 3 daemons, quorum c05-h33-6018r,c06-h29-6018r,c07-h29-6018r", 
        "    mgr: c07-h30-6018r(active)", 
        "    osd: 144 osds: 144 up, 144 in", 
        "    rgw: 12 daemons active", 
        " ", 
        "  data:", 
        "    pools:   4 pools, 32 pgs", 
        "    objects: 209 objects, 12.1KiB", 
        "    usage:   147GiB used, 262TiB / 262TiB avail", 
        "    pgs:     32 active+clean", 
        " "
    ]
}

INSTALLER STATUS ***************************************************************
Install Ceph Monitor        : Complete (0:05:10)
Install Ceph Manager        : Complete (0:03:38)
Install Ceph OSD            : Complete (0:16:18)
Install Ceph RGW            : Complete (0:05:06)
Install Ceph Client         : Complete (0:18:34)

---------------------------------
The osds.yml file looks like this
---------------------------------
osd_objectstore: bluestore
# use 'ceph-volume lvm batch' mode
osd_scenario: lvm
devices:
  - /dev/sdc
  - /dev/sdd
  - /dev/sde
  - /dev/sdf
  - /dev/sdg
  - /dev/sdh
  - /dev/sdi
  - /dev/sdj
  - /dev/sdk
  - /dev/sdl
  - /dev/sdm
  - /dev/sdn
  - /dev/nvme0n1

Created attachment 1506531 [details]
Success using ceph-ansibleRC2 w/one NVMe

When using two NVMe devices, in the case that fails, you've mentioned that the steps are:

* purge         -> ansible-playbook purge-cluster.yml 
* wipefs script -> ansible osds -m script -a "/root/wipefs_6048.sh"
* deploy        -> ansible-playbook site.yml

The way the ansible implementation works is by checking if the input of devices will change the deployment strategy after filtering. For example:

input: /dev/sda /dev/sdb /dev/nvme0n1

And /dev/nvme0n1 gets filtered out, then this changes the stragey from "mixed devices" (spinning and solid), to "single type" (only one type of device). That would cause an immediate halt to the playbook.

If devices are getting filtered out that don't change the strategy then *there should not be an error*. For example:

input: /dev/sda /dev/sdb /dev/nvme0n1 /dev/nvme1n1

And /dev/nvme0n1 gets filtered, the strategy doesn't change, because there is still a mixed type group of devices.

So something is out of whack here when you say you try with 2 NVMe devices and things don't work, and then try with 1 NVMe device and it works.

One thing I would try is call the report after purging+wipefs, to see if devices are getting filtered out:

> ceph-volume lvm batch --report --format=json  /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/nvme0n1

If that runs *before* deployment, and *after* purging+wipefs, it should work.

The example call I used was with one nvme device, but I would try with two as well and toy around with what the reporting says. The JSON reporting is more verbose for us (and useful for this BZ) but you might find it easier to use the normal (pretty) reporting

How do I run ceph-volume after purging?
The cmd will no longer be installed on the OSD nodes.

Am I understanding the syntax (and ordering) in osds.yml correctly?
For example, using this:
  osd_objectstore: bluestore
  # use 'ceph-volume lvm batch' mode
  osd_scenario: lvm
  devices:
    - /dev/sdc
    - /dev/sdd
    - /dev/nvme0n1
    - /dev/sdq
    - /dev/sdr
    - /dev/nvme1n1

I would expect that the first two HDDs would hold 'block' portions of OSDs
and /dev/nvme0n1 would house their 'db'. The next two HDDs (sdq and sdr) would
be another set of 'block' OSDs and their 'db' would be on /dev/nvme1n1.
Should I be specifying devices in a different order?
Again the cfg I am trying to get to is to have 4 HDDs in use as OSDs and place
two of their 'db' on one nvme and the other two 'db' on the other nvme.

thanks

Another thing that would help the output is to increase the verbosity with -vv
and export:

> ANSIBLE_STDOUT_CALLBACK=debug

John, the ordering doesn't matter, the batch sub-command will create one large VG for both NVMe devices.

You cannot split them like that in one go. If you really want to you could try a first run with half of it and the rest with the other half.

"batch" mode is not meant to be super flexible, it allows you to do without the LV creation, which takes a great deal of code to implement right so the constraints are there to allow a robust execution.

But with LVM it is possible to state that you want a particular LV created on a particular PV within the VG.  So it is possible to round-robin the RocksDB and journaling LVs across the available SSD PVs, even if all SSDs are in the same VG.  For example, if you have an array of PVs, you can index into the SSD PV list using the LV's array index modulo N where N is the number of SSD PVs.

This is really critical functionality for ceph-ansible.  For example, one Ceph site that I know of has 60 HDDs/host, they will definitely have to balance their Bluestore RocksDB and journal space evenly across the SSD devices, and this will not be possible with this release of ceph-ansible if I understand John H correctly.  Failure to implement this results in very substandard performance where the SSD becomes the bottleneck instead of the HDD, particularly for sequential writes or large random writes.

@Ben, you are right that LVM does allow all of these configurations. The `lvm batch` sub-command was *not* meant to allow further customization from what it already offers.

That is why we have the ability to receive pre-made LVs to produce OSDs: so that ceph-volume doesn't need to accommodate every configuration possible with LVM.

In the end it is a decision between an easy deployment and a highly configurable one, we can't do both in `lvm batch`

Having said that, if we really want to push forward with more configurable scenarios and LVM, then that should probably go into ceph-ansible, that way the LVs could just be consumed in whatever way they were produced.

The purge-cluster.yml output indicates that it *did* run the osd lvm zap
commands on the OSD nodes. The complete output from the purge run is in
attachment "purge runlog using ceph-ansible RC2"

Here is an excerpt

TASK [zap and destroy osds created by ceph-volume with devices] ****************
Friday 16 November 2018  15:10:22 +0000 (0:00:01.692)       0:01:23.648 ******* 
changed: [c07-h09-6048r] => (item=/dev/sdc)
changed: [c07-h05-6048r] => (item=/dev/sdc)
changed: [c07-h01-6048r] => (item=/dev/sdc)
changed: [c07-h17-6048r] => (item=/dev/sdc)
changed: [c07-h13-6048r] => (item=/dev/sdc)
changed: [c07-h09-6048r] => (item=/dev/sdd)
changed: [c07-h25-6048r] => (item=/dev/sdc)
changed: [c07-h05-6048r] => (item=/dev/sdd)
changed: [c07-h21-6048r] => (item=/dev/sdc)
changed: [c07-h01-6048r] => (item=/dev/sdd)
changed: [c07-h17-6048r] => (item=/dev/sdd)
changed: [c06-h01-6048r] => (item=/dev/sdc)
changed: [c07-h13-6048r] => (item=/dev/sdd)
changed: [c05-h29-6048r] => (item=/dev/sdc)
changed: [c06-h05-6048r] => (item=/dev/sdc)
changed: [c07-h25-6048r] => (item=/dev/sdd)
changed: [c07-h21-6048r] => (item=/dev/sdd)
changed: [c07-h09-6048r] => (item=/dev/nvme0n1)
changed: [c06-h09-6048r] => (item=/dev/sdc)
changed: [c07-h05-6048r] => (item=/dev/nvme0n1)
changed: [c07-h01-6048r] => (item=/dev/nvme0n1)
changed: [c06-h01-6048r] => (item=/dev/sdd)
changed: [c06-h13-6048r] => (item=/dev/sdc)
changed: [c05-h29-6048r] => (item=/dev/sdd)
changed: [c07-h17-6048r] => (item=/dev/nvme0n1)
changed: [c06-h05-6048r] => (item=/dev/sdd)
changed: [c07-h13-6048r] => (item=/dev/nvme0n1)
changed: [c07-h09-6048r] => (item=/dev/sdq)
changed: [c07-h05-6048r] => (item=/dev/sdq)
changed: [c06-h09-6048r] => (item=/dev/sdd)
changed: [c07-h21-6048r] => (item=/dev/nvme0n1)
changed: [c07-h01-6048r] => (item=/dev/sdq)
changed: [c06-h13-6048r] => (item=/dev/sdd)
changed: [c07-h17-6048r] => (item=/dev/sdq)
changed: [c07-h13-6048r] => (item=/dev/sdq)
changed: [c07-h25-6048r] => (item=/dev/nvme0n1)
changed: [c07-h09-6048r] => (item=/dev/sdr)
changed: [c07-h05-6048r] => (item=/dev/sdr)
changed: [c06-h01-6048r] => (item=/dev/nvme0n1)
changed: [c05-h29-6048r] => (item=/dev/nvme0n1)
changed: [c07-h21-6048r] => (item=/dev/sdq)
changed: [c07-h01-6048r] => (item=/dev/sdr)
changed: [c06-h05-6048r] => (item=/dev/nvme0n1)
changed: [c07-h17-6048r] => (item=/dev/sdr)
changed: [c07-h09-6048r] => (item=/dev/nvme1n1)
changed: [c07-h13-6048r] => (item=/dev/sdr)
changed: [c06-h09-6048r] => (item=/dev/nvme0n1)
changed: [c07-h05-6048r] => (item=/dev/nvme1n1)
changed: [c05-h29-6048r] => (item=/dev/sdq)
changed: [c06-h01-6048r] => (item=/dev/sdq)
changed: [c07-h01-6048r] => (item=/dev/nvme1n1)
changed: [c07-h21-6048r] => (item=/dev/sdr)
changed: [c07-h25-6048r] => (item=/dev/sdq)
changed: [c06-h05-6048r] => (item=/dev/sdq)
changed: [c06-h13-6048r] => (item=/dev/nvme0n1)
changed: [c07-h17-6048r] => (item=/dev/nvme1n1)
changed: [c07-h13-6048r] => (item=/dev/nvme1n1)
changed: [c06-h09-6048r] => (item=/dev/sdq)
changed: [c07-h21-6048r] => (item=/dev/nvme1n1)
changed: [c05-h29-6048r] => (item=/dev/sdr)
changed: [c06-h01-6048r] => (item=/dev/sdr)
changed: [c07-h25-6048r] => (item=/dev/sdr)
changed: [c06-h05-6048r] => (item=/dev/sdr)
changed: [c06-h13-6048r] => (item=/dev/sdq)
changed: [c06-h09-6048r] => (item=/dev/sdr)
changed: [c05-h29-6048r] => (item=/dev/nvme1n1)
changed: [c06-h01-6048r] => (item=/dev/nvme1n1)
changed: [c07-h25-6048r] => (item=/dev/nvme1n1)
changed: [c06-h05-6048r] => (item=/dev/nvme1n1)
changed: [c06-h13-6048r] => (item=/dev/sdr)
changed: [c06-h09-6048r] => (item=/dev/nvme1n1)
changed: [c06-h13-6048r] => (item=/dev/nvme1n1)

I remain concerned that purge is not fully cleaning up to allow a clean
redeploy when using ceph-volume lvm batch mode.

- John

Could you ensure you are running with:

> ANSIBLE_STDOUT_CALLBACK=debug

And with the -vv flag in ansible?

John,

This PR should handle the issue we found with subsequent deploys not being idempotent because of the strategy change.

https://github.com/ceph/ceph-ansible/pull/3348

maybe blocker -- investigating now

we can't reproduce this by deploying on a new system with several NVMe devices, there is a chance the usage of purge+redeploy might hit this, and there is a fix already for the idempotency issue that was found. 

We don't think this is a blocker

Did some additional testing today using RC4. I am still not able to deploy
using ceph lvm batch

Tested deploy/purge cycle first with ceph-disk (successful) and then ceph-volume (failed).
I will attach the full ceph-ansible logfile "DeployLVMbatchRC4.Nov28"


RHCS 3.2 CEPH-DISK : ceph-ansible.noarch         3.2.0-0.1.rc2.el7cp
==================
1) Purged existing RHCS 3.2 cluster (ceph-disk non-collocated, bluestore)

2) Deployed RHCS 3.2 cluster (ceph-disk non-collocated, bluestore)
SUCCESS - no failed tasks and running cluster with expected number OSDs and RGWs

3) Purged existing RHCS 3.2 cluster (ceph-disk non-collocated, bluestore)

===================================> CEPH-VOLUME <======================
4) Installed latest ceph-ansible (RC4)
# yum update ceph-ansible
  ceph-ansible.noarch         3.2.0-0.1.rc4.el7cp

5) Deployed RHCS 3.2 cluster (ceph-volume lvm batch, bluestore)
# cat osds.yml
  #--------------------------------------------------------------------
  osd_objectstore: bluestore
  # use 'ceph-volume lvm batch' mode
  osd_scenario: lvm
  devices:
    - /dev/sdc
    - /dev/sdd
    - /dev/nvme0n1
    - /dev/sdq
    - /dev/sdr
    - /dev/nvme1n1
# export ANSIBLE_STDOUT_CALLBACK=debug
# ansible-playbook -vv site.yml 2>&1 | tee -a DeployLVMbatchRC4.Nov28 
<...SNIP...>
TASK [ceph-config : generate ceph configuration file: ceph.conf] ***************
task path: /usr/share/ceph-ansible/roles/ceph-config/tasks/main.yml:77
Wednesday 28 November 2018  18:20:33 +0000 (0:00:01.020)       0:15:19.542 **** 
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: 	[line 16]: u' # non_hci_safety_factor is the safety factor for dedicated nodes\n'
fatal: [c07-h01-6048r]: FAILED! => {}

MSG:

Unexpected failure during module execution.

PLAY RECAP *********************************************************************
c03-h15-r620               : ok=22   changed=1    unreachable=0    failed=0   
c03-h17-r620               : ok=22   changed=1    unreachable=0    failed=0   
c03-h19-r620               : ok=22   changed=1    unreachable=0    failed=0   
c03-h21-r620               : ok=22   changed=1    unreachable=0    failed=0   
c04-h33-6018r              : ok=22   changed=1    unreachable=0    failed=0   
c05-h33-6018r              : ok=93   changed=12   unreachable=0    failed=0   
c06-h29-6018r              : ok=83   changed=10   unreachable=0    failed=0   
c07-h01-6048r              : ok=60   changed=6    unreachable=0    failed=1   
c07-h05-6048r              : ok=57   changed=6    unreachable=0    failed=1   
c07-h09-6048r              : ok=57   changed=6    unreachable=0    failed=1   
c07-h13-6048r              : ok=57   changed=6    unreachable=0    failed=1   
c07-h17-6048r              : ok=57   changed=6    unreachable=0    failed=1   
c07-h21-6048r              : ok=57   changed=6    unreachable=0    failed=1   
c07-h25-6048r              : ok=57   changed=6    unreachable=0    failed=1   
c07-h29-6018r              : ok=85   changed=13   unreachable=0    failed=0   
c07-h30-6018r              : ok=83   changed=11   unreachable=0    failed=0   


INSTALLER STATUS ***************************************************************
Install Ceph Monitor        : Complete (0:04:40)
Install Ceph Manager        : Complete (0:03:29)
Install Ceph OSD            : In Progress (0:03:59)
	This phase can be restarted by running: roles/ceph-osd/tasks/main.yml

Wednesday 28 November 2018  18:20:43 +0000 (0:00:09.903)       0:15:29.446 **** 
=============================================================================== 
<...SNIP...>

Created attachment 1509615 [details]
ansible -vv using ceph-ansibleRC4 - FAILED

> An exception occurred during task execution. To see the full traceback, use 
> -vvv. The error was: 	[line 16]: u' # non_hci_safety_factor is the safety factor for dedicated nodes\n'


This looks like it is unrelated to ceph-volume? The ceph configuration is failing to be generated.

What is "non_hci_safety_factor" ? Maybe Sebastien can help here

I think they were trying to conditionalize how much memory was reserved for Bluestore OSDs (i.e. OSD caching layer), depending on whether or not the Ceph OSD host had to also run other things (example: hyperconverged OpenStack).  Since you are running dedicated OSD hosts, you need the non_hci_safety_factor parameter.

@John, I suspect that Sebastien might want to see the traceback that is hidden from the log output at the verbose levels used. Can you re-run the ansible-playbook command with the -vvv flag and paste the full task failure?

Hopefully that will have the traceback, and enough information to see what is going on there.

reran using -vvv flag and here is the full task failure, repeated for
each of the OSD nodes...

MSG:

Unexpected failure during module execution.
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 139, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 584, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/share/ceph-ansible/plugins/actions/config_template.py", line 641, in run
    default_section=_vars.get('default_section', 'DEFAULT')
  File "/usr/share/ceph-ansible/plugins/actions/config_template.py", line 330, in return_config_overrides_ini
    config.readfp(config_object)
  File "/usr/lib64/python2.7/ConfigParser.py", line 324, in readfp
    self._read(fp, filename)
  File "/usr/share/ceph-ansible/plugins/actions/config_template.py", line 289, in _read
    raise e
ParsingError: File contains parsing errors: <???>
	[line 16]: u' # non_hci_safety_factor is the safety factor for dedicated nodes\n'
fatal: [c07-h25-6048r]: FAILED! => {}

MSG:

Unexpected failure during module execution.

Created attachment 1509646 [details]
ansible runlog using -vvv

John, which version of Ansible are you using?

I just tried enabling this, and I'm not able to reproduce your issue.
I suspect, it's a matter of Ansible version, I'm running 2.7.2.

# ansible --version
ansible 2.6.7
  config file = /usr/share/ceph-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

What ansible version is required for RHCS 3.2 ?

> What ansible version is required for RHCS 3.2 ?

Ansible 2.6 (documented in bug 1613941)

The same failure has been reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1654441, let's change BZ for this conversation.

I'm moving this one to POST again, as the original issue has been solved.

Verified with ceph-ansible 3.2.0-0.1.rc8.el7cp.  Ceph-ansible deploys cluster two NVMe devices successfully.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0020