Bug 1911669

Description of problem:

There are multiple BZs on this issue & some of them are fixed but the issue still persists which is mentioned in [1] but should be fixed as per [2].

THT version in my env

~~~
(undercloud) [stack@undercloud ~]$ rpm -qa | grep  openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-11.3.2-1.20200914170177.el8ost.noarch

#ceph-ansible

[stack@undercloud ~]$ sudo rpm -qa | grep -i ceph-ansible
ceph-ansible-4.0.31-1.el8cp.noarch
~~~

Followed WA mentioned in our doc[3] & discussed in BZ -https://bugzilla.redhat.com/show_bug.cgi?id=1902153 but issue persists.

Steps followed:

1. Ran overcloud deploy with --stack-only option with "osd_objectstore: filestore"

~~~
cat /home/stack/templates/ceph-config.yaml
parameter_defaults:
  CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
  CephAnsibleExtraConfig:
    osd_objectstore: bluestore

2. In overcloud plan filestore is set

~~~
[stack@undercloud plan]$ grep -ir objectstore .

./plan-environment.yaml:    osd_objectstore: filestore
~~~

3. Ran migration command, the command succeeds but no change is objectstore, it's still showing as 'filestore'.
   As explained in other BZ it simply skipped that task since it sees bluestore is already set.

~~~
/var/lib/mistral/b8ba0cb4-da83-4d3f-9239-2a9e4344093f/ceph-ansible/ceph_ansible_command.log
2020-12-30 10:17:15,625 p=208169 u=root n=ansible | Using /usr/share/ceph-ansible/ansible.cfg as config file
2020-12-30 10:17:16,398 p=208169 u=root n=ansible | PLAY [osds] ********************************************************************
2020-12-30 10:17:16,426 p=208169 u=root n=ansible | TASK [Gathering Facts] *********************************************************
2020-12-30 10:17:16,426 p=208169 u=root n=ansible | Wednesday 30 December 2020  10:17:16 -0500 (0:00:00.086)       0:00:00.086 **** 
2020-12-30 10:17:18,922 p=208169 u=root n=ansible | ok: [overcloud-computehci-0]
2020-12-30 10:17:19,189 p=208169 u=root n=ansible | TASK [gather and delegate facts] ***********************************************
2020-12-30 10:17:19,189 p=208169 u=root n=ansible | Wednesday 30 December 2020  10:17:19 -0500 (0:00:02.763)       0:00:02.849 **** 
2020-12-30 10:17:20,520 p=208169 u=root n=ansible | ok: [overcloud-computehci-0 -> 172.16.0.23] => (item=overcloud-controller-0)
2020-12-30 10:17:21,901 p=208169 u=root n=ansible | ok: [overcloud-computehci-0 -> 172.16.0.39] => (item=overcloud-controller-1)
2020-12-30 10:17:23,179 p=208169 u=root n=ansible | ok: [overcloud-computehci-0 -> 172.16.0.32] => (item=overcloud-controller-2)
2020-12-30 10:17:23,294 p=208169 u=root n=ansible | TASK [set_fact current_objectstore] ********************************************
2020-12-30 10:17:23,294 p=208169 u=root n=ansible | Wednesday 30 December 2020  10:17:23 -0500 (0:00:04.105)       0:00:06.954 **** 
2020-12-30 10:17:23,318 p=208169 u=root n=ansible | ok: [overcloud-computehci-0] => {"ansible_facts": {"current_objectstore": "bluestore"}, "changed": false}
2020-12-30 10:17:23,359 p=208169 u=root n=ansible | TASK [warn user about osd already using bluestore] *****************************
2020-12-30 10:17:23,359 p=208169 u=root n=ansible | Wednesday 30 December 2020  10:17:23 -0500 (0:00:00.064)       0:00:07.019 **** 
2020-12-30 10:17:23,386 p=208169 u=root n=ansible | ok: [overcloud-computehci-0] => {
    "msg": "WARNING: overcloud-computehci-0 is already using bluestore. Skipping all tasks."
}
2020-12-30 10:17:23,426 p=208169 u=root n=ansible | TASK [ceph-facts : include facts.yml] ******************************************

}

[root@overcloud-controller-0 ~]# podman exec -it ceph-mon-overcloud-controller-0 sh -c "ceph -f json osd metadata" | jq -c '.[] | select(.hostname == "overcloud-computehci-0") |["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
["host","overcloud-computehci-0","osd_id",2,"objectstore","filestore"]
["host","overcloud-computehci-0","osd_id",5,"objectstore","filestore"]
~~~
4. remove osd_objectstore from template hence it should be default falls to 'bluestore' but same issue.

~~~
Running /var/lib/mistral/ed1614f6-7c57-4607-9802-d716886853f6/ceph-ansible/ceph_ansible_command.sh
2020-12-30 11:03:19,584 p=285045 u=root n=ansible | Using /usr/share/ceph-ansible/ansible.cfg as config file
2020-12-30 11:03:20,288 p=285045 u=root n=ansible | PLAY [osds] ********************************************************************
2020-12-30 11:03:20,558 p=285045 u=root n=ansible | TASK [gather and delegate facts] ***********************************************
2020-12-30 11:03:20,558 p=285045 u=root n=ansible | Wednesday 30 December 2020  11:03:20 -0500 (0:00:00.327)       0:00:00.327 **** 
2020-12-30 11:03:22,248 p=285045 u=root n=ansible | ok: [overcloud-computehci-0 -> 172.16.0.23] => (item=overcloud-controller-0)
2020-12-30 11:03:24,127 p=285045 u=root n=ansible | ok: [overcloud-computehci-0 -> 172.16.0.39] => (item=overcloud-controller-1)
2020-12-30 11:03:25,956 p=285045 u=root n=ansible | ok: [overcloud-computehci-0 -> 172.16.0.32] => (item=overcloud-controller-2)
2020-12-30 11:03:26,069 p=285045 u=root n=ansible | TASK [set_fact current_objectstore] ********************************************
2020-12-30 11:03:26,070 p=285045 u=root n=ansible | Wednesday 30 December 2020  11:03:26 -0500 (0:00:05.511)       0:00:05.839 **** 
2020-12-30 11:03:26,097 p=285045 u=root n=ansible | ok: [overcloud-computehci-0] => {"ansible_facts": {"current_objectstore": "bluestore"}, "changed": false}
2020-12-30 11:03:26,142 p=285045 u=root n=ansible | TASK [warn user about osd already using bluestore] *****************************
2020-12-30 11:03:26,142 p=285045 u=root n=ansible | Wednesday 30 December 2020  11:03:26 -0500 (0:00:00.072)       0:00:05.911 **** 
2020-12-30 11:03:26,174 p=285045 u=root n=ansible | ok: [overcloud-computehci-0] => {
    "msg": "WARNING: overcloud-computehci-0 is already using bluestore. Skipping all tasks."
}
~~~



[1]https://bugzilla.redhat.com/show_bug.cgi?id=1853721
[2]https://bugzilla.redhat.com/show_bug.cgi?id=1733577
[3]
~~~
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index#migrating-OSDs-from-FileStore-to-BlueStore
~~~


Version-Release number of selected component (if applicable):

OSP16.1

How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:
migration not working

Expected results:
migration should work.

Additional info: