If a customer wishes to upgrade from RHEL7 to RHEL8 but RHEL8 doesn't offer an option to do an in-place upgrade and the server would need to be re-provisioned with RHEL8, then how will this be handled if each of those RHEL7 servers are parts of a Ceph cluster?

In that scenario, it seems better to tell Ceph that the service has been removed and add it back as if it's a new service than to simply turn the node off because Ceph will keep looking for the missing service.

ceph-ansible has infrastructure playbooks to remove parts of a cluster and add those parts back which are useful when nodes are reprovisioned. We don't have playbooks like this for all Ceph daemons however. This is an RFE to track a request for those missing daemons, particularly, the ones that Red Hat OpenStack Platform deploys.

I. scale up:

- ceph-osd: https://github.com/ceph/ceph-ansible/blob/master/infrastructure-playbooks/add-osd.yml
- ceph-mon: https://github.com/ceph/ceph-ansible/pull/3547
- ceph-mds: https://github.com/ceph/ceph-ansible/pull/3599
- ceph-rgw: missing
- ceph-rbd-mirror: missing
- ceph-mgr: missing

II. scale down:

- ceph-osd: https://github.com/ceph/ceph-ansible/blob/master/infrastructure-playbooks/shrink-osd.yml
- ceph-mon: https://github.com/ceph/ceph-ansible/blob/master/infrastructure-playbooks/shrink-mon.yml
- ceph-mds: missing
- ceph-rgw: missing
- ceph-rbd-mirror: missing
- ceph-mgr: missing