Bug 2089535

Describe the issue
==================

Ceph deployment via cephadm as explained in the text is performing both
bootstrap and cluster configuration in one step, which may be useful in
some cases, but in the text it's done in both highly non optimal and confusing
way. As a consequence, the procedure in the text requires additional relocation
of ceph services. Such step should not be necessary.

Describe the task you were trying to accomplish
===============================================

Deploy a stretched ceph cluster for MetroDR.

Suggestions for improvement
===========================

Better align with and reference to ceph docs.

Distinguish bootstrap and cluster setup as 2 separate steps.

Moreover we can use service configuration yaml file to allocate ceph services
in the cluster so that it matches the table 3.1 (physical server locations and
Ceph component layout for Red Hat Ceph Storage cluster deployment) and whole
chapter "3.6. Relocating the monitors and managers to an appropriate location"
could be dropped as it won't be necessary.

Document URL
============

https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.10/html/configuring_openshift_data_foundation_for_metro-dr_with_advanced_cluster_management/requirements-for-deploying-rhcs-stretch-cluster_rhodf

Chapter/Section Number and Title
================================

3.5. Cluster bootstrapping with Cephadm
3.6. Relocating the monitors and managers to an appropriate location

Product Version
===============

ODF 4.10

Environment Details
===================

MetroDR with ACM: deployment with stretched RHCS cluster.

Any other versions of this document that also needs this update
===============================================================

This text was introduced in 4.10, and should be fixed at least in 4.11.

Additional information
======================

An example of service configuration yaml which can be used to setup cluster
after bootstrap is done:

```
service_type: host
hostname: osd-0.mbukatov-ceph01.qe.example.com
addr: 10.1.161.15
labels:
- osd
- mon
---
service_type: host
hostname: osd-1.mbukatov-ceph01.qe.example.com
addr: 10.1.160.255
labels:
- osd
- mon
---
service_type: host
hostname: osd-2.mbukatov-ceph01.qe.example.com
addr: 10.1.161.36
labels:
- osd
- mds
- rgw
---
service_type: host
hostname: osd-3.mbukatov-ceph01.qe.example.com
addr: 10.1.161.37
labels:
- osd
- mon
---
service_type: host
hostname: osd-4.mbukatov-ceph01.qe.example.com
addr: 10.1.161.40
labels:
- osd
- mon
---
service_type: host
hostname: osd-5.mbukatov-ceph01.qe.example.com
addr: 10.1.161.39
labels:
- osd
- mds
- rgw
---
service_type: host
hostname: arbiter.mbukatov-ceph01.qe.example.com
addr: 10.1.161.33
labels:
- mon
---
service_type: osd
service_id: default_drive_group
data_devices:
  limit: 2
placement:
  label: "osd"
---
service_type: mon
placement:
  label: "mon"
---
service_type: mds
service_id: fs_name
placement:
  label: "mds"
```

Note that labels are only one option how to allocate ceph services in a
cluster, but here it seems to match nicely with the table 3.1. We can always
reference ceph docs stating that there are other options.