Describe the issue ================== In chapter dealing with installation of ODF storage system in external mode, we are just referencing to "Deploying OpenShift Data foundation in external mode" guide without highlighting peculiarities of MetroDR environment. Describe the task you were trying to accomplish =============================================== Install storage system for MetroDR with ACM environment. Suggestions for improvement =========================== In chapter dealing with installation of ODF storage system in external mode, we need to explain/highlight at least the following peculiarities of MetroDR with ACM which could be missed: - list of options (or group of options) of ceph-external-cluster-details-exporter.py script which should be specified in MetroDR environment (I assume that one needs to specify at least --rbd-data-pool-name option, --cephfs-* options and --rgw-endpoint) - if some option won't work with metro dr, we should highlight it as not supported (I'm not aware of such option, but let's recheck that when fixing this) - note whether I need monitoring-endpoint for proper function of MetroDR setup - note that one needs to run the exporter script twice, since each ODF storage system needs to get reference to rgw endopoint in the same zone Listing this would stress what is expected and supported configuration, prevent misconfiguration and make deployment of MetroDR systems easier. We don't need to go into details, but we need to address assumptions which stems from the MetroDR use case. Document URL ============ https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.10/html/configuring_openshift_data_foundation_for_metro-dr_with_advanced_cluster_management/installing-odf-operator-on-managed-clusters_rhodf Chapter/Section Number and Title ================================ Chapter 5. Installing OpenShift Data Foundation on managed clusters Product Version =============== ODF 4.10 Environment Details =================== MetroDR with ACM: connecting ODF storage system on 2 OCP cluster with one external ceph stretched in zones of these 2 OCP clusters. Any other versions of this document that also needs this update =============================================================== This text was introduced in 4.10, and should be fixed at least in 4.11. Additional information ====================== The assumptions should be acknowledged by ODF developers. I could be wrong (which is another reason to state these details directly in the docs). Additional information about RGW and zones in MetroDR environment ================================================================= Assuming I have: - 2 RGW endpoints, one in each zone of a stretched cluster - OCP1 is in the zone where 1st half of stretched ceph, RGW is running there in ceph-2 machine - OCP2 is in the zone where 2nd half of stretched ceph, RGW is running there ceph-5 machine Which would look like this from ceph point of view: ``` [root@ceph-0 ~]# ceph -s | grep rgw rgw: 2 daemons active (2 hosts, 1 zones) [ceph: root@ceph-0 /]# ceph orch ps --daemon_type=rgw NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID rgw.objectgw.ceph-2.upbjyy ceph-2.mbukatov-ceph01.qe.example.com *:8080 running (2d) 9m ago 2d 68.7M - 16.2.7-112.el8cp cf1c8f1da91f 18d05e3f4d1d rgw.objectgw.ceph-5.qswtya ceph-5.mbukatov-ceph01.qe.example.com *:8080 running (2d) 9m ago 2d 68.4M - 16.2.7-112.el8cp cf1c8f1da91f 9645cba03f00 ``` I need to export the json 2 times, for each OCP/ODF/zone, for example: ``` # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --cephfs-filesystem-name fs_foo --cephfs-metadata-pool-name cephfs.fs_foo.meta --cephfs-data-pool-name cephfs.fs_foo.data --rgw-endpoint ceph-2.mbukatov-ceph01.qe.example.com:8080 > ocp1.json # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --cephfs-filesystem-name fs_foo --cephfs-metadata-pool-name cephfs.fs_foo.meta --cephfs-data-pool-name cephfs.fs_foo.data --rgw-endpoint ceph-5.mbukatov-ceph01.qe.example.com:8080 > ocp2.json ```
I see that: - list of mandatory options of ceph-external-cluster-details-exporter.py script is provided - no mention of monitoring endpoint indicates that this part is not necessary - instructions to run the exporter script twice are present During checking of the new text, I also manually imported external stretch cluster following the new instructions to make sure the setup works as expected omitting the cephfs details: ``` # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --rgw-endpoint 192.0.2.98:8080 --run-as-user client.odf.cluster1 > ocp-cluster1.json # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --rgw-endpoint 192.0.2.69:8080 --run-as-user client.odf.cluster2 > ocp-cluster2.json ``` And the script was able to figure out the name of cephfs, and the external ODF setup finished with success. Versions used: RHCS 5.2 on RHEL 8.6 OCP 4.11.0-0.nightly-2022-08-19-091806 ODF 4.11.0-137 (RC3)