Fedora Account System
Red Hat Associate
Red Hat Customer
Moving this to ON_QA as we have the fix from the product in ODF 4.11, and ODF 4.11.10 is now GA, so we can verify this bug.
Test on V2.1.0 multicidr Privatelink cluster setup Below are oc command output and observation on provider and consumer cluster: ----------------- Provider Nodes o/p ------------- $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-206-38-37.us-east-2.compute.internal Ready infra,worker 11h v1.24.15+a9da4a8 ip-10-206-39-205.us-east-2.compute.internal Ready worker 11h v1.24.15+a9da4a8 ip-10-206-39-28.us-east-2.compute.internal Ready master 11h v1.24.15+a9da4a8 ip-10-206-41-105.us-east-2.compute.internal Ready master 11h v1.24.15+a9da4a8 ip-10-206-41-173.us-east-2.compute.internal Ready worker 11h v1.24.15+a9da4a8 ip-10-206-41-5.us-east-2.compute.internal Ready infra,worker 11h v1.24.15+a9da4a8 ip-10-206-42-174.us-east-2.compute.internal Ready infra,worker 11h v1.24.15+a9da4a8 ip-10-206-43-0.us-east-2.compute.internal Ready worker 11h v1.24.15+a9da4a8 ip-10-206-43-247.us-east-2.compute.internal Ready master 11h v1.24.15+a9da4a8 -------------------------------------------------- Endpoint used in consumer for testing Worker node IP: 10.206.41.173 Infra node IP: 10.206.41.5 -------------------------------------------------- -----------------Consumer storage endpoint with worker node------------------ $ rosa describe addon-installation --cluster sgatfane-14sc1 --addon ocs-consumer-qe Id: ocs-consumer-qe Href: /api/clusters_mgmt/v1/clusters/2683r7ai7au8kns4dcpmjl3n7oqjrlr5/addons/ocs-consumer-qe Addon state: ready Parameters: "onboarding-ticket" : "eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=\n" "storage-provider-endpoint" : "10.206.41.173:31659" "notification-email-0" : "odf-ms-qe" "notification-email-1" : "sgatfane" "notification-email-2" : "" $ oc get managedocs -o yaml apiVersion: v1 items: - apiVersion: ocs.openshift.io/v1alpha1 kind: ManagedOCS metadata: creationTimestamp: "2023-09-14T05:50:31Z" finalizers: - managedocs.ocs.openshift.io generation: 1 name: managedocs namespace: openshift-storage resourceVersion: "593815" uid: f95d5223-133f-42c3-8cd0-62904d4ba034 spec: {} status: components: alertmanager: state: Ready prometheus: state: Ready storageCluster: state: Pending reconcileStrategy: strict kind: List metadata: resourceVersion: "" selfLink: "" -------- lastHeartbeat updated every min on provider --------- o/p1 of storage cluster yaml. at Thu Sep 14 16:35:03 UTC 2023 apiVersion: v1 items: - apiVersion: ocs.openshift.io/v1alpha1 kind: StorageConsumer metadata: annotations: ocs.openshift.io/provider-onboarding-ticket: | eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw= creationTimestamp: "2023-09-14T15:56:37Z" generation: 2 name: storageconsumer-96df7c11-6fe3-4eb7-894b-fb564215d6c8 namespace: openshift-storage resourceVersion: "593181" uid: 667761cc-c13c-4599-b204-952aae6b157a spec: capacity: 1Pi enable: true status: cephResources: - kind: CephClient name: 193b27194ba00d5964610811aca008a8 status: Ready grantedCapacity: 1Pi lastHeartbeat: "2023-09-14T16:34:05Z" state: Ready kind: List metadata: resourceVersion: "" selfLink: "" o/p2 of storage cluster yaml. at Thu Sep 14 16:35:12 UTC 2023 apiVersion: v1 items: - apiVersion: ocs.openshift.io/v1alpha1 kind: StorageConsumer metadata: annotations: ocs.openshift.io/provider-onboarding-ticket: | eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw= creationTimestamp: "2023-09-14T15:56:37Z" generation: 2 name: storageconsumer-96df7c11-6fe3-4eb7-894b-fb564215d6c8 namespace: openshift-storage resourceVersion: "593926" uid: 667761cc-c13c-4599-b204-952aae6b157a spec: capacity: 1Pi enable: true status: cephResources: - kind: CephClient name: 193b27194ba00d5964610811aca008a8 status: Ready grantedCapacity: 1Pi lastHeartbeat: "2023-09-14T16:35:04Z" state: Ready kind: List metadata: resourceVersion: "" selfLink: "" ------------------------------------------------------- Changed the storage endpoint on consumer with infra or infra,worker nodeIP: $ rosa edit addon --cluster sgatfane-14sc1 ocs-consumer-qe --storage-provider-endpoint "10.206.41.5:31659" I: Add-on 'ocs-consumer-qe' is now updating. To check the status run 'rosa list addons -c sgatfane-14sc1' $ oc get managedocs -o yaml apiVersion: v1 items: - apiVersion: ocs.openshift.io/v1alpha1 kind: ManagedOCS metadata: creationTimestamp: "2023-09-14T05:50:31Z" finalizers: - managedocs.ocs.openshift.io generation: 1 name: managedocs namespace: openshift-storage resourceVersion: "600121" uid: f95d5223-133f-42c3-8cd0-62904d4ba034 spec: {} status: components: alertmanager: state: Ready prometheus: state: Ready storageCluster: state: Pending reconcileStrategy: strict kind: List metadata: resourceVersion: "" selfLink: "" $ rosa describe addon-installation --cluster sgatfane-14sc1 --addon ocs-consumer-qe Id: ocs-consumer-qe Href: /api/clusters_mgmt/v1/clusters/2683r7ai7au8kns4dcpmjl3n7oqjrlr5/addons/ocs-consumer-qe Addon state: ready Parameters: "onboarding-ticket" : "eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=\n" "storage-provider-endpoint" : "10.206.41.5:31659" "notification-email-0" : "odf-ms-qe" "notification-email-1" : "sgatfane" "notification-email-2" : "" ------------------------------------------------------------------ On Provider, lastHeartbeat value does not get updated when consumer get diconnected , storageclyster on consumer went to pending state. o/p of storageconsumer yaml at utc time Thu Sep 14 16:40:08 UTC 2023 apiVersion: v1 items: - apiVersion: ocs.openshift.io/v1alpha1 kind: StorageConsumer metadata: annotations: ocs.openshift.io/provider-onboarding-ticket: | eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw= creationTimestamp: "2023-09-14T15:56:37Z" generation: 2 name: storageconsumer-96df7c11-6fe3-4eb7-894b-fb564215d6c8 namespace: openshift-storage resourceVersion: "593926" uid: 667761cc-c13c-4599-b204-952aae6b157a spec: capacity: 1Pi enable: true status: cephResources: - kind: CephClient name: 193b27194ba00d5964610811aca008a8 status: Ready grantedCapacity: 1Pi lastHeartbeat: "2023-09-14T16:35:04Z" state: Ready kind: List metadata: resourceVersion: "" selfLink: "" --------------------------------------------------------------------
Based on Observation of "lastHeartbeat" parameter from provider storageconsumer CR with respect to storagecluster readyness of consumer, marking this BZ as verified. $ oc get storagecluster NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 11h Ready true 2023-09-14T05:50:48Z [jenkins@odf-ms-stage ~]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.11.10 NooBaa Operator 4.11.10 mcg-operator.v4.11.9 Succeeded observability-operator.v0.0.25 Observability Operator 0.0.25 observability-operator.v0.0.25-rc Succeeded ocs-operator.v4.11.10 OpenShift Container Storage 4.11.10 ocs-operator.v4.11.9 Succeeded ocs-osd-deployer.v2.1.0 OCS OSD Deployer 2.1.0 ocs-osd-deployer.v2.0.13 Succeeded odf-csi-addons-operator.v4.11.10 CSI Addons 4.11.10 odf-csi-addons-operator.v4.11.9 Succeeded odf-operator.v4.11.10 OpenShift Data Foundation 4.11.10 odf-operator.v4.11.9 Succeeded ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 ose-prometheus-operator.4.8.0 Succeeded route-monitor-operator.v0.1.570-71112a2 Route Monitor Operator 0.1.570-71112a2 route-monitor-operator.v0.1.568-8024e29 Succeeded [jenkins@odf-ms-stage ~]$ oc get pods NAME READY STATUS RESTARTS AGE 0ed573a1a553d98358b65b6f5d76d8167f5bb965a4495f5290d5c846859vh5j 0/1 Completed 0 11h 1dfdb110aeb675d61f888e0b30d6210e253cf317203ae8fd20c56dd8c5f85zc 0/1 Completed 0 11h 1ef9c4506c288432b5733bbdcc56f1b1df297a84aa1e8a9993cf995f0cdj8fg 0/1 Completed 0 11h 5be715c1f41baa1aa3b925135fe62ae151c953c6a8c20a2f0aab78659d8pm7t 0/1 Completed 0 11h addon-ocs-consumer-qe-catalog-s2cdp 1/1 Running 0 11h alertmanager-managed-ocs-alertmanager-0 2/2 Running 0 11h b6153cdbca00685efbb04a91478ec6382b1cc8a080125a027494cfead74qglk 0/1 Completed 0 11h csi-addons-controller-manager-66fbb9b5f4-6bmnl 2/2 Running 0 11h csi-cephfsplugin-27zns 2/2 Running 0 68m csi-cephfsplugin-bfppm 2/2 Running 0 68m csi-cephfsplugin-f7gk5 2/2 Running 0 68m csi-cephfsplugin-provisioner-84464d7757-5t6gs 5/5 Running 0 68m csi-cephfsplugin-provisioner-84464d7757-jrrbp 5/5 Running 0 68m csi-rbdplugin-588qj 3/3 Running 0 68m csi-rbdplugin-7qm77 3/3 Running 0 68m csi-rbdplugin-fm47z 3/3 Running 0 68m csi-rbdplugin-provisioner-5b6fcff8cf-8lvtv 6/6 Running 0 68m csi-rbdplugin-provisioner-5b6fcff8cf-8z8dm 6/6 Running 0 68m eea43d0ece9a5d5c02eb930023be5a7b31fa226f00d4891fc837b0c639rbgsl 0/1 Completed 0 11h ocs-metrics-exporter-96cf8f78c-cnzd7 1/1 Running 0 11h ocs-operator-5f499d9888-bfqtm 1/1 Running 0 68m ocs-osd-aws-data-gather-6d7f9ff58f-hn46z 1/1 Running 0 11h ocs-osd-controller-manager-54b7d5d7c5-n6ck8 3/3 Running 0 11h odf-console-558569cfcf-4f8q8 1/1 Running 0 11h odf-operator-controller-manager-bcfb88f9d-l2n8b 2/2 Running 0 11h prometheus-managed-ocs-prometheus-0 3/3 Running 0 11h prometheus-operator-75bd4b559b-4ljc2 1/1 Running 0 11h redhat-operators-xlw6r 1/1 Running 0 11h report-status-to-provider-28245185-bkhw5 0/1 Completed 0 3m report-status-to-provider-28245186-r7hsz 0/1 Completed 2 2m report-status-to-provider-28245187-tsg6w 0/1 Completed 0 60s report-status-to-provider-28245188-kqd8t 0/1 ContainerCreating 0 0s rook-ceph-operator-fd8d7c44-6kq69 1/1 Running 0 11h rook-ceph-tools-789847c4d7-6zpvc 1/1 Running 0 68m
The ODF Managed Service Project has sunset and is now consider obsolete