Bug 2136765
| Summary: | Provider cannot deduce API server reachability of API server | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Jilju Joy <jijoy> |
| Component: | odf-managed-service | Assignee: | Dhruv Bindra <dbindra> |
| Status: | CLOSED EOL | QA Contact: | suchita <sgatfane> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.10 | CC: | aeyal, dbindra, kramdoss, lgangava, mmuench, muagarwa, nberry, odf-bz-bot, resoni, sgatfane, sostapov |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2112852 | Environment: | |
| Last Closed: | 2024-07-11 10:26:45 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2112852 | ||
| Bug Blocks: | 2120314 | ||
|
Comment 14
Rewant
2023-08-30 05:58:15 UTC
Test on V2.1.0 multicidr Privatelink cluster setup
Below are oc command output and observation on provider and consumer cluster:
----------------- Provider Nodes o/p -------------
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-206-38-37.us-east-2.compute.internal Ready infra,worker 11h v1.24.15+a9da4a8
ip-10-206-39-205.us-east-2.compute.internal Ready worker 11h v1.24.15+a9da4a8
ip-10-206-39-28.us-east-2.compute.internal Ready master 11h v1.24.15+a9da4a8
ip-10-206-41-105.us-east-2.compute.internal Ready master 11h v1.24.15+a9da4a8
ip-10-206-41-173.us-east-2.compute.internal Ready worker 11h v1.24.15+a9da4a8
ip-10-206-41-5.us-east-2.compute.internal Ready infra,worker 11h v1.24.15+a9da4a8
ip-10-206-42-174.us-east-2.compute.internal Ready infra,worker 11h v1.24.15+a9da4a8
ip-10-206-43-0.us-east-2.compute.internal Ready worker 11h v1.24.15+a9da4a8
ip-10-206-43-247.us-east-2.compute.internal Ready master 11h v1.24.15+a9da4a8
--------------------------------------------------
Endpoint used in consumer for testing
Worker node IP: 10.206.41.173
Infra node IP: 10.206.41.5
--------------------------------------------------
-----------------Consumer storage endpoint with worker node------------------
$ rosa describe addon-installation --cluster sgatfane-14sc1 --addon ocs-consumer-qe
Id: ocs-consumer-qe
Href: /api/clusters_mgmt/v1/clusters/2683r7ai7au8kns4dcpmjl3n7oqjrlr5/addons/ocs-consumer-qe
Addon state: ready
Parameters:
"onboarding-ticket" : "eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=\n"
"storage-provider-endpoint" : "10.206.41.173:31659"
"notification-email-0" : "odf-ms-qe"
"notification-email-1" : "sgatfane"
"notification-email-2" : ""
$ oc get managedocs -o yaml
apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1alpha1
kind: ManagedOCS
metadata:
creationTimestamp: "2023-09-14T05:50:31Z"
finalizers:
- managedocs.ocs.openshift.io
generation: 1
name: managedocs
namespace: openshift-storage
resourceVersion: "593815"
uid: f95d5223-133f-42c3-8cd0-62904d4ba034
spec: {}
status:
components:
alertmanager:
state: Ready
prometheus:
state: Ready
storageCluster:
state: Pending
reconcileStrategy: strict
kind: List
metadata:
resourceVersion: ""
selfLink: ""
-------- lastHeartbeat updated every min on provider ---------
o/p1 of storage cluster yaml. at Thu Sep 14 16:35:03 UTC 2023
apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1alpha1
kind: StorageConsumer
metadata:
annotations:
ocs.openshift.io/provider-onboarding-ticket: |
eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=
creationTimestamp: "2023-09-14T15:56:37Z"
generation: 2
name: storageconsumer-96df7c11-6fe3-4eb7-894b-fb564215d6c8
namespace: openshift-storage
resourceVersion: "593181"
uid: 667761cc-c13c-4599-b204-952aae6b157a
spec:
capacity: 1Pi
enable: true
status:
cephResources:
- kind: CephClient
name: 193b27194ba00d5964610811aca008a8
status: Ready
grantedCapacity: 1Pi
lastHeartbeat: "2023-09-14T16:34:05Z"
state: Ready
kind: List
metadata:
resourceVersion: ""
selfLink: ""
o/p2 of storage cluster yaml. at Thu Sep 14 16:35:12 UTC 2023
apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1alpha1
kind: StorageConsumer
metadata:
annotations:
ocs.openshift.io/provider-onboarding-ticket: |
eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=
creationTimestamp: "2023-09-14T15:56:37Z"
generation: 2
name: storageconsumer-96df7c11-6fe3-4eb7-894b-fb564215d6c8
namespace: openshift-storage
resourceVersion: "593926"
uid: 667761cc-c13c-4599-b204-952aae6b157a
spec:
capacity: 1Pi
enable: true
status:
cephResources:
- kind: CephClient
name: 193b27194ba00d5964610811aca008a8
status: Ready
grantedCapacity: 1Pi
lastHeartbeat: "2023-09-14T16:35:04Z"
state: Ready
kind: List
metadata:
resourceVersion: ""
selfLink: ""
-------------------------------------------------------
Changed the storage endpoint on consumer with infra or infra,worker nodeIP:
$ rosa edit addon --cluster sgatfane-14sc1 ocs-consumer-qe --storage-provider-endpoint "10.206.41.5:31659"
I: Add-on 'ocs-consumer-qe' is now updating. To check the status run 'rosa list addons -c sgatfane-14sc1'
$ oc get managedocs -o yaml
apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1alpha1
kind: ManagedOCS
metadata:
creationTimestamp: "2023-09-14T05:50:31Z"
finalizers:
- managedocs.ocs.openshift.io
generation: 1
name: managedocs
namespace: openshift-storage
resourceVersion: "600121"
uid: f95d5223-133f-42c3-8cd0-62904d4ba034
spec: {}
status:
components:
alertmanager:
state: Ready
prometheus:
state: Ready
storageCluster:
state: Pending
reconcileStrategy: strict
kind: List
metadata:
resourceVersion: ""
selfLink: ""
$ rosa describe addon-installation --cluster sgatfane-14sc1 --addon ocs-consumer-qe
Id: ocs-consumer-qe
Href: /api/clusters_mgmt/v1/clusters/2683r7ai7au8kns4dcpmjl3n7oqjrlr5/addons/ocs-consumer-qe
Addon state: ready
Parameters:
"onboarding-ticket" : "eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=\n"
"storage-provider-endpoint" : "10.206.41.5:31659"
"notification-email-0" : "odf-ms-qe"
"notification-email-1" : "sgatfane"
"notification-email-2" : ""
------------------------------------------------------------------
On Provider, lastHeartbeat value does not get updated when consumer get diconnected , storageclyster on consumer went to pending state.
o/p of storageconsumer yaml at utc time Thu Sep 14 16:40:08 UTC 2023
apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1alpha1
kind: StorageConsumer
metadata:
annotations:
ocs.openshift.io/provider-onboarding-ticket: |
eyJpZCI6ImFiZDllMmU3LTk0ZTUtNDVlMS05MWUyLWI2Y2U0ZGY2MWI2YSIsImV4cGlyYXRpb25EYXRlIjoiMTY5NDg0MzI5MiJ9.LvtbsgplDUMVTaTNRrC5KinDJKhOwFKgx6BLRqTQpV7IXTRp7+LDZ+LOAUgOFu3zPKq8VobWiHag3Arw4X/RhGvo0a3tBv9G19G+2f2ZfG2IICOTXdc/wwpN5MWeLhOb2fVjRvi2KiXETHnuu11D0u7cq++Z2L8CatliheJj7Dvw8zKKLtuXYOkaGaOF3EHHwHSpZ95ogzn1Bo4XuET/s2nAWThTBOzWH75DTLWQsUuBt3c8sEpkaKe10M7bYPrchCa2yC7n9cx8K9wZ3ywXjiGoUrXk69yAp4k1hbtlDAyRNeajUQZH0kGtmxVT695WporUAjg8Q3Czd8Of8/Hq3Q5+9XKdHmOdw6+VL3CtwyelwjjxQVfAqiaGtc1tTvB6OAAukCBaoKQjAgvGLV2mNgtGJB0uXbtjPk3DyAJzqjBidGDDeUo/sZrajQ0AHWSnfhpmwDTDFUboVEHyGuzM210HMLt3V5/6Y3KVWGxOTauKwgQaH8bRCsZfE+N+AGYOS0qHqJieT71CH9lOKtQhoz5+DxTYjn8lwR5EfTjWENALzkDOzUGZBxPE6gWtJmGEhlcTbW4X3+cQx5lkvKr2R+t7LXWUxWjd0n37fYcnB7ir0zQL09Hc6x2yt8oG2Eu4lr6WWlanasPzN2/DDt2aOIiae7ysdFbeWpkquTs3bNw=
creationTimestamp: "2023-09-14T15:56:37Z"
generation: 2
name: storageconsumer-96df7c11-6fe3-4eb7-894b-fb564215d6c8
namespace: openshift-storage
resourceVersion: "593926"
uid: 667761cc-c13c-4599-b204-952aae6b157a
spec:
capacity: 1Pi
enable: true
status:
cephResources:
- kind: CephClient
name: 193b27194ba00d5964610811aca008a8
status: Ready
grantedCapacity: 1Pi
lastHeartbeat: "2023-09-14T16:35:04Z"
state: Ready
kind: List
metadata:
resourceVersion: ""
selfLink: ""
--------------------------------------------------------------------
Based on Observation of "lastHeartbeat" parameter from provider storageconsumer CR with respect to storagecluster readyness of consumer, marking this BZ as verified. $ oc get storagecluster NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 11h Ready true 2023-09-14T05:50:48Z [jenkins@odf-ms-stage ~]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.11.10 NooBaa Operator 4.11.10 mcg-operator.v4.11.9 Succeeded observability-operator.v0.0.25 Observability Operator 0.0.25 observability-operator.v0.0.25-rc Succeeded ocs-operator.v4.11.10 OpenShift Container Storage 4.11.10 ocs-operator.v4.11.9 Succeeded ocs-osd-deployer.v2.1.0 OCS OSD Deployer 2.1.0 ocs-osd-deployer.v2.0.13 Succeeded odf-csi-addons-operator.v4.11.10 CSI Addons 4.11.10 odf-csi-addons-operator.v4.11.9 Succeeded odf-operator.v4.11.10 OpenShift Data Foundation 4.11.10 odf-operator.v4.11.9 Succeeded ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 ose-prometheus-operator.4.8.0 Succeeded route-monitor-operator.v0.1.570-71112a2 Route Monitor Operator 0.1.570-71112a2 route-monitor-operator.v0.1.568-8024e29 Succeeded [jenkins@odf-ms-stage ~]$ oc get pods NAME READY STATUS RESTARTS AGE 0ed573a1a553d98358b65b6f5d76d8167f5bb965a4495f5290d5c846859vh5j 0/1 Completed 0 11h 1dfdb110aeb675d61f888e0b30d6210e253cf317203ae8fd20c56dd8c5f85zc 0/1 Completed 0 11h 1ef9c4506c288432b5733bbdcc56f1b1df297a84aa1e8a9993cf995f0cdj8fg 0/1 Completed 0 11h 5be715c1f41baa1aa3b925135fe62ae151c953c6a8c20a2f0aab78659d8pm7t 0/1 Completed 0 11h addon-ocs-consumer-qe-catalog-s2cdp 1/1 Running 0 11h alertmanager-managed-ocs-alertmanager-0 2/2 Running 0 11h b6153cdbca00685efbb04a91478ec6382b1cc8a080125a027494cfead74qglk 0/1 Completed 0 11h csi-addons-controller-manager-66fbb9b5f4-6bmnl 2/2 Running 0 11h csi-cephfsplugin-27zns 2/2 Running 0 68m csi-cephfsplugin-bfppm 2/2 Running 0 68m csi-cephfsplugin-f7gk5 2/2 Running 0 68m csi-cephfsplugin-provisioner-84464d7757-5t6gs 5/5 Running 0 68m csi-cephfsplugin-provisioner-84464d7757-jrrbp 5/5 Running 0 68m csi-rbdplugin-588qj 3/3 Running 0 68m csi-rbdplugin-7qm77 3/3 Running 0 68m csi-rbdplugin-fm47z 3/3 Running 0 68m csi-rbdplugin-provisioner-5b6fcff8cf-8lvtv 6/6 Running 0 68m csi-rbdplugin-provisioner-5b6fcff8cf-8z8dm 6/6 Running 0 68m eea43d0ece9a5d5c02eb930023be5a7b31fa226f00d4891fc837b0c639rbgsl 0/1 Completed 0 11h ocs-metrics-exporter-96cf8f78c-cnzd7 1/1 Running 0 11h ocs-operator-5f499d9888-bfqtm 1/1 Running 0 68m ocs-osd-aws-data-gather-6d7f9ff58f-hn46z 1/1 Running 0 11h ocs-osd-controller-manager-54b7d5d7c5-n6ck8 3/3 Running 0 11h odf-console-558569cfcf-4f8q8 1/1 Running 0 11h odf-operator-controller-manager-bcfb88f9d-l2n8b 2/2 Running 0 11h prometheus-managed-ocs-prometheus-0 3/3 Running 0 11h prometheus-operator-75bd4b559b-4ljc2 1/1 Running 0 11h redhat-operators-xlw6r 1/1 Running 0 11h report-status-to-provider-28245185-bkhw5 0/1 Completed 0 3m report-status-to-provider-28245186-r7hsz 0/1 Completed 2 2m report-status-to-provider-28245187-tsg6w 0/1 Completed 0 60s report-status-to-provider-28245188-kqd8t 0/1 ContainerCreating 0 0s rook-ceph-operator-fd8d7c44-6kq69 1/1 Running 0 11h rook-ceph-tools-789847c4d7-6zpvc 1/1 Running 0 68m The ODF Managed Service Project has sunset and is now consider obsolete |