Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1824486 [details] Screenshot of the ACM console reflecting message ***Upgrading to 4.7.28***. Description of problem: After upgrading the RHOCP cluster from v4.7.28 to v4.7.29, the upgrade is successful and the overall cluster is healthy and stable. The output of ***$ oc get clusterversion*** is reflecting OCP v4.7.29, However, the ACM console is reflecting it as ***Upgrading to 4.7.28***. ~~~ $ oc get clusterversion [[ us-acm ]] NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.29 True False 31h Cluster version is 4.7.29 [[ us-test ]] NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.29 True False 31h Cluster version is 4.7.29 [[ us-dev ]] NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.29 True False 31h Cluster version is 4.7.29 [[ us-prod ]] NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.29 True False 31h Cluster version is 4.7.29 [[ eu-dev ]] NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.29 True False 31h Cluster version is 4.7.29 [[ eu-prod ]] NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.29 True False 30h Cluster version is 4.7.29 ~~~ [1] have analyzed the must-gather report, no signs of errors caused during the upgrade. [2] have analyzed the ns/open-cluster-management(ACM instance), no signs of errors, and all pods are up and running. [3] have tried workaround as clearing web browser cache/cookies, accessing URL via Incognito/private window. Version-Release number of selected component (if applicable): RHACM v2.3.2 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: The cluster distribution version as seen in the ACM console should reflect the updated cluster version. Additional info: The customer has upgraded six RHOCP clusters from v4.6.28 to v4.6.29, all clusters have upgraded successfully and are stable, however, three of the six clusters are still refecting ***Upgrading to 4.7.28*** in the ACM console. I will attach the must-gather report if needed.
G2Bsync 933626538 comment jamestalton Mon, 04 Oct 2021 16:00:33 UTC G2Bsync ``` isUpgradeCuration = clusterCurator?.spec?.desiredCuration === 'upgrade' || checkCuratorLatestOperation(CuratorCondition.upgrade, curatorConditions) || checkCuratorLatestFailedOperation(CuratorCondition.upgrade, curatorConditions) curatorIsUpgrading = isUpgradeCuration && clusterCurator?.spec?.upgrade?.desiredUpdate !== managedClusterInfo?.status?.distributionInfo?.ocp?.version isUpgrading = curatorIsUpgrading || desiredVersion !== managedClusterInfo?.status?.distributionInfo?.ocp?.version ``` We need the ClusterCurator and ManagedClusterInfo resources for a cluster in this state to determine why it is displaying the wrong status.
Utkarsh, we are not able to access the support case you created. (Neither jtalton nor kcormier) Do you know how we can access?
Yes, we are able to see these files and are investigating. Thank you!
Can you please clarify the new query from the customer: "We are unable to Select a channel from the ACM clusters page for the 4.7.30" 1. From the lastest screen captures, all of the clusters are on OCP 4.8.11 while "us-dev" shows as upgrading to 4.7.30. Is it this one that is having the problem with selecting a channel? 2. Is the option to change the channel available? Is the list of available channels not as expected? Or is there an error when you select a new channel? Or does ACM appear to allow you to change the channel, but it has no effect on the actual cluster?
Hello Team, To avoid this confusion, Let me summarize things from the start. [1] The case was initially opened for customer-facing issues as, the customer have upgraded the ***six*** clusters from v4.7.28 to v4.7.29, out of which ***three*** clusters were reflecting ***upgrading to v4.7.28*** though the upgrade was completed successfully and all the operators were on v4.7.29. Can be clearly seen from the attachment[1] on this bug on ***2021-09-20***. [2] Then the customer aging upgraded these six clusters from ***v4.7.29*** to ***v4.7.30***, the issue still persisted as the affected three clusters were reflecting the same ***upgrading to v4.7.28***, though the cluster was upgraded successfully and all operators were at v4.7.30. [3] Then the customer aging upgraded the six clusters from ***v4.7.30*** to ***v4.8.11***, the issue still persisted as the affected three clusters were reflecting the same ***upgrading to ***v4.7.28***, though the cluster was upgraded successfully and all operators were at ***v4.8.11. But this time cu had new concerns as that customer has updated some OCP clusters to ***v4.8.11*** and the current version for ACM is ***2.3.2*** and some clusters stuck in still in ***v4.7.30***. Can be seen clearly from the output shared on the case on comment 20 on the case. ~~~ Just an observation, we are unable to Select channels from the ACM clusters page for the 4.7.30 clusters and those stuck at "Upgrading". Name | Distribution version | ... us-dev | OpenShift 4.7.30 | Edit labels, Search cluster, Detach cluster, Destroy cluster us-prod | OpenShift 4.7.30 - Upgrading to 4.7.28 | Edit labels, Search cluster, Detach cluster, Destroy cluster eu-prod | OpenShift 4.7.30 - Upgrading to 4.7.28 | Edit labels, Search cluster, Detach cluster, Destroy cluster eu-dev | OpenShift 4.8.11 - Upgrading to 4.7.28 | Edit labels, Search cluster, Detach cluster, Destroy cluster us-test | OpenShift 4.8.11 | Edit labels, Select channel, Search cluster, Detach cluster, Destroy cluster local-cluster | OpenShift 4.8.11 | Edit labels, Select channel, Search cluster, Detach cluster ~~~ [4] After this engineering team requested for ***ClusterCurator and ManagedClusterInfo resources for a cluster in this state to determine why it is displaying the wrong status.*** as attached to the bug on comments 4 and 7. [5] After this, the CU has not upgraded the cluster from v4.8.11 to further. I hope the above summarized the issue till now and clears CU's new query ***We are unable to Select a channel from the ACM clusters page for the 4.7.30***. If need I can ask the customer for a fresh screenshot of the console, to check which clusters are still reflecting the issue ***upgrading to v4.7.28*** and which clusters are reflecting ***unable to select channel issue***. Do let me know your thoughts on the same. Regards,
B2Gsync We are still investigating the problem with clusters stuck showing an upgrading status. As a workaround, you can delete the ClusterCurator resource in the cluster namespace for each cluster showing the stuck status. For example, for eu-dev, on the ACM hub cluster run: oc -n eu-dev delete clustercurator eu-dev For clusters that are not offering the "Select channel" option, we would need to see the ManagedClusterInfo from the hub, and also the ClusterVersion resource from the managed cluster, current as of when the UI issue is occurring. In the table above, 3 of the clusters (us-dev, us-prod, eu-prod) show version 4.7.30, but in the data you collected, they are all at 4.8.11. In order to show the "Select channel" option, the ClusterVersion resource on the managed cluster needs to provide available channels. This is information is then reflected in the ManagedClusterInfo. If the cluster is in a state where its current version does not belong to its current selected channel, the list of available channels is not provided, so we will not have the "Select channel" option available in ACM. It can be changed by manually typing in a new channel name in the OCP console directly on the managed cluster. If this is the case for those clusters, in OCP under Administration > Cluster Settings, you would see an error alert saying "Version x.y.z not found in channel channel-x.y. To request update recommendations, configure a channel that supports your version."
Hello Team, Thanks for your response and suggesting the workaround as above. I have the latest update from the customer side as CU has updated the ACM version from ***2.3.3 to 2.4.0*** and now facing the issue as the clusters are stuck at the "Resuming" state and the misleading version issue is resolved ***Upgrading to 4.7.28*** from the web-console. To elaborate more: [1] eu-dev is on OCP v4.8.17 and reflecting error on console as "Resuming" [2] eu-prod is on OCP v4.8.17 and reflecting error on console as "Resuming" [3] local-cluster is on OCP v4.8.17 and reflects no error and even upgrade path is available ------> no issue for this cluster. [4] us-dev is on OCP v4.8.17 and reflecting error on console as "Resuming" [5] us-prod is on OCP v4.8.17 and reflecting error on console as "Resuming" [6] us-test is on OCP v4.8.17 and reflecting error on console as "Resuming" I will be attaching the screenshot of the console as well for your reference. Now Cu has concerns as: ~~~ [a] Everything is OK with local-cluster, status Ready and Distribution version OpenShift 4.8.17 with Upgrade available are the expected values. [b] The other clusters are supposed to be showing the same Ready and 4.8.17 values as local-cluster but as you can see their Status is incorrect at Resuming, their Distribution version is correct but we are not seeing the expected Upgrade available link. ~~~ Now should I suggest the workaround mentioned in your update or there is a change in action? Let me know if more data is required from the Cu side. Regards, Utkarsh Wagh
B2Gsync Hi Utkarsh, No, the workaround of deleting the ClusterCurator resources is only applicable if the UI is showing an upgrade in progress when that is not the case. Since the Status of these clusters is "Resuming", the available updates will not appear in the UI. They are only offered if the cluster has "Ready" status. There are 3 questions I have: [1] Does the UI status match what is shown in the ClusterDeployment resources? (If so, we should transfer this to the server foundation squad for investigation.) [2] Did the customer hibernate these clusters and then resume them, or is the "Resuming" status unexpected? [3] Are the clusters with "Resuming" status actually still resuming, or can the customer access them? Thanks
Hello Team, Following are the answers to the above questions. [1] Does the UI status match what is shown in the ClusterDeployment resources? -> Yes, the UI and ClusterDeployment resource status match. [us-acm@aws-us-bastion ~]$ oc get clusterdeployment -A NAMESPACE NAME PLATFORM REGION CLUSTERTYPE INSTALLED INFRAID VERSION POWERSTATE AGE eu-dev eu-dev aws eu-west-1 true eu-dev-wtw7x 4.7.19 Resuming 159d eu-prod eu-prod aws eu-west-1 true eu-prod-hbpkw 4.7.19 Resuming 154d us-dev us-dev aws us-east-1 true us-dev-chxj5 4.7.19 Resuming 181d us-prod us-prod aws us-east-1 true us-prod-w2d54 4.7.19 Resuming 175d us-test us-test aws us-east-1 true us-test-kmrwc 4.7.19 Resuming 194d [us-acm@aws-us-bastion ~]$ [2] Have you hibernated these clusters before and then resume them, or is the "Resuming" status unexpected? -> We have not hibernated any of our clusters before, "Resuming" status is unexpected. [3] Are the clusters with "Resuming" status actually still resuming, or can you be able to access them? -> No clusters are resuming, I can access all of them. Let me know if more information is required from Cu's side. Regards, Utkarsh Wagh
G2Bsync 988891350 comment KevinFCormier Wed, 08 Dec 2021 15:01:32 UTC G2Bsync Utkarsh, can you please provide the full ClusterDeployment YAML?
Utkarsh, please help to provide the full Clusterdeployment Yaml in eu-dev namespace and the hive-controllers pod log in hive namespace.
@pmagotra Could you help confirm with customer follow https://bugzilla.redhat.com/show_bug.cgi?id=2005759#c20, if they change the certificates? or other action will change the certificates? If the secret `eu-dev-0-zpcgh-admin-kubeconfig` in eu-dev ns can connect to the cluster?
Hello Team, There is an update from the customer for the following: ~~~ [1] If you had changed the certificates? [2] or perform other actions that will change the certificates? [3] Can you check If the secret `eu-dev-0-zpcgh-admin-kubeconfig` in eu-dev ns can connect to the cluster? ~~~ According to the customer: [1] He followed Adding API server certificates to add our corporate-signed API certificates as part of standard security configurations on all clusters, below is a sample configuration on eu-dev cluster. ~~~ [eu-dev@aws-us-bastion ~]$ oc describe secret -n openshift-config bi-api-certificate Name: bi-api-certificate Namespace: openshift-config Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 8422 bytes tls.key: 1704 bytes [eu-dev@aws-us-bastion ~]$ oc describe apiserver Name: cluster Namespace: Labels: <none> Annotations: include.release.openshift.io/self-managed-high-availability: true include.release.openshift.io/single-node-developer: true oauth-apiserver.openshift.io/secure-token-storage: true release.openshift.io/create-only: true API Version: config.openshift.io/v1 Kind: APIServer Metadata: Creation Timestamp: 2021-07-01T09:07:49Z Generation: 4 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:include.release.openshift.io/self-managed-high-availability: f:include.release.openshift.io/single-node-developer: f:oauth-apiserver.openshift.io/secure-token-storage: f:release.openshift.io/create-only: f:spec: .: f:audit: .: f:profile: Manager: cluster-version-operator Operation: Update Time: 2021-07-01T09:07:49Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:additionalCORSAllowedOrigins: Manager: OpenAPI-Generator Operation: Update Time: 2021-07-14T09:50:10Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:encryption: .: f:type: f:servingCerts: .: f:namedCertificates: Manager: config-policy-controller Operation: Update Time: 2021-07-27T09:23:32Z Resource Version: 61339482 UID: 732a405d-9ea9-449e-8afe-9a58b977d947 Spec: Additional CORS Allowed Origins: (?i)//migration-openshift-migration\.apps\.eu-dev\.ocp\.aws\.boehringer\.com(:|\z) //127.0.0.1(:|$) //localhost(:|$) Audit: Profile: Default Encryption: Type: aescbc Serving Certs: Named Certificates: Names: api.eu-dev.ocp.aws.boehringer.com Serving Certificate: Name: bi-api-certificate Events: <none> [eu-dev@aws-us-bastion ~]$ ~~~ [2] He confirms eu-dev/eu-dev-0-zpcgh-admin-kubeconfig is not working and have a query ***how can we update this to allow for our own certificates?*** ~~~ [root@aws-us-bastion ~]# oc --kubeconfig=eu-dev.kubeconfig version Client Version: 4.8.17 Unable to connect to the server: x509: certificate signed by unknown authority [root@aws-us-bastion ~]# ~~~ Do let me know if more information is required from Cu's side. Regards, Utkarsh Wagh
@jdiaz The customer has changed the certificates https://bugzilla.redhat.com/show_bug.cgi?id=2005759#c22, Is there a detail doc about how to add the customer certificates in clusterdeployment? and for current customer, could you give a steps to make the clusterdeployment running? Thank you.
There should be two options to get hive to trust the remote cluster's certificates. 1) Let hive apply the new certificates to the managed cluster: This involves populating the ClusterDeployment's spec.certificateBundles[] list https://github.com/openshift/hive/blob/master/apis/hive/v1/clusterdeployment_types.go#L638-L653 . Give the certificate bundle containing the certs for the 'api' route a name, and create a Secret containing the certificate information (keep 'generate' set to 'false' as setting 'generate' to 'true' would have no effect as hive doesn't support generating certificates today). Then you update the ClusterDeployment's spec.controlPlaneConfig.servingCertificates.default https://github.com/openshift/hive/blob/master/apis/hive/v1/clusterdeployment_types.go#L615-L619 to reference the name of the certificate bundle added under spec.certificateBundles[]. Hive will apply the certificates defined in the certificateBundle Secret for the 'api.<cluster>.domain.com' route, and will use the Certificate Authority information in that Secret when connecting to the cluster. 2) Tell hive that it should trust additional Certificate Authorities for all connections: This involves configuring the HiveConfig .spec.additionalCertificateAuthoritiesSecretRef[] https://github.com/openshift/hive/blob/master/apis/hive/v1/hiveconfig_types.go#L29-L34 to tell hive about additional Certificate Authorities that hive should trust when making connections.
Hello Team, I have an update from the CU side. They have followed [2] as mentioned above and have the following query: ~~~ We have decided to use [2] since all our clusters are similarly signed, we created the following secret and updated hiveconfig as documented but still not able to get it to work. Perhaps you can share more details on the exact steps to configure it? Thank you. ~~~ ~~~ [us-acm@aws-us-bastion ~]$ oc describe secret -n hive bi-ca Name: bi-ca Namespace: hive Labels: <none> Annotations: <none> Type: Opaque Data ==== bundle.cer: 5364 bytes [us-acm@aws-us-bastion ~]$ oc describe hiveconfig Name: hive Namespace: Labels: installer.name=multiclusterhub installer.namespace=open-cluster-management Annotations: <none> API Version: hive.openshift.io/v1 Kind: HiveConfig Metadata: Creation Timestamp: 2021-05-20T04:28:31Z Generation: 2 Managed Fields: API Version: hive.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:labels: .: f:installer.name: f:installer.namespace: f:spec: .: f:failedProvisionConfig: .: f:skipGatherLogs: Manager: multiclusterhub-operator Operation: Update Time: 2021-05-20T04:28:31Z API Version: hive.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:additionalCertificateAuthoritiesSecretRef: Manager: kubectl-edit Operation: Update Time: 2021-12-23T03:39:10Z API Version: hive.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:argoCDConfig: .: f:enabled: f:backup: .: f:velero: f:serviceProviderCredentialsConfig: f:status: f:aggregatorClientCAHash: f:conditions: f:configApplied: f:observedGeneration: Manager: hive-operator Operation: Update Time: 2021-12-23T03:39:11Z Resource Version: 648198222 UID: dc2ab216-f5f0-4e61-a493-8c5e4d70c1d4 Spec: Additional Certificate Authorities Secret Ref: Name: bi-ca Failed Provision Config: Skip Gather Logs: true Status: Aggregator Client CA Hash: 38aee507acba7dd44dfb030e2d8711d9 Conditions: Last Probe Time: 2021-09-22T03:50:30Z Last Transition Time: 2021-09-22T03:50:30Z Message: Hive is deployed successfully Reason: DeploymentSuccess Status: True Type: Ready Config Applied: true Observed Generation: 2 Events: <none> [us-acm@aws-us-bastion ~]$ ~~~ Do let me know if more data is required from the CU side. Regards, Utkarsh Wagh
The certs in the Secret should be stored in the key named 'ca.crt' (not 'bundle.cer').
have to clean up the resources in open-cluster-management-agent-addon project manually since the status of cluster is unknown. 1. check if there are failed jobs in open-cluster-management-agent-addon on the managed cluster. if yes, force delete them. oc get jobs -n open-cluster-management-agent-addon oc delete jobs xxx -n open-cluster-management-agent-addon 2. remove all finalizers of the deployments in the open-cluster-management-agent-addon project on the managed cluster manually. 3. force delete these appliedmanifestworks.work.open-cluster-management.io CRs. oc delete appliedmanifestworks.work.open-cluster-management.io \ xxxx-<cluster name>-klusterlet-addon-appmgr xxxx-<cluster name>-klusterlet-addon-certpolicyctrl xxxx-<cluster name>-klusterlet-addon-iampolicyctrl xxxx-<cluster name>-klusterlet-addon-policyctrl xxxx-<cluster name>-klusterlet-addon-workmgr xxxx-<cluster name>-klusterlet-addon-operator 4. force delete the agent CRs. remove the finalizers manually if they have during the deletion. oc delete applicationmanagers.agent.open-cluster-management.io klusterlet-addon-appmgr -n open-cluster-management-agent-addon oc delete certpolicycontrollers.agent.open-cluster-management.io klusterlet-addon-certpolicyctrl -n open-cluster-management-agent-addon oc delete iampolicycontrollers.agent.open-cluster-management.io klusterlet-addon-iampolicyctrl -n open-cluster-management-agent-addon oc delete policycontrollers.agent.open-cluster-management.io klusterlet-addon-policyctrl -n open-cluster-management-agent-addon oc delete searchcollectors.agent.open-cluster-management.io klusterlet-addon-search -n open-cluster-management-agent-addon oc delete workmanagers.agent.open-cluster-management.io klusterlet-addon-workmgr -n open-cluster-management-agent-addon
Hello Team, There is an update from the CU side for the above. No failed jobs or deployments in open-cluster-management-agent-addon and appliedmanifestworks.work.open-cluster-management.io CRs, only the agent CRs exist on the managed clusters. Please let us know how to force delete the agent CRs (and remove the finalizers manually if they have). ~~~ [root@aws-us-bastion ~]# om - get jobs -n open-cluster-management-agent-addon [[ us-acm ]] No resources found in open-cluster-management-agent-addon namespace. [[ us-test ]] No resources found in open-cluster-management-agent-addon namespace. [[ us-dev ]] No resources found in open-cluster-management-agent-addon namespace. [[ us-prod ]] No resources found in open-cluster-management-agent-addon namespace. [[ eu-dev ]] No resources found in open-cluster-management-agent-addon namespace. [[ eu-prod ]] No resources found in open-cluster-management-agent-addon namespace. [root@aws-us-bastion ~]# om - get deployments -n open-cluster-management-agent-addon [[ us-acm ]] NAME READY UP-TO-DATE AVAILABLE AGE klusterlet-addon-appmgr 1/1 1 1 230d klusterlet-addon-certpolicyctrl 1/1 1 1 230d klusterlet-addon-iampolicyctrl 1/1 1 1 230d klusterlet-addon-operator 1/1 1 1 230d klusterlet-addon-policyctrl-config-policy 1/1 1 1 230d klusterlet-addon-policyctrl-framework 1/1 1 1 230d klusterlet-addon-workmgr 1/1 1 1 230d [[ us-test ]] No resources found in open-cluster-management-agent-addon namespace. [[ us-dev ]] No resources found in open-cluster-management-agent-addon namespace. [[ us-prod ]] No resources found in open-cluster-management-agent-addon namespace. [[ eu-dev ]] No resources found in open-cluster-management-agent-addon namespace. [[ eu-prod ]] No resources found in open-cluster-management-agent-addon namespace. [root@aws-us-bastion ~]# om - get appliedmanifestworks.work.open-cluster-management.io -n open-cluster-management-agent-addon [[ us-acm ]] NAME AGE 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet 232d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-appmgr 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-certpolicyctrl 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-crds 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-iampolicyctrl 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-operator 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-policyctrl 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-addon-workmgr 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-klusterlet-crds 230d 50afaaf31ceafe61ee70473433780c8f3b36a5046257ddab1dbfeb9ee1c5d3d9-local-cluster-observability 91d [[ us-test ]] error: the server doesn't have a resource type "appliedmanifestworks" [[ us-dev ]] error: the server doesn't have a resource type "appliedmanifestworks" [[ us-prod ]] error: the server doesn't have a resource type "appliedmanifestworks" [[ eu-dev ]] error: the server doesn't have a resource type "appliedmanifestworks" [[ eu-prod ]] error: the server doesn't have a resource type "appliedmanifestworks" [root@aws-us-bastion ~]# om - get applicationmanagers.agent.open-cluster-management.io klusterlet-addon-appmgr -n open-cluster-management-agent-addon [[ us-acm ]] NAME AGE klusterlet-addon-appmgr 230d [[ us-test ]] NAME AGE klusterlet-addon-appmgr 223d [[ us-dev ]] NAME AGE klusterlet-addon-appmgr 211d [[ us-prod ]] NAME AGE klusterlet-addon-appmgr 204d [[ eu-dev ]] NAME AGE klusterlet-addon-appmgr 188d [[ eu-prod ]] NAME AGE klusterlet-addon-appmgr 183d [root@aws-us-bastion ~]# om - get certpolicycontrollers.agent.open-cluster-management.io klusterlet-addon-certpolicyctrl -n open-cluster-management-agent-addon [[ us-acm ]] NAME AGE klusterlet-addon-certpolicyctrl 230d [[ us-test ]] NAME AGE klusterlet-addon-certpolicyctrl 223d [[ us-dev ]] NAME AGE klusterlet-addon-certpolicyctrl 211d [[ us-prod ]] NAME AGE klusterlet-addon-certpolicyctrl 204d [[ eu-dev ]] NAME AGE klusterlet-addon-certpolicyctrl 188d [[ eu-prod ]] NAME AGE klusterlet-addon-certpolicyctrl 183d [root@aws-us-bastion ~]# om - get iampolicycontrollers.agent.open-cluster-management.io klusterlet-addon-iampolicyctrl -n open-cluster-management-agent-addon [[ us-acm ]] NAME AGE klusterlet-addon-iampolicyctrl 230d [[ us-test ]] NAME AGE klusterlet-addon-iampolicyctrl 223d [[ us-dev ]] NAME AGE klusterlet-addon-iampolicyctrl 211d [[ us-prod ]] NAME AGE klusterlet-addon-iampolicyctrl 204d [[ eu-dev ]] NAME AGE klusterlet-addon-iampolicyctrl 188d [[ eu-prod ]] NAME AGE klusterlet-addon-iampolicyctrl 183d [root@aws-us-bastion ~]# om - get policycontrollers.agent.open-cluster-management.io klusterlet-addon-policyctrl -n open-cluster-management-agent-addon [[ us-acm ]] NAME AGE klusterlet-addon-policyctrl 230d [[ us-test ]] NAME AGE klusterlet-addon-policyctrl 223d [[ us-dev ]] NAME AGE klusterlet-addon-policyctrl 211d [[ us-prod ]] NAME AGE klusterlet-addon-policyctrl 204d [[ eu-dev ]] NAME AGE klusterlet-addon-policyctrl 188d [[ eu-prod ]] NAME AGE klusterlet-addon-policyctrl 183d [root@aws-us-bastion ~]# om - get searchcollectors.agent.open-cluster-management.io klusterlet-addon-search -n open-cluster-management-agent-addon [[ us-acm ]] Error from server (NotFound): searchcollectors.agent.open-cluster-management.io "klusterlet-addon-search" not found [[ us-test ]] NAME AGE klusterlet-addon-search 223d [[ us-dev ]] NAME AGE klusterlet-addon-search 211d [[ us-prod ]] NAME AGE klusterlet-addon-search 204d [[ eu-dev ]] NAME AGE klusterlet-addon-search 188d [[ eu-prod ]] NAME AGE klusterlet-addon-search 183d [root@aws-us-bastion ~]# om - get workmanagers.agent.open-cluster-management.io klusterlet-addon-workmgr -n open-cluster-management-agent-addon [[ us-acm ]] om - get workmanagers.agent.open-cluster-management.io klusterlet-addon-workmgr -n open-cluster-management-agent-addon NAME AGE klusterlet-addon-workmgr 230d [[ us-test ]] NAME AGE klusterlet-addon-workmgr 223d [[ us-dev ]] NAME AGE klusterlet-addon-workmgr 211d [[ us-prod ]] NAME AGE klusterlet-addon-workmgr 204d [[ eu-dev ]] NAME AGE klusterlet-addon-workmgr 188d [[ eu-prod ]] NAME AGE klusterlet-addon-workmgr 183d [root@aws-us-bastion ~]# ~~~ Let me know if more information is required from CU's side. Regards,
it looks like the work agents are not working except the one on the hub cluster. login to the managed cluster, delete the agent CRs on the managed cluster (not on the hub cluster) oc delete applicationmanagers.agent.open-cluster-management.io klusterlet-addon-appmgr -n open-cluster-management-agent-addon oc delete certpolicycontrollers.agent.open-cluster-management.io klusterlet-addon-certpolicyctrl -n open-cluster-management-agent-addon oc delete iampolicycontrollers.agent.open-cluster-management.io klusterlet-addon-iampolicyctrl -n open-cluster-management-agent-addon oc delete policycontrollers.agent.open-cluster-management.io klusterlet-addon-policyctrl -n open-cluster-management-agent-addon oc delete searchcollectors.agent.open-cluster-management.io klusterlet-addon-search -n open-cluster-management-agent-addon oc delete workmanagers.agent.open-cluster-management.io klusterlet-addon-workmgr -n open-cluster-management-agent-addon if there are some other resources which prevent the project open-cluster-management-agent-addon from being deleted, delete them manually too.
Can the steps appended above helps to force delete the agent CRs, and is there still any problems? Thanks!
Hello Team, ~~~ Can the steps append above helps to force delete the agent CRs, and are there still any problems? Thanks! ~~~ Yes, the CU has updated that the issue is resolved and I have archived the case. Thank you for your help. Regards, Utkarsh Wagh
Glad to know, I'll close the issue then and feel free to reopen in case of any further problems. Thank you!