Bug 2060309
| Summary: | multiclusterhub stuck in installing on "ManagedClusterConditionAvailable" [intermittent] | ||
|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | nshidlin <nshidlin> |
| Component: | Cluster Lifecycle | Assignee: | Jian Qiu <jqiu> |
| Status: | CLOSED ERRATA | QA Contact: | Hui Chen <huichen> |
| Severity: | high | Docs Contact: | Christopher Dawson <cdawson> |
| Priority: | unspecified | ||
| Version: | rhacm-2.5 | CC: | dhuynh, jqiu, njean, sasha, smiron, tanya21owens, yuhe, zyin |
| Target Milestone: | --- | Keywords: | AutomationBlocker, TestBlocker |
| Target Release: | rhacm-2.5 | Flags: | bot-tracker-sync:
rhacm-2.5+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-09 02:09:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
reproducing with:
OCP: 4.10.0-0.nightly-2022-03-16-041611
2.5.0-DOWNSTREAM-2022-03-14-18-18-07
oc get mch
NAME STATUS AGE
multiclusterhub Installing 3m45s
oc get mch
NAME STATUS AGE
multiclusterhub Installing 3m45s
[root@sealusa34 ~]# oc describe mch multiclusterhub
Name: multiclusterhub
Namespace: rhacm
Labels: <none>
Annotations: <none>
API Version: operator.open-cluster-management.io/v1
Kind: MultiClusterHub
Metadata:
Creation Timestamp: 2022-03-17T02:04:23Z
Finalizers:
finalizer.operator.open-cluster-management.io
Generation: 3
Managed Fields:
API Version: operator.open-cluster-management.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:imagePullSecret:
Manager: kubectl-create
Operation: Update
Time: 2022-03-17T02:04:23Z
API Version: operator.open-cluster-management.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizer.operator.open-cluster-management.io":
f:spec:
f:availabilityConfig:
f:components:
f:enableClusterBackup:
f:ingress:
.:
f:sslCiphers:
f:separateCertificateManagement:
Manager: multiclusterhub-operator
Operation: Update
Time: 2022-03-17T02:04:32Z
API Version: operator.open-cluster-management.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:components:
.:
f:application-chart-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:cluster-lifecycle-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:console-chart-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:grc-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:local-cluster:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:management-ingress-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:multicluster-engine:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:multicluster-engine-csv:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:multicluster-engine-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:multiclusterhub-repo:
.:
f:lastTransitionTime:
f:reason:
f:status:
f:type:
f:policyreport-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:search-prod-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:volsync-addon-controller-sub:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:conditions:
f:desiredVersion:
f:phase:
Manager: multiclusterhub-operator
Operation: Update
Subresource: status
Time: 2022-03-17T02:06:15Z
Resource Version: 302497
UID: 87ac28ab-d5bb-4cd8-acff-1258c8d7334f
Spec:
Availability Config: High
Components:
Enabled: true
Name: multiclusterhub-repo
Enabled: true
Name: search
Enabled: true
Name: management-ingress
Enabled: true
Name: application-ui
Enabled: true
Name: console
Enabled: true
Name: insights
Enabled: true
Name: grc
Enabled: true
Name: cluster-lifecycle
Enabled: true
Name: volsync
Enabled: true
Name: multicluster-engine
Enabled: false
Name: cluster-backup
Enabled: false
Name: cluster-proxy-addon
Enable Cluster Backup: false
Image Pull Secret: pull-secret
Ingress:
Ssl Ciphers:
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
Separate Certificate Management: false
Status:
Components:
Application - Chart - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Cluster - Lifecycle - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Console - Chart - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Grc - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Local - Cluster:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Management - Ingress - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Multicluster - Engine:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Multicluster - Engine - Csv:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Multicluster - Engine - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Multiclusterhub - Repo:
Last Transition Time: 2022-03-17T02:05:00Z
Reason: MinimumReplicasAvailable
Status: True
Type: Available
Policyreport - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Search - Prod - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Volsync - Addon - Controller - Sub:
Last Transition Time: 2022-03-17T02:02:51Z
Message: No conditions available
Reason: No conditions available
Status: Unknown
Type: Unknown
Conditions:
Last Transition Time: 2022-03-17T02:04:43Z
Last Update Time: 2022-03-17T02:04:54Z
Message: Created new resource
Reason: NewResourceCreated
Status: True
Type: Progressing
Desired Version: 2.5.0
Phase: Installing
Events: <none>
multiclusterhub-operator log shows:
2022-03-17T02:28:55.522Z ERROR controller.multiclusterhub Reconciler error {"reconciler group": "operator.open-cluster-management.io", "reconciler kind": "MultiClusterHub", "name": "multiclusterhub", "namespace": "rhacm", "error": "CSV not located for subscription: multicluster-engine/multicluster-engine"}
the second issue is different from the original one. all components are Available in mch status in the original issue. we just finished dev sprints today. I think the codes changes are frequent in the past week. could you try to install the latest snapshot to verify. reproduced with 2.5.0-DOWNSTREAM-2022-03-17-03-36-41 Hi @jqiu , can we get an outlook for resolution for this bug. Thx. G2Bsync 1085295727 comment zhiweiyin318 Fri, 01 Apr 2022 01:21:54 UTC G2Bsync did we meet the issue in the latest snapshot? I think the issue is because the code changes frequently in the dev sprint as we discussed in slack. Did not reproduce this with multiple redeploys of RHACM 2.5.0-DOWNSTREAM-2022-04-11-09-21-38 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:4956 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days This Bugzilla entry describes an intermittent problem where https://www.kaisermemberservices.org the Multiclusterhub (MCH) installation process stalls, remaining "stuck in installing" because it's waiting for the required "ManagedClusterConditionAvailable" status. |
Description of the problem: Attempting to install RHACM 2.5.0-DOWNSTREAM-2022-03-01-18-33-27 mutliclusterhub is stuck in installing. looking at the components see: oc get mch multiclusterhub -n rhacm -o json | jq '.status.components | map(select(.status == "Unknown"))' [ { "lastTransitionTime": "2022-03-02T11:09:52Z", "message": "Registration agent stopped updating its lease.", "reason": "ManagedClusterLeaseUpdateStopped", "status": "Unknown", "type": "ManagedClusterConditionAvailable" } ] Further klusterlet CRD is not created: oc get klusterlets klusterlet -o yaml error: the server doesn't have a resource type "klusterlets" Operator snapshot version: 2.5.0-DOWNSTREAM-2022-03-01-18-33-27 OCP version: 4.10.0-0.nightly-2022-02-26-230022 Steps to reproduce: 1. Attempt to install RHACM 2.5.0-DOWNSTREAM-2022-03-01-18-33-27 Actual results: mutliclusterhub is stuck in installing Expected results: mutliclusterhub to complete installation Additional info: deleting local-cluster-import secret to trigger reconcile allows mutliclusterhub to complete installation