Description of problem: Upgrade from any version to 4.8.0-0.nightly-2021-05-09-085627 are being blocked due to the error “QuorumGuardController_Error QuorumGuardControllerDegraded: the server could not find the requested resource”. Have hit this four times in upgrade ci env yesterday. Also says “Unable to apply 4.8.0-0.nightly-2021-05-09-085627: wait has exceeded 40 minutes for these operators: etcd” Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-05-09-085627 How reproducible: Always Steps to Reproduce: 1. Install 4.7 cluster 2. Run upgrade to 4.8 nightly build 4.8.0-0.nightly-2021-05-09-085627 3. Actual results: Upgrade to 4.8 is not successful and always hit error as described above Expected results: Upgrade to 4.8 should be successful Additional info: [2021-05-09T11:26:00.008Z] 2021-05-09T08:26:34Z [etcd] -->>Degraded True<<-- QuorumGuardController_Error QuorumGuardControllerDegraded: the server could not find the requested resource [2021-05-09T11:26:00.008Z] 2021-05-09T08:28:40Z [etcd] Progressing False AsExpected NodeInstallerProgressing: 3 nodes are at revision 4 [2021-05-09T11:26:00.008Z] EtcdMembersAvailable: 3 members are available [2021-05-09T11:26:00.008Z] EtcdMembersProgressing: No unstarted etcd members found [2021-05-09T11:26:00.008Z] [2021-05-09T11:26:00.008Z] -------------------------- [2021-05-09T11:26:00.008Z] [2021-05-09T11:26:00.946Z] Name: etcd [2021-05-09T11:26:00.946Z] Namespace: [2021-05-09T11:26:00.946Z] Labels: <none> [2021-05-09T11:26:00.946Z] Annotations: exclude.release.openshift.io/internal-openshift-hosted: true [2021-05-09T11:26:00.946Z] include.release.openshift.io/self-managed-high-availability: true [2021-05-09T11:26:00.946Z] include.release.openshift.io/single-node-developer: true [2021-05-09T11:26:00.946Z] API Version: config.openshift.io/v1 [2021-05-09T11:26:00.946Z] Kind: ClusterOperator [2021-05-09T11:26:00.946Z] Metadata: [2021-05-09T11:26:00.946Z] Creation Timestamp: 2021-05-09T06:48:14Z [2021-05-09T11:26:00.946Z] Generation: 1 [2021-05-09T11:26:00.946Z] Managed Fields: [2021-05-09T11:26:00.946Z] API Version: config.openshift.io/v1 [2021-05-09T11:26:00.946Z] Fields Type: FieldsV1 [2021-05-09T11:26:00.946Z] fieldsV1: [2021-05-09T11:26:00.946Z] f:metadata: [2021-05-09T11:26:00.946Z] f:annotations: [2021-05-09T11:26:00.946Z] .: [2021-05-09T11:26:00.946Z] f:exclude.release.openshift.io/internal-openshift-hosted: [2021-05-09T11:26:00.946Z] f:include.release.openshift.io/self-managed-high-availability: [2021-05-09T11:26:00.946Z] f:include.release.openshift.io/single-node-developer: [2021-05-09T11:26:00.946Z] f:spec: [2021-05-09T11:26:00.946Z] f:status: [2021-05-09T11:26:00.946Z] .: [2021-05-09T11:26:00.946Z] f:extension: [2021-05-09T11:26:00.946Z] f:relatedObjects: [2021-05-09T11:26:00.946Z] Manager: cluster-version-operator [2021-05-09T11:26:00.946Z] Operation: Update [2021-05-09T11:26:00.946Z] Time: 2021-05-09T06:48:14Z [2021-05-09T11:26:00.946Z] API Version: config.openshift.io/v1 [2021-05-09T11:26:00.946Z] Fields Type: FieldsV1 [2021-05-09T11:26:00.946Z] fieldsV1: [2021-05-09T11:26:00.946Z] f:status: [2021-05-09T11:26:00.946Z] f:conditions: [2021-05-09T11:26:00.946Z] f:versions: [2021-05-09T11:26:00.946Z] Manager: cluster-etcd-operator [2021-05-09T11:26:00.946Z] Operation: Update [2021-05-09T11:26:00.946Z] Time: 2021-05-09T06:57:04Z [2021-05-09T11:26:00.946Z] Resource Version: 68500 [2021-05-09T11:26:00.946Z] Self Link: /apis/config.openshift.io/v1/clusteroperators/etcd [2021-05-09T11:26:00.947Z] UID: 1c662ab6-03bf-4168-a875-22414290bfe1 [2021-05-09T11:26:00.947Z] Spec: [2021-05-09T11:26:00.947Z] Status: [2021-05-09T11:26:00.947Z] Conditions: [2021-05-09T11:26:00.947Z] Last Transition Time: 2021-05-09T08:26:34Z [2021-05-09T11:26:00.947Z] Message: QuorumGuardControllerDegraded: the server could not find the requested resource [2021-05-09T11:26:00.947Z] Reason: QuorumGuardController_Error [2021-05-09T11:26:00.947Z] Status: True [2021-05-09T11:26:00.947Z] Type: Degraded [2021-05-09T11:26:00.947Z] Last Transition Time: 2021-05-09T08:28:40Z [2021-05-09T11:26:00.947Z] Message: NodeInstallerProgressing: 3 nodes are at revision 4 [2021-05-09T11:26:00.947Z] EtcdMembersProgressing: No unstarted etcd members found [2021-05-09T11:26:00.947Z] Reason: AsExpected [2021-05-09T11:26:00.947Z] Status: False [2021-05-09T11:26:00.947Z] Type: Progressing [2021-05-09T11:26:00.947Z] Last Transition Time: 2021-05-09T06:59:05Z [2021-05-09T11:26:00.947Z] Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4 [2021-05-09T11:26:00.947Z] EtcdMembersAvailable: 3 members are available [2021-05-09T11:26:00.947Z] Reason: AsExpected [2021-05-09T11:26:00.947Z] Status: True [2021-05-09T11:26:00.947Z] Type: Available [2021-05-09T11:26:00.947Z] Last Transition Time: 2021-05-09T06:57:04Z [2021-05-09T11:26:00.947Z] Message: All is well [2021-05-09T11:26:00.947Z] Reason: AsExpected [2021-05-09T11:26:00.947Z] Status: True [2021-05-09T11:26:00.947Z] Type: Upgradeable [2021-05-09T11:26:00.947Z] Extension: <nil> [2021-05-09T11:26:00.947Z] Related Objects: [2021-05-09T11:26:00.947Z] Group: operator.openshift.io [2021-05-09T11:26:00.947Z] Name: cluster [2021-05-09T11:26:00.947Z] Resource: etcds [2021-05-09T11:26:00.947Z] Group: [2021-05-09T11:26:00.947Z] Name: openshift-config [2021-05-09T11:26:00.947Z] Resource: namespaces [2021-05-09T11:26:00.947Z] Group: [2021-05-09T11:26:00.947Z] Name: openshift-config-managed [2021-05-09T11:26:00.947Z] Resource: namespaces [2021-05-09T11:26:00.947Z] Group: [2021-05-09T11:26:00.947Z] Name: openshift-etcd-operator [2021-05-09T11:26:00.947Z] Resource: namespaces [2021-05-09T11:26:00.947Z] Group: [2021-05-09T11:26:00.947Z] Name: openshift-etcd [2021-05-09T11:26:00.947Z] Resource: namespaces [2021-05-09T11:26:00.947Z] Versions: [2021-05-09T11:26:00.947Z] Name: operator [2021-05-09T11:26:00.947Z] Version: 4.8.0-0.nightly-2021-05-09-015732 [2021-05-09T11:26:00.947Z] Name: raw-internal [2021-05-09T11:26:00.947Z] Version: 4.8.0-0.nightly-2021-05-09-015732 [2021-05-09T11:26:00.947Z] Name: etcd [2021-05-09T11:26:00.947Z] Version: 4.8.0-0.nightly-2021-05-09-015732 [2021-05-09T11:26:00.947Z] Events: <none> [2021-05-09T11:26:00.947Z] [2021-05-09T11:26:00.947Z] ~~~~~~~~~~~~~~~~~~~~~~~ Must-gather logs: ========================= http://10.73.131.57:9000/openshift-must-gather/2021-05-10-06-52-56/must-gather.local.5993375849336813795.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=openshift%2F20210510%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210510T065309Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=ee416d5db3c7453f67f935973bf8cab35273f027892d4cff0eeaf93ca96c2fec
Marked the issue as urgent because QE upgrade CI on May 11th was all blocked by this issue.
Thanks for report, please retest with the latest nightly[1] this should have been resolved by reverting a change we made to PDB version[2]. Feel free to reopen if that does not work but as nightly promotion is unblocked I believe this issue has been fixed. [1] https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.8.0-0.nightly/release/4.8.0-0.nightly-2021-05-12-002851 [2] https://github.com/openshift/cluster-etcd-operator/pull/592
Hello sam, May i know why the bug is closed as Not a bug ? I see that the bug existed and there was a patch which merged into the code to resolve the issue. IMO, bug should be closed as duplicate if there is already an existing bug else we should be moving it to ON_QA for qe to verify it with the latest nightly. please correct me if my understanding is incorrect, thanks !!
> May i know why the bug is closed as Not a bug ? Fair it is no longer is a bug but we should let QE validate.
Verified. upgrade successfuly: 4.7.10-x86_64--> 4.8.0-0.nightly-2021-05-12-072240
(In reply to Sam Batschelet from comment #4) > > May i know why the bug is closed as Not a bug ? > > Fair it is no longer is a bug but we should let QE validate. Thanks !!
@melbeher, the bug status is VERIFIED, what's mean of it still open?
Sorry, I thought there is something to be done here :)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.8.49 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6308