Bug 2016174
| Summary: | etcd pod on CrashLoopBackOff after master replacement procedure | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | Etcd | Assignee: | Sam Batschelet <sbatsche> |
| Status: | CLOSED ERRATA | QA Contact: | ge liu <geliu> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.9 | CC: | aharchin, htariq, jrosenta, mcornea, yprokule |
| Target Milestone: | --- | ||
| Target Release: | 4.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-10 21:02:40 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2003775 | ||
| Bug Blocks: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.9.6 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4119 |
Verified with replaced crashed master, # oc get node NAME STATUS ROLES AGE VERSION ip-10-0-149-187.us-east-2.compute.internal Ready worker 141m v1.22.0-rc.0+a44d0f0 ip-10-0-159-4.us-east-2.compute.internal Ready master 146m v1.22.0-rc.0+a44d0f0 ip-10-0-187-82.us-east-2.compute.internal Ready master 146m v1.22.0-rc.0+a44d0f0 ip-10-0-188-189.us-east-2.compute.internal Ready worker 140m v1.22.0-rc.0+a44d0f0 ip-10-0-205-206.us-east-2.compute.internal Ready worker 135m v1.22.0-rc.0+a44d0f0 ip-10-0-209-57.us-east-2.compute.internal Ready master 5m15s v1.22.0-rc.0+a44d0f0 then patch etcd cluster: # oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge etcd.operator.openshift.io/cluster patched # oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.9.0-0.nightly-2021-11-01-180550 True False False 13m baremetal 4.9.0-0.nightly-2021-11-01-180550 True False False 144m cloud-controller-manager 4.9.0-0.nightly-2021-11-01-180550 True False False 147m cloud-credential 4.9.0-0.nightly-2021-11-01-180550 True False False 147m cluster-autoscaler 4.9.0-0.nightly-2021-11-01-180550 True False False 144m config-operator 4.9.0-0.nightly-2021-11-01-180550 True False False 146m console 4.9.0-0.nightly-2021-11-01-180550 True False False 126m csi-snapshot-controller 4.9.0-0.nightly-2021-11-01-180550 True False False 145m dns 4.9.0-0.nightly-2021-11-01-180550 True False False 144m etcd 4.9.0-0.nightly-2021-11-01-180550 True True False 144m NodeInstallerProgressing: 2 nodes are at revision 5; 1 nodes are at revision 6 image-registry 4.9.0-0.nightly-2021-11-01-180550 True False False 132m ingress 4.9.0-0.nightly-2021-11-01-180550 True False False 130m insights 4.9.0-0.nightly-2021-11-01-180550 True False False 130m kube-apiserver 4.9.0-0.nightly-2021-11-01-180550 True True False 132m NodeInstallerProgressing: 1 nodes are at revision 8; 2 nodes are at revision 9 kube-controller-manager 4.9.0-0.nightly-2021-11-01-180550 True False False 144m kube-scheduler 4.9.0-0.nightly-2021-11-01-180550 True False False 143m kube-storage-version-migrator 4.9.0-0.nightly-2021-11-01-180550 True False False 145m ........................... .............. etcd member recovered quorum: sh-4.4# etcdctl member list -w table +------------------+---------+-------------------------------------------+--------------------------+--------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+-------------------------------------------+--------------------------+--------------------------+------------+ | 3820c8756d3e1144 | started | ip-10-0-187-82.us-east-2.compute.internal | https://10.0.187.82:2380 | https://10.0.187.82:2379 | false | | af71e9bd16b6458f | started | ip-10-0-209-57.us-east-2.compute.internal | https://10.0.209.57:2380 | https://10.0.209.57:2379 | false | | f83c9bc01cbbe3e7 | started | ip-10-0-159-4.us-east-2.compute.internal | https://10.0.159.4:2380 | https://10.0.159.4:2379 | false | +------------------+---------+-------------------------------------------+--------------------------+--------------------------+------------+