Bug 1836927 - Cluster bootstrap fails with etcd rollout errors
Summary: Cluster bootstrap fails with etcd rollout errors
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Dan Mace
QA Contact: ge liu
URL:
Whiteboard:
: 1836915 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-18 13:54 UTC by Dan Mace
Modified: 2020-07-13 17:39 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:39:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-etcd-operator pull 354 0 None closed Bug 1836927: Replace etcd endpoint representation with configmap 2021-02-19 05:44:33 UTC
Github openshift cluster-kube-apiserver-operator pull 859 0 None closed Bug 1836927: Prefer the new etcd endpoints configmap for storage URL discovery 2021-02-19 05:44:33 UTC
Github openshift cluster-openshift-apiserver-operator pull 364 0 None closed Bug 1836927: Prefer etcd endpoints configmap for storage URL discovery 2021-02-19 05:44:34 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:39:51 UTC

Internal Links: 1821589 1931033

Description Dan Mace 2020-05-18 13:54:18 UTC
Description of problem:

IPI cluster bootstrap fails sporadically because of repeated deletion/recreation of the openshift-etcd/host-etcd-2 service/endpoint pair. The timing of these deletions/recreations can cause failures during bootstrap and initial etcd rollout depending on when the events happen.

David recently introduced a change[1] which causes the problem to surface more clearly.

Here are some examples of what the errors look like from the installer perspective:


https://search.apps.build01.ci.devcluster.openshift.com/?search=endpoints+%22host-etcd-2%22+not+found&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job

https://search.apps.build01.ci.devcluster.openshift.com/?search=ClusterMemberController_Error&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job


level=error msg="Cluster operator etcd Degraded is True with BootstrapTeardown_Error::ClusterMemberController_Error::EtcdMemberIPMigrator_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::NodeInstaller_InstallerPodFailed::StaticPods_Error: NodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: static pod of revision 2 has been installed, but is not ready while new revision 3 is pending\nStaticPodsDegraded: pod/etcd-master-0.ostest.test.metalkube.org container \"etcd\" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-0.ostest.test.metalkube.org_openshift-etcd(ff1db3d42cee82e3767349164792ccd7)\nStaticPodsDegraded: pod/etcd-master-0.ostest.test.metalkube.org container \"etcd\" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-0.ostest.test.metalkube.org_openshift-etcd(ff1db3d42cee82e3767349164792ccd7)\nStaticPodsDegraded: pod/etcd-master-1.ostest.test.metalkube.org container \"etcd\" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-1.ostest.test.metalkube.org_openshift-etcd(183c06bbf79b259aa2d43252e03ddfea)\nStaticPodsDegraded: pod/etcd-master-1.ostest.test.metalkube.org container \"etcd\" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-1.ostest.test.metalkube.org_openshift-etcd(183c06bbf79b259aa2d43252e03ddfea)\nStaticPodsDegraded: pods \"etcd-master-2.ostest.test.metalkube.org\" not found\nEtcdMembersControllerDegraded: context deadline exceeded\nBootstrapTeardownDegraded: context deadline exceeded\nEtcdMemberIPMigratorDegraded: context deadline exceeded\nClusterMemberControllerDegraded: context deadline exceeded"

May 18 12:57:08.566: FAIL: Some cluster operators never became ready: etcd (Degraded=True BootstrapTeardown_Error::ClusterMemberController_Error::EnvVarController_Error::EtcdMemberIPMigrator_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::HostEndpoints2_ErrorUpdatingHostEndpoints2: EtcdMembersControllerDegraded: endpoints "host-etcd-2" not found


[1] https://github.com/openshift/cluster-etcd-operator/pull/346

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Michal Fojtik 2020-05-19 12:08:36 UTC
*** Bug 1836915 has been marked as a duplicate of this bug. ***

Comment 4 ge liu 2020-06-22 07:43:17 UTC
Run some IPI installation, have not hit this issue.

Comment 5 errata-xmlrpc 2020-07-13 17:39:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.