Description of problem: IPI cluster bootstrap fails sporadically because of repeated deletion/recreation of the openshift-etcd/host-etcd-2 service/endpoint pair. The timing of these deletions/recreations can cause failures during bootstrap and initial etcd rollout depending on when the events happen. David recently introduced a change[1] which causes the problem to surface more clearly. Here are some examples of what the errors look like from the installer perspective: https://search.apps.build01.ci.devcluster.openshift.com/?search=endpoints+%22host-etcd-2%22+not+found&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job https://search.apps.build01.ci.devcluster.openshift.com/?search=ClusterMemberController_Error&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job level=error msg="Cluster operator etcd Degraded is True with BootstrapTeardown_Error::ClusterMemberController_Error::EtcdMemberIPMigrator_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::NodeInstaller_InstallerPodFailed::StaticPods_Error: NodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: static pod of revision 2 has been installed, but is not ready while new revision 3 is pending\nStaticPodsDegraded: pod/etcd-master-0.ostest.test.metalkube.org container \"etcd\" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-0.ostest.test.metalkube.org_openshift-etcd(ff1db3d42cee82e3767349164792ccd7)\nStaticPodsDegraded: pod/etcd-master-0.ostest.test.metalkube.org container \"etcd\" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-0.ostest.test.metalkube.org_openshift-etcd(ff1db3d42cee82e3767349164792ccd7)\nStaticPodsDegraded: pod/etcd-master-1.ostest.test.metalkube.org container \"etcd\" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-1.ostest.test.metalkube.org_openshift-etcd(183c06bbf79b259aa2d43252e03ddfea)\nStaticPodsDegraded: pod/etcd-master-1.ostest.test.metalkube.org container \"etcd\" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-1.ostest.test.metalkube.org_openshift-etcd(183c06bbf79b259aa2d43252e03ddfea)\nStaticPodsDegraded: pods \"etcd-master-2.ostest.test.metalkube.org\" not found\nEtcdMembersControllerDegraded: context deadline exceeded\nBootstrapTeardownDegraded: context deadline exceeded\nEtcdMemberIPMigratorDegraded: context deadline exceeded\nClusterMemberControllerDegraded: context deadline exceeded" May 18 12:57:08.566: FAIL: Some cluster operators never became ready: etcd (Degraded=True BootstrapTeardown_Error::ClusterMemberController_Error::EnvVarController_Error::EtcdMemberIPMigrator_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::HostEndpoints2_ErrorUpdatingHostEndpoints2: EtcdMembersControllerDegraded: endpoints "host-etcd-2" not found [1] https://github.com/openshift/cluster-etcd-operator/pull/346 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
*** Bug 1836915 has been marked as a duplicate of this bug. ***
Run some IPI installation, have not hit this issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409