Bug 1836927

Description of problem:

IPI cluster bootstrap fails sporadically because of repeated deletion/recreation of the openshift-etcd/host-etcd-2 service/endpoint pair. The timing of these deletions/recreations can cause failures during bootstrap and initial etcd rollout depending on when the events happen.

David recently introduced a change[1] which causes the problem to surface more clearly.

Here are some examples of what the errors look like from the installer perspective:


https://search.apps.build01.ci.devcluster.openshift.com/?search=endpoints+%22host-etcd-2%22+not+found&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job

https://search.apps.build01.ci.devcluster.openshift.com/?search=ClusterMemberController_Error&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job


level=error msg="Cluster operator etcd Degraded is True with BootstrapTeardown_Error::ClusterMemberController_Error::EtcdMemberIPMigrator_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::NodeInstaller_InstallerPodFailed::StaticPods_Error: NodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: static pod of revision 2 has been installed, but is not ready while new revision 3 is pending\nStaticPodsDegraded: pod/etcd-master-0.ostest.test.metalkube.org container \"etcd\" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-0.ostest.test.metalkube.org_openshift-etcd(ff1db3d42cee82e3767349164792ccd7)\nStaticPodsDegraded: pod/etcd-master-0.ostest.test.metalkube.org container \"etcd\" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-0.ostest.test.metalkube.org_openshift-etcd(ff1db3d42cee82e3767349164792ccd7)\nStaticPodsDegraded: pod/etcd-master-1.ostest.test.metalkube.org container \"etcd\" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-1.ostest.test.metalkube.org_openshift-etcd(183c06bbf79b259aa2d43252e03ddfea)\nStaticPodsDegraded: pod/etcd-master-1.ostest.test.metalkube.org container \"etcd\" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-master-1.ostest.test.metalkube.org_openshift-etcd(183c06bbf79b259aa2d43252e03ddfea)\nStaticPodsDegraded: pods \"etcd-master-2.ostest.test.metalkube.org\" not found\nEtcdMembersControllerDegraded: context deadline exceeded\nBootstrapTeardownDegraded: context deadline exceeded\nEtcdMemberIPMigratorDegraded: context deadline exceeded\nClusterMemberControllerDegraded: context deadline exceeded"

May 18 12:57:08.566: FAIL: Some cluster operators never became ready: etcd (Degraded=True BootstrapTeardown_Error::ClusterMemberController_Error::EnvVarController_Error::EtcdMemberIPMigrator_Error::EtcdMembersController_ErrorUpdatingReportEtcdMembers::HostEndpoints2_ErrorUpdatingHostEndpoints2: EtcdMembersControllerDegraded: endpoints "host-etcd-2" not found


[1] https://github.com/openshift/cluster-etcd-operator/pull/346

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: