Bug 2006717

This bug's PR is dev-approved and not yet merged, so I'm following issue DPTP-660 to do the pre-merge verifying for QE pre-merge verification goal of issue OCPQE-815 by using the bot to launch a cluster with the open PR.  Here is the verification steps:

Compared the results between 4.9 and 4.9 with PR fix.

-4.9 without PR fix.
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-25-094414   True        False         79m     Cluster version is 4.9.0-0.nightly-2021-09-25-094414

$ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')
$ for node in $masters; do echo $node;oc debug no/$node -- chroot /host bash -c "grep -ir 'etcdserver: leader changed' /var/log/ | grep -v debug";done | grep kube-apiserver | wc -l
...
12

Snipped logs, 
/var/log/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-85f45dfb8c-qwhh5_51cdbf88-9631-40c9-afd2-381f72bcc81d/kube-apiserver-operator/1.log:2021-09-27T02:34:45.038539976+00:00 stderr F E0927 02:34:45.037714       1 base_controller.go:272] CertRotationController reconciliation failed: etcdserver: leader changed
/var/log/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-85f45dfb8c-qwhh5_51cdbf88-9631-40c9-afd2-381f72bcc81d/kube-apiserver-operator/1.log:2021-09-27T02:34:45.040668123+00:00 stderr F E0927 02:34:45.039861       1 base_controller.go:272] auditPolicyController reconciliation failed: etcdserver: leader changed
...
/var/log/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-85f45dfb8c-qwhh5_51cdbf88-9631-40c9-afd2-381f72bcc81d/kube-apiserver-operator/1.log:2021-09-27T02:34:54.301287946+00:00 stderr F E0927 02:34:54.301254       1 base_controller.go:272] StaticResourceController reconciliation failed: ["manifests/installer-sa.yaml" (string): etcdserver: leader changed, Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again]
/var/log/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-85f45dfb8c-qwhh5_51cdbf88-9631-40c9-afd2-381f72bcc81d/kube-apiserver-operator/1.log:2021-09-27T02:34:56.857449602+00:00 stderr F E0927 02:34:56.853811       1 base_controller.go:272] StaticResourceController reconciliation failed: ["assets/kube-apiserver/svc.yaml" (string): etcdserver: leader changed, Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again]


- 4.9 with PR fix.
$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.ci.test-2021-09-27-021440-ci-ln-8ftswck-latest   True        False         94m     Cluster version is 4.9.0-0.ci.test-2021-09-27-021440-ci-ln-8ftswck-latest

$ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')
$ for node in $masters; do echo $node;oc debug no/$node -- chroot /host bash -c "grep -ir 'etcdserver: leader changed' /var/log/ | grep -v debug";done | grep kube-apiserver | wc -l
...
/var/log/kube-apiserver/termination.log:{"level":"warn","ts":"2021-09-27T02:37:33.612Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ac9500/#initially=[https://10.0.0.3:2379;https://10.0.0.4:2379;https://10.0.0.5:2379;https://localhost:2379]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: leader changed"}

Above warning logs from etcd-client, not from by the API server, after applied the PR fix, won't see certain errors of etcd. So the bug is pre-merge verified. After the PR gets merged, the bug will be moved to VERIFIED by the bot automatically or, if not working, by me manually.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.5 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4005