The etcd client used by the API server should retry certain types of errors. The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72. This is already implemented by the etcd client just wasn't "enabled" in the API server The fix seems to be real compare [1] to how it was just a few days ago [2] [1] https://search.ci.openshift.org/?search=etcdserver%3A+leader+changed&maxAge=24h&context=1&type=junit&name=periodic.*4.10.*upgrade&excludeName=4.9%7C4.8%7Csingle-node&maxMatches=5&maxBytes=20971520&groupBy=job [2] https://search.ci.openshift.org/?search=etcdserver%3A+leader+changed&maxAge=48h&context=1&type=junit&name=periodic.*4.10.*upgrade&excludeName=4.9%7C4.8%7Csingle-node&maxMatches=5&maxBytes=20971520&groupBy=job
Compared the results between 4.9 and 4.10 with PR fix. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-09-25-094414 True False 79m Cluster version is 4.9.0-0.nightly-2021-09-25-094414 $ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}') $ for node in $masters; do echo $node;oc debug no/$node -- chroot /host bash -c "grep -ir 'etcdserver: leader changed' /var/log/ | grep -v debug";done | grep kube-apiserver | wc -l ... 12 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2021-09-23-210724 True False 12m Cluster version is 4.10.0-0.nightly-2021-09-23-210724 $ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}') $ for node in $masters; do echo $node;oc debug no/$node -- chroot /host bash -c "grep -ir 'etcdserver: leader changed' /var/log/ | grep -v debug";done | grep kube-apiserver | wc -l ... /var/log/kube-apiserver/termination.log:{"level":"warn","ts":"2021-09-26T11:11:24.026Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00315e380/#initially=[https://10.0.0.2:2379;https://10.0.0.4:2379;https://10.0.0.5:2379;https://localhost:2379]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: leader changed"} /var/log/kube-apiserver/termination.log:{"level":"warn","ts":"2021-09-26T11:11:24.026Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f3500/#initially=[https://10.0.0.2:2379;https://10.0.0.4:2379;https://10.0.0.5:2379;https://localhost:2379]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: leader changed"} Above warning logs from etcd-client, not from by the API server, so on 4.10, won't see certain errors of etcd, move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056