Operator upgrade kube-apiserver has begun failing frequently in CI, see: https://sippy.ci.openshift.org/sippy-ng/tests/4.10/analysis?test=Operator%20upgrade%20kube-apiserver Aggregated jobs on CI payloads appear to have caught a regression in this test, historically passing 100% of the time, now failing 20-30% of the time. A good sample prow job would be: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-azure-upgrade/1461619400548290560 Test failure looks as follows: Failed to upgrade kube-apiserver, operator was degraded (ValidatingAdmissionWebhookConfiguration_WebhookServiceConnectionError): ValidatingAdmissionWebhookConfigurationDegraded: vprovisioning.kb.io: dial tcp 172.30.203.253:443: connect: connection refused It is often accompanied by: operator conditions kube-apiserver expand_less 0s Operator degraded (ValidatingAdmissionWebhookConfiguration_WebhookServiceConnectionError): ValidatingAdmissionWebhookConfigurationDegraded: vprovisioning.kb.io: dial tcp 172.30.203.253:443: connect: connection refused The problem appears to have begun last night, somewhere around this CI release: https://amd64.ocp.releases.ci.openshift.org/releasestream/4.10.0-0.ci/release/4.10.0-0.ci-2021-11-19-045525 This payload did contain a kube apiserver operator change: cluster-kube-apiserver-operator set kube-apiserver degraded=true if a webhook service is missing or down #1245
That webhook comes from https://github.com/openshift/cluster-baremetal-operator/blob/7bfc0bfab5d1ed278c8f8c18fd01c1b2bd4c7157/provisioning/webhook.go.
Problem has likely been around for awhile, but new checks went in last night which caught the problem: https://github.com/openshift/cluster-kube-apiserver-operator/pull/1256 has been opened to revert the new checks while a proper fix is pursued. Reverting so we can get payloads flowing again, the checks look great, just need to solve this before they can go in.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056