This problem was fixed on 4.6, bug reproduced on OCP 4.7 4.7.0-0.nightly-2020-12-20-031835, $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-12-20-031835 True False 160m Error while reconciling 4.7.0-0.nightly-2020-12-20-031835: the cluster operator kube-apiserver is degraded $ oc get no NAME STATUS ROLES AGE VERSION juzhao-share-bbmw6-master-0 Ready master 3h13m v1.20.0+87544c5 juzhao-share-bbmw6-master-1 Ready master 3h13m v1.20.0+87544c5 juzhao-share-bbmw6-master-2 Ready master 3h13m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-bhprx Ready worker 179m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-fs2h5 Ready worker 175m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-p48wh Ready worker 178m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-qxktd Ready worker 176m v1.20.0+87544c5 $ oc get co/kube-apiserver NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE kube-apiserver 4.7.0-0.nightly-2020-12-20-031835 True True True 3h16m $ oc describe co/kube-apiserver Name: kube-apiserver Namespace: ... Spec: Status: Conditions: Last Transition Time: 2020-12-21T03:35:21Z Message: NodeInstallerDegraded: 1 nodes are failing on revision 8: NodeInstallerDegraded: no detailed termination message, see `oc get -n "openshift-kube-apiserver" pods/"installer-8-juzhao-share-bbmw6-master-1" -oyaml` Reason: NodeInstaller_InstallerPodFailed Status: True Type: Degraded Last Transition Time: 2020-12-21T03:32:06Z Message: NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8 Reason: NodeInstaller Status: True Type: Progressing Last Transition Time: 2020-12-21T03:02:45Z Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 8 Reason: AsExpected Status: True Type: Available Last Transition Time: 2020-12-21T02:59:28Z Message: All is well Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> ... $ oc get pods -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-juzhao-share-bbmw6-master-0 5/5 Running 0 3h17m apiserver=true,app=openshift-kube-apiserver,revision=6 kube-apiserver-juzhao-share-bbmw6-master-1 5/5 Running 0 3h4m apiserver=true,app=openshift-kube-apiserver,revision=6 kube-apiserver-juzhao-share-bbmw6-master-2 5/5 Running 0 3h22m apiserver=true,app=openshift-kube-apiserver,revision=6 $ oc logs -n openshift-kube-apiserver-operator kube-apiserver-operator-5c77c9c859-k684x | grep -n '1 nodes are failing on revision 8' 74:I1221 03:36:04.463569 1 status_controller.go:213] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2020-12-21T03:35:21Z","message":"NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled","reason":"NodeInstaller_InstallerPodFailed::StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-12-21T03:32:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-12-21T03:02:45Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-12-21T02:59:28Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} 85:I1221 03:36:04.614567 1 status_controller.go:213] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2020-12-21T03:35:21Z","message":"NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled","reason":"NodeInstaller_InstallerPodFailed::StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-12-21T03:32:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-12-21T03:02:45Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-12-21T02:59:28Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} 86:I1221 03:36:04.621298 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"d6219038-bd2e-4744-98ef-9cf9a42858e7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`" to "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled" 107:I1221 03:36:04.807779 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"d6219038-bd2e-4744-98ef-9cf9a42858e7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`" to "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled" ...
Tried the following workaround, cluster went back to normal. oc patch etcd/cluster --type=json -p '[ {"op": "replace", "path": "/spec/forceRedeploymentReason", "value": "forced test 1" } ]' oc patch kubeapiserver/cluster --type=json -p '[ {"op": "replace", "path": "/spec/forceRedeploymentReason", "value": "forced test 1" } ]'
I would need to see `kubectl get kubeapiservers -o yaml` before the workaround of comment 3. The one in the must-gather is on revision 97, not 8 where the issue happened.
Disregard comment 4. I mixed up must-gather.
The installer pod is evicted by machine config daemon: quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-5d0410a767aeccfc527c7455bb828f07d56ae2849605f8796fcaba51ff838f08/namespaces/openshift-machine-config-operator/pods/machine-config-daemon-fq8j5/machine-config-daemon/machine-config-daemon/logs/current.log 47:2020-12-21T03:33:18.068185011Z E1221 03:33:18.068115 14063 daemon.go:342] WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-4pjpc, openshift-controller-manager/controller-manager-s7g27, openshift-dns/dns-default-zclmp, openshift-image-registry/node-ca-gnbgk, openshift-machine-config-operator/machine-config-daemon-fq8j5, openshift-machine-config-operator/machine-config-server-xxsmr, openshift-manila-csi-driver/csi-nodeplugin-nfsplugin-8pvpv, openshift-manila-csi-driver/openstack-manila-csi-nodeplugin-pq5rr, openshift-monitoring/node-exporter-mmntv, openshift-multus/multus-admission-controller-skdrr, openshift-multus/multus-x4vbt, openshift-multus/network-metrics-daemon-f992j, openshift-network-diagnostics/network-check-target-qdtgc, openshift-ovn-kubernetes/ovnkube-master-z6ns5, openshift-ovn-kubernetes/ovnkube-node-nfsgr, openshift-ovn-kubernetes/ovs-node-7wwqv; deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: openshift-kube-apiserver/installer-8-juzhao-share-bbmw6-master-1 57:2020-12-21T03:33:18.077433978Z I1221 03:33:18.077397 14063 daemon.go:342] evicting pod openshift-kube-apiserver/installer-8-juzhao-share-bbmw6-master-1 134:2020-12-21T03:33:37.157026872Z I1221 03:33:37.156888 14063 daemon.go:328] Evicted pod openshift-kube-apiserver/installer-8-juzhao-share-bbmw6-master-1 This looks like the cause of the missed installer.
Checked back with MCO team about this eviction on reboot. This behaviour has not changed. So this BZ is not a blocker for 4.7 as far as we see. Nevertheless, work is ongoing to improve resilience to installer pod evictions (and other installer failures).
Today, When we upgraded from 4.6.0-0.nightly-2021-03-21-131139--> 4.7.0-0.nightly-2021-03-22-025559, hit the same issue again. cluster operator kube-apiserver error message: ... Spec: Status: Conditions: Last Transition Time: 2021-03-24T03:53:37Z Message: NodeInstallerDegraded: 1 nodes are failing on revision 11: NodeInstallerDegraded: no detailed termination message, see `oc get -n "openshift-kube-apiserver" pods/"installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal" -oyaml` Reason: NodeInstaller_InstallerPodFailed Status: True Type: Degraded Last Transition Time: 2021-03-24T03:51:13Z Message: NodeInstallerProgressing: 3 nodes are at revision 10; 0 nodes have achieved new revision 11 Reason: NodeInstaller Status: True Type: Progressing Last Transition Time: 2021-03-24T01:39:36Z Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 10; 0 nodes have achieved new revision 11 Reason: AsExpected Status: True Type: Available Last Transition Time: 2021-03-24T01:35:51Z Message: All is well Reason: AsExpected Status: True Type: Upgradeable must-gather.local.5989955761737491992/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-de5a26ae6ddea0ad63182c508ca5e730bdc14c52daffa5c42d9a718aa7720e9b/namespaces/openshift-machine-config-operator/pods # grep -n 'installer-11-yinzhou240913' machine-config-daemon-sldps/machine-config-daemon/machine-config-daemon/logs/current.log 37:2021-03-24T03:51:28.901402277Z E0324 03:51:28.901336 103262 daemon.go:344] WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-csi-drivers/gcp-pd-csi-driver-node-f6pp4, openshift-cluster-node-tuning-operator/tuned-64sx8, openshift-controller-manager/controller-manager-77n7g, openshift-dns/dns-default-tptbh, openshift-image-registry/node-ca-zf5mz, openshift-machine-config-operator/machine-config-daemon-sldps, openshift-machine-config-operator/machine-config-server-4jk7s, openshift-monitoring/node-exporter-kj9l8, openshift-multus/multus-admission-controller-wm6gb, openshift-multus/multus-lnbwb, openshift-multus/network-metrics-daemon-rthb2, openshift-network-diagnostics/network-check-target-gqblf, openshift-ovn-kubernetes/ovnkube-master-rfpvh, openshift-ovn-kubernetes/ovnkube-node-6h5c7, openshift-ovn-kubernetes/ovs-node-qp6vg; deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: openshift-kube-apiserver/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal 46:2021-03-24T03:51:28.941813425Z I0324 03:51:28.941788 103262 daemon.go:344] evicting pod openshift-kube-apiserver/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal 91:2021-03-24T03:51:46.256188435Z I0324 03:51:46.254499 103262 daemon.go:330] Evicted pod openshift-kube-apiserver/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal 106:2021-03-24T03:53:10.714342933Z I0324 03:53:10.714234 103262 daemon.go:344] evicting pod openshift-kube-controller-manager/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal 108:2021-03-24T03:53:10.775250978Z I0324 03:53:10.775195 103262 daemon.go:330] Evicted pod openshift-kube-controller-manager/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal In kubelet log, found PLEG error, $ grep 'PLEG is not healthy' * kubelet_service.log:Mar 24 02:49:53.776520 yinzhou240913-vfdzl-m-0.c.openshift-qe.internal hyperkube[1725]: E0324 02:49:53.776332 1725 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] kubelet_service.log:Mar 24 02:55:13.930773 yinzhou240913-vfdzl-m-1.c.openshift-qe.internal hyperkube[1712]: E0324 02:55:13.930693 1712 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] kubelet_service.log:Mar 24 02:59:43.970704 yinzhou240913-vfdzl-m-2.c.openshift-qe.internal hyperkube[1748]: E0324 02:59:43.969052 1748 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] kubelet_service.log:Mar 24 03:54:43.225651 yinzhou240913-vfdzl-m-1.c.openshift-qe.internal hyperkube[1747]: E0324 03:54:43.225600 1747 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] kubelet_service.log:Mar 24 04:00:55.054332 yinzhou240913-vfdzl-m-0.c.openshift-qe.internal hyperkube[1734]: E0324 04:00:55.054119 1734 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] kubelet_service.log:Mar 24 04:07:24.362134 yinzhou240913-vfdzl-m-2.c.openshift-qe.internal hyperkube[1756]: E0324 04:07:24.362105 1756 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Ke Wang added UpgradeBlocker on the 24th, which sucks this bug into the update-blocker lifecycle [1]. So we're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the ImpactStatementRequested label has been added to this bug. When responding, please remove ImpactStatementRequested and set the ImpactStatementProposed label. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? * example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet * example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? * example: Up to 2 minute disruption in edge routing * example: Up to 90 seconds of API downtime * example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? * example: Issue resolves itself after five minutes * example: Admin uses oc to fix things * example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? * example: No, it has always been like this we just never noticed * example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1 [1]: https://github.com/openshift/enhancements/pull/475
> Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? We have one upgrade profile Disconnected IPI on vSphere 7.0 with RHCOS & http_proxy from 4.6.z to 4.7.z, 100% hit this bug. > What is the impact? Is it serious enough to warrant blocking edges? Only impact vSphere. > How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? Not sure if it is serious impact, no one solution to mitigate now, will plan to have new upgrade test with latest 4.7.z payload with the same upgrade profile. > Is this a regression? Yes, it is cloned form bug 1858763 on 4.6.
No longer blocker issue.
Upgraded OCP vSphere disconnected cluster from 4.6.24 to 4.7.nightly，then to 4.8 nightly successfully. $ oc get infrastructures.config.openshift.io -o yaml | grep -A1 " platformSpec" platformSpec: type: VSphere $ oc get clusterversion -o json|jq ".items[0].status.history" [ { "completionTime": "2021-04-13T10:32:35Z", "image": "registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-222447", "startedTime": "2021-04-13T09:15:24Z", "state": "Completed", "verified": false, "version": "4.8.0-0.nightly-2021-04-09-222447" }, { "completionTime": "2021-04-13T08:29:09Z", "image": "registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-04-10-082109", "startedTime": "2021-04-13T07:02:37Z", "state": "Completed", "verified": false, "version": "4.7.0-0.nightly-2021-04-10-082109" }, { "completionTime": "2021-04-13T07:01:52Z", "image": "registry.ci.openshift.org/ocp/release@sha256:d422c5f6a531374dc03484bb0adaeb26b7b179dec055e15ecd6042cc750dd140", "startedTime": "2021-04-13T07:01:07Z", "state": "Completed", "verified": false, "version": "4.6.24" }, { "completionTime": "2021-04-13T07:01:07Z", "image": "registry.ci.openshift.org/ocp/release@sha256:a78b87ed93b759ceed7f62bc6c477ba6a3927d34d82a18f56fe2fa3147b2fd3e", "startedTime": "2021-04-13T06:54:22Z", "state": "Partial", "verified": false, "version": "4.7.0-0.nightly-2021-04-10-082109" }, { "completionTime": "2021-04-13T04:45:52Z", "image": "registry.ci.openshift.org/ocp/release@sha256:d422c5f6a531374dc03484bb0adaeb26b7b179dec055e15ecd6042cc750dd140", "startedTime": "2021-04-13T03:42:34Z", "state": "Completed", "verified": true, "version": "4.6.24" }, { "completionTime": "2021-04-13T03:12:29Z", "image": "quay.io/openshift-release-dev/ocp-release@sha256:5c8ab6c4a863f9ac077bc743579d9387bb7cd311a36c3197e609f8be63d17981", "startedTime": "2021-04-13T02:44:51Z", "state": "Completed", "verified": false, "version": "4.6.23" } ] $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-09-222447 True False 21m Cluster version is 4.8.0-0.nightly-2021-04-09-222447 $ oc get no NAME STATUS ROLES AGE VERSION kewang-134601-lnvpv-master-0 Ready master 8h v1.20.0+5f82cdb kewang-134601-lnvpv-master-1 Ready master 8h v1.20.0+5f82cdb kewang-134601-lnvpv-master-2 Ready master 8h v1.20.0+5f82cdb kewang-134601-lnvpv-worker-25n5k Ready worker 7h56m v1.20.0+5f82cdb kewang-134601-lnvpv-worker-bdjnz Ready worker 7h56m v1.20.0+5f82cdb $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.8.0-0.nightly-2021-04-09-222447 True False False 27m baremetal 4.8.0-0.nightly-2021-04-09-222447 True False False 3h26m cloud-credential 4.8.0-0.nightly-2021-04-09-222447 True False False 8h cluster-autoscaler 4.8.0-0.nightly-2021-04-09-222447 True False False 8h config-operator 4.8.0-0.nightly-2021-04-09-222447 True False False 8h console 4.8.0-0.nightly-2021-04-09-222447 True False False 34m csi-snapshot-controller 4.8.0-0.nightly-2021-04-09-222447 True False False 34m dns 4.8.0-0.nightly-2021-04-09-222447 True False False 8h etcd 4.8.0-0.nightly-2021-04-09-222447 True False False 8h image-registry 4.8.0-0.nightly-2021-04-09-222447 True False False 44m ingress 4.8.0-0.nightly-2021-04-09-222447 True False False 7h59m insights 4.8.0-0.nightly-2021-04-09-222447 True False False 8h kube-apiserver 4.8.0-0.nightly-2021-04-09-222447 True False False 8h kube-controller-manager 4.8.0-0.nightly-2021-04-09-222447 True False False 8h kube-scheduler 4.8.0-0.nightly-2021-04-09-222447 True False False 8h kube-storage-version-migrator 4.8.0-0.nightly-2021-04-09-222447 True False False 44m machine-api 4.8.0-0.nightly-2021-04-09-222447 True False False 8h machine-approver 4.8.0-0.nightly-2021-04-09-222447 True False False 8h machine-config 4.8.0-0.nightly-2021-04-09-222447 True False False 26m marketplace 4.8.0-0.nightly-2021-04-09-222447 True False False 41m monitoring 4.8.0-0.nightly-2021-04-09-222447 True False False 32m network 4.8.0-0.nightly-2021-04-09-222447 True False False 8h node-tuning 4.8.0-0.nightly-2021-04-09-222447 True False False 72m openshift-apiserver 4.8.0-0.nightly-2021-04-09-222447 True False False 27m openshift-controller-manager 4.8.0-0.nightly-2021-04-09-222447 True False False 3h22m openshift-samples 4.8.0-0.nightly-2021-04-09-222447 True False False 72m operator-lifecycle-manager 4.8.0-0.nightly-2021-04-09-222447 True False False 8h operator-lifecycle-manager-catalog 4.8.0-0.nightly-2021-04-09-222447 True False False 8h operator-lifecycle-manager-packageserver 4.8.0-0.nightly-2021-04-09-222447 True False False 34m service-ca 4.8.0-0.nightly-2021-04-09-222447 True False False 8h storage 4.8.0-0.nightly-2021-04-09-222447 True False False 34m All is fine, so move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438
Based on the impact statement in comment 12 we have not blocked update recommendations based on this bug. If folks bump into it more frequently, feel free to restore the UpgradeBlocker keyword (which was already removed after comment 13).