Cause: When an ingresscontroller is configured to use a LoadBalancer-type service, the ingress operator creates and manages this service, and if the operator detects that the user has modified an annotation that the operator manages on this service, then the operator sets the ingress clusteroperator's "Upgradeable" status condition "False" to block upgrades. However, the operator's check of the service's annotations had a logic error that could falsely report that the user had modified the annotations when service had no annotations.
Consequence: The ingress operator could erroneously set the ingress clusteroperator's "Upgradeable" status condition to "False", blocking upgrades, if the service had no annotations.
Fix: The logic that checks the service's annotations was fixed to handle empty annotations correctly.
Result: The ingress operator should no longer erroneously block upgrades.
Description of problem:
The cluster-ingress-operator is causing 4.9 -> 4.10 CI upgrade tests to fail:
{ fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Jun 15 18:22:34.283: Some cluster operators are not ready: ingress (Upgradeable=False IngressControllersNotUpgradeable: Some ingresscontrollers are not upgradeable: ingresscontroller "default" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: load balancer service has been modified; changes must be reverted before upgrading: )}
OpenShift release version: 4.9
Cluster Platform: GCP, OVN
How reproducible: Unclear, but most recent nightly GCP OVN upgrade tests are failing due to this bug, even with three retries per nightly run.
Steps to Reproduce (in detail):
1. Run OCP 4.9 to 4.10 nightly upgrade test
Actual results: OCP cluster upgrade fails
Expected results: OCP cluster upgrade fails
Impact of the problem: This is currently blocking releases in the 4.10.z stream.
Additional info:
Examples of failing runs:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1537128423896387584https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1536843529286848512
Comment 2Miciah Dashiel Butler Masters
2022-06-16 22:01:42 UTC
The linked PR fixes a logic error, which exists in the release-4.11 branch as well as the release-4.10 and release-4.9 branches. The logic error isn't known to cause any problems in in 4.11, but it does cause a problem in 4.9 on Alibaba, Azure, and GCP clusters when using OVN with a public (not internal) load balancer, so we need to fix the problem in 4.11 and backport to 4.9.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2022:5069
Description of problem: The cluster-ingress-operator is causing 4.9 -> 4.10 CI upgrade tests to fail: { fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Jun 15 18:22:34.283: Some cluster operators are not ready: ingress (Upgradeable=False IngressControllersNotUpgradeable: Some ingresscontrollers are not upgradeable: ingresscontroller "default" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: load balancer service has been modified; changes must be reverted before upgrading: )} OpenShift release version: 4.9 Cluster Platform: GCP, OVN How reproducible: Unclear, but most recent nightly GCP OVN upgrade tests are failing due to this bug, even with three retries per nightly run. Steps to Reproduce (in detail): 1. Run OCP 4.9 to 4.10 nightly upgrade test Actual results: OCP cluster upgrade fails Expected results: OCP cluster upgrade fails Impact of the problem: This is currently blocking releases in the 4.10.z stream. Additional info: Examples of failing runs: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1537128423896387584 https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1536843529286848512