Bug 2097555 - IngressControllersNotUpgradeable: load balancer service has been modified; changes must be reverted before upgrading
Summary: IngressControllersNotUpgradeable: load balancer service has been modified; ch...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 4.11.0
Assignee: Miciah Dashiel Butler Masters
QA Contact: Hongan Li
URL:
Whiteboard:
: 2104135 (view as bug list)
Depends On:
Blocks: 2097735
TreeView+ depends on / blocked
 
Reported: 2022-06-16 00:22 UTC by Joe Lanford
Modified: 2022-10-12 03:50 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When an ingresscontroller is configured to use a LoadBalancer-type service, the ingress operator creates and manages this service, and if the operator detects that the user has modified an annotation that the operator manages on this service, then the operator sets the ingress clusteroperator's "Upgradeable" status condition "False" to block upgrades. However, the operator's check of the service's annotations had a logic error that could falsely report that the user had modified the annotations when service had no annotations. Consequence: The ingress operator could erroneously set the ingress clusteroperator's "Upgradeable" status condition to "False", blocking upgrades, if the service had no annotations. Fix: The logic that checks the service's annotations was fixed to handle empty annotations correctly. Result: The ingress operator should no longer erroneously block upgrades.
Clone Of:
Environment:
Last Closed: 2022-08-10 11:18:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-ingress-operator pull 783 0 None Merged Bug 2097555: Fix loadBalancerServiceAnnotationsChanged check and update 2022-06-17 13:17:41 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:18:38 UTC

Description Joe Lanford 2022-06-16 00:22:29 UTC 
Description of problem:

The cluster-ingress-operator is causing 4.9 -> 4.10 CI upgrade tests to fail:

{  fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Jun 15 18:22:34.283: Some cluster operators are not ready: ingress (Upgradeable=False IngressControllersNotUpgradeable: Some ingresscontrollers are not upgradeable: ingresscontroller "default" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: load balancer service has been modified; changes must be reverted before upgrading: )}


OpenShift release version: 4.9


Cluster Platform: GCP, OVN


How reproducible: Unclear, but most recent nightly GCP OVN upgrade tests are failing due to this bug, even with three retries per nightly run.


Steps to Reproduce (in detail):
1. Run OCP 4.9 to 4.10 nightly upgrade test


Actual results: OCP cluster upgrade fails


Expected results: OCP cluster upgrade fails


Impact of the problem: This is currently blocking releases in the 4.10.z stream.


Additional info:

Examples of failing runs:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1537128423896387584

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1536843529286848512
Comment 2 Miciah Dashiel Butler Masters 2022-06-16 22:01:42 UTC 
The linked PR fixes a logic error, which exists in the release-4.11 branch as well as the release-4.10 and release-4.9 branches.  The logic error isn't known to cause any problems in in 4.11, but it does cause a problem in 4.9 on Alibaba, Azure, and GCP clusters when using OVN with a public (not internal) load balancer, so we need to fix the problem in 4.11 and backport to 4.9.
Comment 4 Hongan Li 2022-06-21 11:08:03 UTC 
verified with 4.11.0-0.ci-2022-06-20-211630 and no issues found.
Comment 6 W. Trevor King 2022-07-06 05:52:40 UTC 
*** Bug 2104135 has been marked as a duplicate of this bug. ***
Comment 7 errata-xmlrpc 2022-08-10 11:18:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069
Note You need to log in before you can comment on or make changes to this bug.