Bug 2097737

Summary: IngressControllersNotUpgradeable: load balancer service has been modified; changes must be reverted before upgrading
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: NetworkingAssignee: Miciah Dashiel Butler Masters <mmasters>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED NOTABUG Docs Contact:
Severity: urgent    
Priority: high CC: aos-bugs, mmasters, wking
Version: 4.9   
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-16 15:54:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2097736    
Bug Blocks: 2097738    

Description OpenShift BugZilla Robot 2022-06-16 12:40:34 UTC 
+++ This bug was initially created as a clone of Bug #2097736 +++

+++ This bug was initially created as a clone of Bug #2097735 +++

+++ This bug was initially created as a clone of Bug #2097555 +++

Description of problem:

The cluster-ingress-operator is causing 4.9 -> 4.10 CI upgrade tests to fail:

{  fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Jun 15 18:22:34.283: Some cluster operators are not ready: ingress (Upgradeable=False IngressControllersNotUpgradeable: Some ingresscontrollers are not upgradeable: ingresscontroller "default" is not upgradeable: OperandsNotUpgradeable: One or more managed resources are not upgradeable: load balancer service has been modified; changes must be reverted before upgrading: )}


OpenShift release version: 4.9


Cluster Platform: GCP, OVN


How reproducible: Unclear, but most recent nightly GCP OVN upgrade tests are failing due to this bug, even with three retries per nightly run.


Steps to Reproduce (in detail):
1. Run OCP 4.9 to 4.10 nightly upgrade test


Actual results: OCP cluster upgrade fails


Expected results: OCP cluster upgrade fails


Impact of the problem: This is currently blocking releases in the 4.10.z stream.


Additional info:

Examples of failing runs:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1537128423896387584

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-ovn-upgrade/1536843529286848512