Description of problem:

For managed clusters, upgrades from 4.9 to 4.10 got stuck on AWS clusters with below message:
~~~
ingress                                    4.10.0-rc.1   True        True          False      254d    ingresscontroller "default" is progressing: ScopeChanged: The IngressController scope was changed from "External" to "Internal".  To effectuate this change, you must delete the service: `oc -n openshift-ingress delete svc/router-default`; the service load-balancer will then be deprovisioned and a new one created.  This will most likely cause the new load-balancer to have a different host name and IP address from the old one's.  Alternatively, you can revert the change to the IngressController: `oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"loadBalancer":{"scope":"External"}}}}'.  
~~~

In order to reproduce the issue, went through BZ https://bugzilla.redhat.com/show_bug.cgi?id=2035193 and enhancement https://github.com/openshift/enhancements/blob/master/enhancements/ingress/mutable-publishing-scope.md#proposal to understand the working details better.

However, based on the enhancement its expected that the Ingress ClusterOperator would be stuck in Progressing state when the scope is changed from "External" to "Internal" before 4.9 to 4.10 upgrade of an AWS cluster. But what was observed is that the above message to manually delete the router-default service occurred only when the router-default service has the below annotation:
~~~
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
~~~

By default, for 4.8+ clusters, the value of this annotation is the following:
~~~
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
~~~

Thus it was noticed that for old clusters which were installed at version 4.7 or earlier, the clusters maintained the annotation value to be "0.0.0.0/0" and were stuck in upgrade from 4.9 to 4.10. But a cluster installed at version 4.9 had this annotation value of "true" and managed to finish the upgrade without getting stuck. This bugzilla is raised to track this difference in behaviour for long running clusters (4.7 or earlier) and recently installed clusters (4.8+).

OpenShift release version:
4.10

Cluster Platform:
AWS

How reproducible:
Always

Steps to Reproduce (in detail):
Scenario 1:
1. Create AWS cluster at version 4.9
2. Change LB scope to "Internal" initiate upgrade to 4.10 (router-default service annotation should be `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`. Use command `oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"type":"LoadBalancerService","loadBalancer":{"scope":"Internal"}}}}'`
3. Initiate Upgrade to 4.10. It's expected to upgrade fine without getting stuck.

Scenario 2:
1. Create AWS cluster at version 4.9
2. Change LB scope to "Internal" initiate upgrade to 4.10 (router-default service annotation should be `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`. Use command `oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"type":"LoadBalancerService","loadBalancer":{"scope":"Internal"}}}}'`.
3. Edit router-default service and change the annotation from `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` to `service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0`. 
4. Initiate Upgrade to 4.10. It's expected to getting stuck in upgrade by throwing the above message to manually delete the router-default service.


Actual results:
- Upgrades from 4.9 to 4.10 on AWS clusters behave differently based on the value of `service.beta.kubernetes.io/aws-load-balancer-internal` annotation in router-default service.

Expected results:
- Upgrades from 4.9 to 4.10 on AWS clusters should behave the same based on the value of `service.beta.kubernetes.io/aws-load-balancer-internal` annotation in router-default service. This is considering the fact that the default value of this annotation was changed from "0.0.0.0/0" to "true" from OCP 4.8.


Impact of the problem:
- Impact is high as it would block lot of 4.9 to 4.10 upgrades for the managed clusters fleet. Manual intervention to delete router-default service is not scalable.
- Considering https://github.com/openshift/enhancements/blob/master/enhancements/ingress/mutable-publishing-scope.md#as-the-provider-of-a-managed-service-i-want-to-automate-changing-an-ingresscontrollers-scope, the workaround exists to annotate the default ingresscontroller to automatically delete the router-default service. This was tested and it worked as expected. 


Additional info:

Tests were mostly done on managed OSD clusters which also has https://github.com/openshift/cloud-ingress-operator deployed. However, this operator does not manage the `service.beta.kubernetes.io/aws-load-balancer-internal` annotation.

Following tests were done:
1. Create 4.9 AWS cluster and upgrade to 4.10 after changing scope to internal via OCM console (annotation value "true" in 4.9) = UPGRADE PASSED
2. Create 4.7 AWS cluster and upgrade to 4.10 after changing scope to internal via OCM console (annotation value "0.0.0.0/0" in 4.7-4.9) = UPGRADE STUCK
3. Create 4.9 AWS cluster and upgrade to 4.10 after changing scope to internal via manual patch of ingresscontrollers/default (annotation value "true" in 4.9) = UPGRADE PASSED
4. Create 4.9 AWS cluster and upgrade to 4.10 after changing scope to internal and manually changing annotation value to "0.0.0.0/0" = UPGRADE STUCK
5. Create 4.9 AWS cluster and upgrade to 4.10 after changing scope to internal and adding ingress.operator.openshift.io/auto-delete-load-balancer= annotation to default ingresscontroller = UPGRADE PASSED