Bug 2039339

Summary: cluster-ingress-operator should report Unupgradeable if user has modified the aws resources annotations
Product: OpenShift Container Platform Reporter: Arjun Naik <anaik>
Component: NetworkingAssignee: Arjun Naik <anaik>
Networking sub component: router QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, chdeshpa, cwawak, hongli, mjoseph, mmasters, wking
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: All   
OS: All   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The cluster-ingress-operator did not managed the aws resources annotation once the LoadBalancer type Service was created. Users could update the annotation value. Consequence: If the tags in Infrastructure were changed the new values could not be written to the annotation without overwriting the value set the user. Fix: The operator now warns the user that the corresponding IngressController is degraded when the annotation value is out of sync. Result: In subsequent versions of OCP the annotations value can be safely updated by the cluster-ingress-operator based on external sources.
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-12 04:40:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 2058699    

Description Arjun Naik 2022-01-11 14:17:57 UTC
Description of problem: The cio creates a service of type "LoadBalancer" for every IngressController resource. The user can subsequently modify the annotation "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" to specify additional tags on the AWS loadbalancer. When this is the case the operator should report the IngressController as Upgradeable=False and also the ClusterOperator status condition of Upgradeable=False. 

OpenShift release version:

Cluster Platform:

How reproducible:

Steps to Reproduce (in detail):

Actual results:

Expected results:

Impact of the problem:

Additional info:

** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 1 Miciah Dashiel Butler Masters 2022-01-11 17:00:23 UTC
Setting blocker- as this isn't a regression or upgrade blocker.  

Arjun, I notice you have opened https://github.com/openshift/cluster-ingress-operator/pull/693 for this BZ, so I am assigning the BZ to you.  We'll need to discuss the mechanics of getting the change in the branch in which we need it.

Comment 5 Arvind iyengar 2022-01-24 06:05:57 UTC
Verified in "4.10.0-0.nightly-2022-01-22-102609" release version. With this payload, it is observed that with loadbalancer with "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" service level annotation, the ingress operator and the CO resource logs the warning about the presence of the annotation and marks upgradeable=false state:
oc get clusterversion  
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-22-102609   True        False         135m    Cluster version is 4.10.0-0.nightly-2022-01-22-102609

oc -n openshift-ingress annotate service/router-internalapps service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags='test-cluster-qe'
service/router-internalapps annotated

oc -n openshift-ingress get service/router-internalapps -o yaml
apiVersion: v1
kind: Service
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: test-cluster-qe <-----
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "4"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    traffic-policy.network.alpha.openshift.io/local-with-fallback: ""
  creationTimestamp: "2022-01-24T05:26:09Z"

check the ingresscontroller and the related CO:
oc -n openshift-ingress-operator get ingresscontroller internalapps -o yaml 
  - lastTransitionTime: "2022-01-24T05:26:47Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2022-01-24T05:48:14Z"
    message: 'One or more managed resources are not upgradeable: load balancer service <------
      has been modified; changes must be reverted before upgrading: '
    reason: OperandsNotUpgradeable
    status: "False"   <------
    type: Upgradeable <------
  domain: internalapps.pdhamdhe41024.qe.devcluster.openshift.com
          type: NLB
        type: AWS
      scope: External
    type: LoadBalancerService

oc get co ingress -o yaml
  - lastTransitionTime: "2022-01-24T03:41:38Z"
    message: The "default" ingress controller reports Degraded=False.
    reason: IngressNotDegraded
    status: "False" 
    type: Degraded
  - lastTransitionTime: "2022-01-24T05:48:15Z"
    message: 'Some ingresscontrollers are not upgradeable: ingresscontroller "internalapps" <------
      is not upgradeable: OperandsNotUpgradeable: One or more managed resources are
      not upgradeable: load balancer service has been modified; changes must be reverted
      before upgrading: '
    reason: IngressControllersNotUpgradeable
    status: "False" <------
    type: Upgradeable <------
  extension: null

Comment 8 errata-xmlrpc 2022-03-12 04:40:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Comment 11 Red Hat Bugzilla 2023-09-18 04:30:04 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days