Bug 2039339

Summary:	cluster-ingress-operator should report Unupgradeable if user has modified the aws resources annotations
Product:	OpenShift Container Platform	Reporter:	Arjun Naik <anaik>
Component:	Networking	Assignee:	Arjun Naik <anaik>
Networking sub component:	router	QA Contact:	Arvind iyengar <aiyengar>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	aos-bugs, chdeshpa, cwawak, hongli, mjoseph, mmasters, wking
Version:	4.10
Target Milestone:	---
Target Release:	4.10.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The cluster-ingress-operator did not managed the aws resources annotation once the LoadBalancer type Service was created. Users could update the annotation value. Consequence: If the tags in Infrastructure were changed the new values could not be written to the annotation without overwriting the value set the user. Fix: The operator now warns the user that the corresponding IngressController is degraded when the annotation value is out of sync. Result: In subsequent versions of OCP the annotations value can be safely updated by the cluster-ingress-operator based on external sources.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-12 04:40:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2058699

Description Arjun Naik 2022-01-11 14:17:57 UTC

Description of problem: The cio creates a service of type "LoadBalancer" for every IngressController resource. The user can subsequently modify the annotation "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" to specify additional tags on the AWS loadbalancer. When this is the case the operator should report the IngressController as Upgradeable=False and also the ClusterOperator status condition of Upgradeable=False. 


OpenShift release version:


Cluster Platform:


How reproducible:


Steps to Reproduce (in detail):
1.
2.
3.


Actual results:


Expected results:


Impact of the problem:


Additional info:



** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 1 Miciah Dashiel Butler Masters 2022-01-11 17:00:23 UTC

Setting blocker- as this isn't a regression or upgrade blocker.  

Arjun, I notice you have opened https://github.com/openshift/cluster-ingress-operator/pull/693 for this BZ, so I am assigning the BZ to you.  We'll need to discuss the mechanics of getting the change in the branch in which we need it.

Comment 5 Arvind iyengar 2022-01-24 06:05:57 UTC

Verified in "4.10.0-0.nightly-2022-01-22-102609" release version. With this payload, it is observed that with loadbalancer with "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" service level annotation, the ingress operator and the CO resource logs the warning about the presence of the annotation and marks upgradeable=false state:
------
oc get clusterversion  
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-22-102609   True        False         135m    Cluster version is 4.10.0-0.nightly-2022-01-22-102609

oc -n openshift-ingress annotate service/router-internalapps service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags='test-cluster-qe'
service/router-internalapps annotated

oc -n openshift-ingress get service/router-internalapps -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: test-cluster-qe <-----
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "4"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    traffic-policy.network.alpha.openshift.io/local-with-fallback: ""
  creationTimestamp: "2022-01-24T05:26:09Z"

check the ingresscontroller and the related CO:
oc -n openshift-ingress-operator get ingresscontroller internalapps -o yaml 
  - lastTransitionTime: "2022-01-24T05:26:47Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2022-01-24T05:48:14Z"
    message: 'One or more managed resources are not upgradeable: load balancer service <------
      has been modified; changes must be reverted before upgrading: '
    reason: OperandsNotUpgradeable
    status: "False"   <------
    type: Upgradeable <------
  domain: internalapps.pdhamdhe41024.qe.devcluster.openshift.com
  endpointPublishingStrategy:
    loadBalancer:
      providerParameters:
        aws:
          type: NLB
        type: AWS
      scope: External
    type: LoadBalancerService
    

oc get co ingress -o yaml
  - lastTransitionTime: "2022-01-24T03:41:38Z"
    message: The "default" ingress controller reports Degraded=False.
    reason: IngressNotDegraded
    status: "False" 
    type: Degraded
  - lastTransitionTime: "2022-01-24T05:48:15Z"
    message: 'Some ingresscontrollers are not upgradeable: ingresscontroller "internalapps" <------
      is not upgradeable: OperandsNotUpgradeable: One or more managed resources are
      not upgradeable: load balancer service has been modified; changes must be reverted
      before upgrading: '
    reason: IngressControllersNotUpgradeable
    status: "False" <------
    type: Upgradeable <------
  extension: null
------

Comment 8 errata-xmlrpc 2022-03-12 04:40:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056

Comment 11 Red Hat Bugzilla 2023-09-18 04:30:04 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days