Bug 2039339 - cluster-ingress-operator should report Unupgradeable if user has modified the aws resources annotations [NEEDINFO]
Summary: cluster-ingress-operator should report Unupgradeable if user has modified the...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: All
OS: All
Target Milestone: ---
: 4.10.0
Assignee: Arjun Naik
QA Contact: Arvind iyengar
Depends On:
Blocks: 2058699
TreeView+ depends on / blocked
Reported: 2022-01-11 14:17 UTC by Arjun Naik
Modified: 2022-12-20 09:16 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The cluster-ingress-operator did not managed the aws resources annotation once the LoadBalancer type Service was created. Users could update the annotation value. Consequence: If the tags in Infrastructure were changed the new values could not be written to the annotation without overwriting the value set the user. Fix: The operator now warns the user that the corresponding IngressController is degraded when the annotation value is out of sync. Result: In subsequent versions of OCP the annotations value can be safely updated by the cluster-ingress-operator based on external sources.
Clone Of:
Last Closed: 2022-03-12 04:40:34 UTC
Target Upstream Version:
chdeshpa: needinfo? (anaik)
cwawak: needinfo? (anaik)

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-ingress-operator pull 693 0 None Merged Bug 2039339: Upgradeable Condition in Operator and IC status 2022-03-16 21:19:08 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:40:44 UTC

Description Arjun Naik 2022-01-11 14:17:57 UTC
Description of problem: The cio creates a service of type "LoadBalancer" for every IngressController resource. The user can subsequently modify the annotation "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" to specify additional tags on the AWS loadbalancer. When this is the case the operator should report the IngressController as Upgradeable=False and also the ClusterOperator status condition of Upgradeable=False. 

OpenShift release version:

Cluster Platform:

How reproducible:

Steps to Reproduce (in detail):

Actual results:

Expected results:

Impact of the problem:

Additional info:

** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 1 Miciah Dashiel Butler Masters 2022-01-11 17:00:23 UTC
Setting blocker- as this isn't a regression or upgrade blocker.  

Arjun, I notice you have opened https://github.com/openshift/cluster-ingress-operator/pull/693 for this BZ, so I am assigning the BZ to you.  We'll need to discuss the mechanics of getting the change in the branch in which we need it.

Comment 5 Arvind iyengar 2022-01-24 06:05:57 UTC
Verified in "4.10.0-0.nightly-2022-01-22-102609" release version. With this payload, it is observed that with loadbalancer with "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" service level annotation, the ingress operator and the CO resource logs the warning about the presence of the annotation and marks upgradeable=false state:
oc get clusterversion  
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-22-102609   True        False         135m    Cluster version is 4.10.0-0.nightly-2022-01-22-102609

oc -n openshift-ingress annotate service/router-internalapps service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags='test-cluster-qe'
service/router-internalapps annotated

oc -n openshift-ingress get service/router-internalapps -o yaml
apiVersion: v1
kind: Service
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: test-cluster-qe <-----
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "4"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    traffic-policy.network.alpha.openshift.io/local-with-fallback: ""
  creationTimestamp: "2022-01-24T05:26:09Z"

check the ingresscontroller and the related CO:
oc -n openshift-ingress-operator get ingresscontroller internalapps -o yaml 
  - lastTransitionTime: "2022-01-24T05:26:47Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2022-01-24T05:48:14Z"
    message: 'One or more managed resources are not upgradeable: load balancer service <------
      has been modified; changes must be reverted before upgrading: '
    reason: OperandsNotUpgradeable
    status: "False"   <------
    type: Upgradeable <------
  domain: internalapps.pdhamdhe41024.qe.devcluster.openshift.com
          type: NLB
        type: AWS
      scope: External
    type: LoadBalancerService

oc get co ingress -o yaml
  - lastTransitionTime: "2022-01-24T03:41:38Z"
    message: The "default" ingress controller reports Degraded=False.
    reason: IngressNotDegraded
    status: "False" 
    type: Degraded
  - lastTransitionTime: "2022-01-24T05:48:15Z"
    message: 'Some ingresscontrollers are not upgradeable: ingresscontroller "internalapps" <------
      is not upgradeable: OperandsNotUpgradeable: One or more managed resources are
      not upgradeable: load balancer service has been modified; changes must be reverted
      before upgrading: '
    reason: IngressControllersNotUpgradeable
    status: "False" <------
    type: Upgradeable <------
  extension: null

Comment 8 errata-xmlrpc 2022-03-12 04:40:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.