Bug 1683436

Summary: deleting clusteringress object leaves AWS resources/entries
Product: OpenShift Container Platform Reporter: Joel Diaz <jdiaz>
Component: NetworkingAssignee: Dan Mace <dmace>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: aos-bugs, dhansen
Version: 4.1.0   
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:44:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joel Diaz 2019-02-26 20:40:58 UTC
Description of problem:
After creating a new ClusterIngress object, the expected Route53 and ELB objects are created in the AWS account.

After deleting the ClusterIngress object, the AWS pieces are not cleaned up.

Version-Release number of selected component (if applicable):
registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-26-105449

How reproducible:
100%

Steps to Reproduce:
1. Create clusteringress object
apiVersion: ingress.openshift.io/v1alpha1
kind: ClusterIngress
metadata:
  name: myingress
  namespace: openshift-ingress-operator
spec:
  defaultCertificateSecret: null
  highAvailability:
    type: Cloud
  ingressDomain: myapps.jdiaz-master.new-installer.openshift.com
  namespaceSelector: null
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/worker: ""
  replicas: 1
  routeSelector: null
  unsupportedExtensions: null

2. Check that AWS objects created successfully
3. Delete ClusterIngress object

Actual results:
AWS route53 entries, and ELBs created as a result of the ClusterIngress object persist after removing the ClusterIngress object.


Expected results:

AWS objects get cleaned up.


Additional info:
Here's the output from the above steps:

CREATE:
[jdiaz@minigoomba ~]$ oc create -f ~/ingress.yaml      
clusteringress.ingress.openshift.io/myingress created
[jdiaz@minigoomba ~]$ oc get clusteringress myingress -o yaml
apiVersion: ingress.openshift.io/v1alpha1                                                       
kind: ClusterIngress                           
metadata:                                    
  creationTimestamp: 2019-02-26T19:35:56Z                              
  generation: 1                           
  name: myingress                               
  namespace: openshift-ingress-operator           
  resourceVersion: "175204"                       
  selfLink: /apis/ingress.openshift.io/v1alpha1/namespaces/openshift-ingress-operator/clusteringresses/myingress
  uid: bc96e96d-39fd-11e9-ba80-0e9e31d5740e                            
spec:                                          
  defaultCertificateSecret: null       
  highAvailability:                   
    type: Cloud                                                                                                                                           
  ingressDomain: myapps.jdiaz-master.new-installer.openshift.com                                                                                        
  namespaceSelector: null                                                                                                                        
  nodePlacement:                                                                                                                                    
    nodeSelector:                                                                                                                                               
      matchLabels:                                                                                                                                              
        node-role.kubernetes.io/worker: ""                                                                                                                      
  replicas: 1                                                                                                               
  routeSelector: null                                                                                                             
  unsupportedExtensions: null                                                                                                
status:                                                                                                           
  highAvailability:                                                   
    type: Cloud                                                              
  ingressDomain: myapps.jdiaz-master.new-installer.openshift.com          
  labelSelector: app=router,router=router-myingress           
  replicas: 1      

QUERY AWS:
[jdiaz@minigoomba ~]$ aws route53 list-resource-record-sets --hosted-zone-id  Z1QIFLK9U9F5AZ | grep -A7 -B1 myapps
        {                                    
            "Name": "\\052.myapps.jdiaz-master.new-installer.openshift.com.",
            "Type": "A",                
            "AliasTarget": {                                                                                                   
                "HostedZoneId": "Z35SXDOTRQ7X7K",
                "DNSName": "abca8ff0439fd11e9ba800e9e31d5740-1286579122.us-east-1.elb.amazonaws.com.",
                "EvaluateTargetHealth": false
            }                                                      
        } 

[jdiaz@minigoomba ~]$ aws elb describe-load-balancers --region us-east-1 --load-balancer-names abca8ff0439fd11e9ba800e9e31d5740 | grep DNSName
            "DNSName": "abca8ff0439fd11e9ba800e9e31d5740-1286579122.us-east-1.elb.amazonaws.com",

DELETE:
[jdiaz@minigoomba ~]$ oc delete clusteringress myingress                                                                                                        
clusteringress.ingress.openshift.io "myingress" deleted                                                                     
[jdiaz@minigoomba ~]$ oc get clusteringress myingress -o yaml                                                                     
Error from server (NotFound): clusteringresses.ingress.openshift.io "myingress" not found                                    

[jdiaz@minigoomba ~]$ aws route53 list-resource-record-sets --hosted-zone-id  Z1QIFLK9U9F5AZ | grep -A7 -B1 myapps
        {                                                             
            "Name": "\\052.myapps.jdiaz-master.new-installer.openshift.com.",
            "Type": "A",                                                  
            "AliasTarget": {                                  
                "HostedZoneId": "Z35SXDOTRQ7X7K",
                "DNSName": "abca8ff0439fd11e9ba800e9e31d5740-1286579122.us-east-1.elb.amazonaws.com.",            
                "EvaluateTargetHealth": false
            }                                                                
        }                               

[jdiaz@minigoomba ~]$ aws elb describe-load-balancers --region us-east-1 --load-balancer-names abca8ff0439fd11e9ba800e9e31d5740 | grep DNSName
            "DNSName": "abca8ff0439fd11e9ba800e9e31d5740-1286579122.us-east-1.elb.amazonaws.com",

Comment 1 Hongan Li 2019-02-27 03:01:50 UTC
might be duplicated to https://bugzilla.redhat.com/show_bug.cgi?id=1682922
please check if the deployment and svc in "openshift-ingress" namespace was deleted when you deleting your clusteringress.

Comment 2 Joel Diaz 2019-02-27 14:31:37 UTC
(In reply to hongli from comment #1)
> might be duplicated to https://bugzilla.redhat.com/show_bug.cgi?id=1682922
> please check if the deployment and svc in "openshift-ingress" namespace was
> deleted when you deleting your clusteringress.

Yes, the referenced bugzilla covers the kubernetes objects being left around, and this BZ covers the cloud pieces, but they are effectively the same issue being reported as I also see the in-cluster objects being left around after deleting the clusteringress object.

Comment 5 Daneyon Hansen 2019-03-12 19:31:24 UTC
https://github.com/openshift/cluster-ingress-operator/commit/8246fb3e170ae6796ab9bcd852dc22cf9609b9a8 changed the api to ingresscontrollers. Here is an updated example manifest:

$ cat test0-ing.yaml 
kind: IngressController
apiVersion: operator.openshift.io/v1
metadata:
  name: test0
  namespace: openshift-ingress-operator
spec:
  domain: tests0.<your_ingress_domain>

Comment 6 Hongan Li 2019-03-15 06:39:29 UTC
verified with 4.0.0-0.nightly-2019-03-13-233958 and the issue has been fixed.

The LB and DNS record related has been removed after ingresscontroller is deleted.

Comment 8 errata-xmlrpc 2019-06-04 10:44:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758