Bug 2044303

Summary: [ovn][cloud-network-config-controller] cloudprivateipconfigs ips were left after deleting egressip objects
Product: OpenShift Container Platform Reporter: huirwang
Component: NetworkingAssignee: Ben Bennett <bbennett>
Networking sub component: ovn-kubernetes QA Contact: jechen <jechen>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: jechen, trozet
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:42:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description huirwang 2022-01-24 11:58:52 UTC 
Description of problem:
[ovn][cloud-network-config-controller] cloudprivateipconfigs ips were left after deleting egressip objects

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-22-102609 

How reproducible:


Steps to Reproduce:
1. Create some egressip objects and then delete them, repeat this for a couple of times.
2. Then delete all the egressip objects
$ oc get egressip
No resources found


Actual results:
Found multiple cloudprivateipconfigs  left.
$ oc get cloudprivateipconfigs 
NAME          AGE
10.0.48.110   4h55m
10.0.48.111   11m
10.0.49.88    18m
10.0.67.200   11m

oc get cloudprivateipconfigs -o yaml
apiVersion: v1
items:
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip-example6
    creationTimestamp: "2022-01-24T06:57:16Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.48.110
    resourceVersion: "129574"
    uid: 2a9191cf-cca3-4a23-ba13-474d87f35d65
  spec:
    node: ip-10-0-63-135.us-east-2.compute.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-24T07:02:51Z"
      message: 'Error processing cloud assignment request, err: <nil>'
      observedGeneration: 1
      reason: CloudResponseError
      status: "False"
      type: Assigned
    node: ip-10-0-63-135.us-east-2.compute.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip-example6
    creationTimestamp: "2022-01-24T11:40:37Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.48.111
    resourceVersion: "224588"
    uid: 21c84cf6-78f3-46ab-946b-8281aefa1011
  spec:
    node: ip-10-0-50-161.us-east-2.compute.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-24T11:40:38Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: ip-10-0-50-161.us-east-2.compute.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip
    creationTimestamp: "2022-01-24T11:34:14Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.49.88
    resourceVersion: "222149"
    uid: 071aa906-8bd1-4f75-9385-7c42be21454b
  spec:
    node: ip-10-0-50-161.us-east-2.compute.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-24T11:34:15Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: ip-10-0-50-161.us-east-2.compute.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip-example6
    creationTimestamp: "2022-01-24T11:40:37Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.67.200
    resourceVersion: "224589"
    uid: d87f60d2-bf56-44f7-94bc-7728cc0fd44b
  spec:
    node: ip-10-0-67-117.us-east-2.compute.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-24T11:40:38Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: ip-10-0-67-117.us-east-2.compute.internal
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Expected results:
After egressip object removed, cloudprivateipconfigs ips should be cleared.

Additional info:
Comment 4 jechen 2022-01-27 19:56:39 UTC 
on GCP, I verified the fix with 4.10.0-0.nightly-2022-01-27-104747

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-27-104747   True        False         26m     Cluster version is 4.10.0-0.nightly-2022-01-27-104747


$ oc get egressip
NAME         EGRESSIPS      ASSIGNED NODE                                               ASSIGNED EGRESSIPS
egressip1    10.0.128.101   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.101
egressip10   10.0.128.110   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.110
egressip2    10.0.128.102   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.102
egressip3    10.0.128.103   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.103
egressip4    10.0.128.104   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.104
egressip5    10.0.128.105   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.105
egressip6    10.0.128.106   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.106
egressip7    10.0.128.107   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.107
egressip8    10.0.128.108   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.108
egressip9    10.0.128.109   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.109


$ oc get cloudprivateipconfigs -o yaml
apiVersion: v1
items:
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip1
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.101
    resourceVersion: "63852"
    uid: 6c29d4c9-28bb-4d46-a39e-6dc6cbc9713b
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:18Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip2
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.102
    resourceVersion: "64101"
    uid: 904ead25-31f7-43ad-85df-1a47e8e76972
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:55Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip3
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.103
    resourceVersion: "63875"
    uid: af92ce3b-2fe6-4e3a-a223-57ca30297032
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:22Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip4
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.104
    resourceVersion: "64090"
    uid: 8ae1ae40-c2b8-4eb5-8b9c-ce3c38f5d6c9
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:52Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip5
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.105
    resourceVersion: "64045"
    uid: f2deec11-55dc-4afe-af9f-61f117389e3f
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:47Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip6
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.106
    resourceVersion: "64028"
    uid: 7e35988f-a400-44b8-a68f-b403561ee9d4
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:43Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip7
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.107
    resourceVersion: "63951"
    uid: 2966523f-0a2f-4d70-930b-bf4c14e1cdbf
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:32Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip8
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.108
    resourceVersion: "63929"
    uid: ea3a7997-4ab7-4af1-b669-076ccfde8f10
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:28Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip9
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.109
    resourceVersion: "63976"
    uid: 0898ed63-2248-4458-940e-dbdc61894e5f
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:25:38Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
- apiVersion: cloud.network.openshift.io/v1
  kind: CloudPrivateIPConfig
  metadata:
    annotations:
      k8s.ovn.org/egressip-owner-ref: egressip10
    creationTimestamp: "2022-01-27T19:25:12Z"
    finalizers:
    - cloudprivateipconfig.cloud.network.openshift.io/finalizer
    generation: 1
    name: 10.0.128.110
    resourceVersion: "64133"
    uid: 7f3537b9-f1ee-4216-9ecf-07e6907dd967
  spec:
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
  status:
    conditions:
    - lastTransitionTime: "2022-01-27T19:26:02Z"
      message: IP address successfully added
      observedGeneration: 1
      reason: CloudResponseSuccess
      status: "True"
      type: Assigned
    node: jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


$ oc delete egressip egressip1
egressip.k8s.ovn.org "egressip1" deleted
$ oc delete egressip egressip2
egressip.k8s.ovn.org "egressip2" deleted
$ oc delete egressip egressip3
egressip.k8s.ovn.org "egressip3" deleted
$ oc delete egressip egressip4
egressip.k8s.ovn.org "egressip4" deleted
$ oc delete egressip egressip5
egressip.k8s.ovn.org "egressip5" deleted
$ oc delete egressip egressip6
egressip.k8s.ovn.org "egressip6" deleted
$ oc delete egressip egressip7
egressip.k8s.ovn.org "egressip7" deleted
$ oc delete egressip egressip8
egressip.k8s.ovn.org "egressip8" deleted
$ oc delete egressip egressip9
egressip.k8s.ovn.org "egressip9" deleted
$ oc delete egressip egressip10
egressip.k8s.ovn.org "egressip10" deleted

$ oc get egressip
No resources found


$ oc get cloudprivateipconfigs -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""



After egressip objects are removed, cloudprivateipconfigs ips are all cleared up


# wait a little bit
Comment 8 errata-xmlrpc 2022-03-10 16:42:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056