Bug 2046296

Summary: When creating multiple consecutive egressIPs on GCP not all of them get assigned to the instance
Product: OpenShift Container Platform Reporter: Patryk Diak <pdiak>
Component: NetworkingAssignee: Patryk Diak <pdiak>
Networking sub component: ovn-kubernetes QA Contact: jechen <jechen>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified    
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-12 04:41:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Patryk Diak 2022-01-26 14:31:25 UTC
Description of problem:
When creating or removing multiple egressIPs quickly they are not correctly removed/added on GCP instance

Version-Release number of selected component (if applicable):
4.10.0-0.ci-2022-01-26-071949

How reproducible:
Always

Steps to Reproduce:
run the following script:

for i in {1..10}
  do
    oc new-project test${i}
    cat << EOF | oc apply -f -
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: egressip-${i}
spec:
  egressIPs:
  - 10.0.128.1${i}
  namespaceSelector:
    matchLabels:
      name: test${i}
EOF
  done


Actual results:
Check ipCidrRange on the instance:

gcloud compute instances describe pdiak-01-26-2022-t8xmx-worker-b-7jmwh | grep ipCidrRange
No zone specified. Using zone [us-east1-b] for instance: [pdiak-01-26-2022-t8xmx-worker-b-7jmwh].
  - ipCidrRange: 10.0.128.11/32
  - ipCidrRange: 10.0.128.12/32
  - ipCidrRange: 10.0.128.15/32
  - ipCidrRange: 10.0.128.18/32
  - ipCidrRange: 10.0.128.110/32

egressip falsly report that all are assigned:
NAME          EGRESSIPS      ASSIGNED NODE                           ASSIGNED EGRESSIPS
egressip-1    10.0.128.11    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.11
egressip-10   10.0.128.110   pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.110
egressip-2    10.0.128.12    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.12
egressip-3    10.0.128.13    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.13
egressip-4    10.0.128.14    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.14
egressip-5    10.0.128.15    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.15
egressip-6    10.0.128.16    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.16
egressip-7    10.0.128.17    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.17
egressip-8    10.0.128.18    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.18
egressip-9    10.0.128.19    pdiak-01-26-2022-t8xmx-worker-b-7jmwh   10.0.128.19


Expected results:
all egress ips should be assigned on the gcp instance

Additional info:

Comment 4 jechen 2022-01-27 19:34:39 UTC
Verified in 4.10.0-0.nightly-2022-01-27-104747

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-27-104747   True        False         26m     Cluster version is 4.10.0-0.nightly-2022-01-27-104747


$ oc get node
NAME                                                        STATUS   ROLES    AGE   VERSION
jechen-0127c-r76dn-master-0.c.openshift-qe.internal         Ready    master   48m   v1.23.0+d30ebbc
jechen-0127c-r76dn-master-1.c.openshift-qe.internal         Ready    master   49m   v1.23.0+d30ebbc
jechen-0127c-r76dn-master-2.c.openshift-qe.internal         Ready    master   49m   v1.23.0+d30ebbc
jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   Ready    worker   37m   v1.23.0+d30ebbc
jechen-0127c-r76dn-worker-b-7qtqd.c.openshift-qe.internal   Ready    worker   37m   v1.23.0+d30ebbc
jechen-0127c-r76dn-worker-c-gdjfb.c.openshift-qe.internal   Ready    worker   37m   v1.23.0+d30ebbc


$ oc label node jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal "k8s.ovn.org/egress-assignable"=""
node/jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal labeled

$ oc label node jechen-0127c-r76dn-worker-b-7qtqd.c.openshift-qe.internal "k8s.ovn.org/egress-assignable"=""
node/jechen-0127c-r76dn-worker-b-7qtqd.c.openshift-qe.internal labeled

$ oc label node jechen-0127c-r76dn-worker-c-gdjfb.c.openshift-qe.internal "k8s.ovn.org/egress-assignable"=""
node/jechen-0127c-r76dn-worker-c-gdjfb.c.openshift-qe.internal labeled

# create script to add 10 egressip objects


$ oc get egressip
NAME         EGRESSIPS      ASSIGNED NODE                                               ASSIGNED EGRESSIPS
egressip1    10.0.128.101   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.101
egressip10   10.0.128.110   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.110
egressip2    10.0.128.102   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.102
egressip3    10.0.128.103   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.103
egressip4    10.0.128.104   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.104
egressip5    10.0.128.105   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.105
egressip6    10.0.128.106   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.106
egressip7    10.0.128.107   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.107
egressip8    10.0.128.108   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.108
egressip9    10.0.128.109   jechen-0127c-r76dn-worker-a-6gcp2.c.openshift-qe.internal   10.0.128.109


$ gcloud compute instances describe jechen-0127c-r76dn-worker-a-6gcp2 |grep ipCidrRange
  - ipCidrRange: 10.0.128.101/32
  - ipCidrRange: 10.0.128.103/32
  - ipCidrRange: 10.0.128.108/32
  - ipCidrRange: 10.0.128.107/32
  - ipCidrRange: 10.0.128.109/32
  - ipCidrRange: 10.0.128.106/32
  - ipCidrRange: 10.0.128.105/32
  - ipCidrRange: 10.0.128.104/32
  - ipCidrRange: 10.0.128.102/32
  - ipCidrRange: 10.0.128.110/32


All egressips are assigned on the gcp instance

Comment 7 errata-xmlrpc 2022-03-12 04:41:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056