Bug 2104803
| Summary: | lr-policy-list for EgressIP was lost after scale down the test pods | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | huirwang |
| Component: | Networking | Assignee: | Andreas Karis <akaris> |
| Networking sub component: | ovn-kubernetes | QA Contact: | huirwang |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | akaris |
| Version: | 4.11 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.12.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-01-17 19:51:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I could reproduce this on 4.11 rc6 with the following files. The important thing here is that the egressIP spec needs 2 different IP addresses, otherwise this will not work: ~~~ configmap.yaml:# https://github.com/openshift/cluster-network-operator/blob/a7c762466c9e74f5f620b22fe117a0623d7775dd/README.md configmap.yaml:kind: ConfigMap configmap.yaml:apiVersion: v1 configmap.yaml:metadata: configmap.yaml: name: env-overrides configmap.yaml: namespace: openshift-ovn-kubernetes configmap.yaml: annotations: configmap.yaml:data: configmap.yaml: # to adjust master log levels, use _master configmap.yaml: _master: | configmap.yaml: OVN_KUBE_LOG_LEVEL=5 configmap.yaml: # OVN_LOG_LEVEL=dbg egressip.yaml:apiVersion: k8s.ovn.org/v1 egressip.yaml:kind: EgressIP egressip.yaml:metadata: egressip.yaml: name: egressip egressip.yaml:spec: egressip.yaml: egressIPs: egressip.yaml: - 10.0.129.10 egressip.yaml: - 10.0.129.11 egressip.yaml: namespaceSelector: egressip.yaml: matchLabels: egressip.yaml: kubernetes.io/metadata.name: egressip fedora.yaml:# oc adm policy add-scc-to-user privileged -z default fedora.yaml:apiVersion: apps/v1 fedora.yaml:kind: Deployment fedora.yaml:metadata: fedora.yaml: name: fedora-deployment fedora.yaml: labels: fedora.yaml: app: fedora-deployment fedora.yaml:spec: fedora.yaml: replicas: 1 fedora.yaml: selector: fedora.yaml: matchLabels: fedora.yaml: app: fedora-pod fedora.yaml: template: fedora.yaml: metadata: fedora.yaml: labels: fedora.yaml: app: fedora-pod fedora.yaml: spec: fedora.yaml: containers: fedora.yaml: - name: fedora fedora.yaml: image: fedora fedora.yaml: command: fedora.yaml: - sleep fedora.yaml: - infinity fedora.yaml: imagePullPolicy: IfNotPresent fedora.yaml: securityContext: fedora.yaml: runAsUser: 0 fedora.yaml: capabilities: fedora.yaml: add: fedora.yaml: - "SETFCAP" fedora.yaml: - "CAP_NET_RAW" fedora.yaml: - "CAP_NET_ADMIN" label.sh:#!/bin/bash label.sh: label.sh:EGRESS_ASSIGNABLE_LABEL="k8s.ovn.org/egress-assignable" label.sh:n=$(oc get nodes -o name -l node-role.kubernetes.io/worker= | head -1) label.sh:oc label $n ${EGRESS_ASSIGNABLE_LABEL}="" --overwrite label.sh: label.sh:echo "When picking the egressip, select it from the following range:" label.sh:oc get -o yaml $n | grep egress-ipconfig label.sh: label.sh:oc new-project egressip || oc project egressip label.sh:oc adm policy add-scc-to-user privileged -z default label.sh:oc apply -f fedora.yaml label.sh:oc apply -f egressip.yaml ~~~ When looking at the logs, I found log entries reminiscent of the issue that I was trying to address with https://github.com/ovn-org/ovn-kubernetes/pull/3066 It's just a guess at the moment, but I am currently verifying if that PR addresses this issue. After running 4.11 with the patch from PR3066, I cannot reproduce the issue.
Before, I would see the following in the logs:
~~~
2022-08-02T15:11:16.085192690Z I0802 15:11:16.085168 1 egressip.go:1386] Patching status on EgressIP egressip: [{ip-10-0-178-21.us-west-1.compute.internal 10.0.129.11}]
2022-08-02T15:11:16.090845483Z I0802 15:11:16.090821 1 obj_retry.go:1119] *v1.CloudPrivateIPConfig retry delete failed for 10.0.129.10, will try again later: no pending operation found for EgressIP: egressip
~~~
And a missing re-route entry in the ovn_cluster_router lr-policy-list as well as 2 NAT entries (default NAT + egressip NAT) on the GR router for the egressip.
My tests are a bit limited (I couldn't test more than a handful of times with each image) but I could reproduce the issue every single time without the patch, and never with the patch.
With the patch applied, none of this happens and the EgressIP is correctly assigned. I will hence wait for the upstream patch to merge downstream.
Downstream merge https://github.com/openshift/ovn-kubernetes/pull/1231 landed Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399 |
Description of problem: lr-policy-list for EgressIP was lost after scale down the test pods Tested on AWS. Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-07-06-145812 How reproducible: Frequently Steps to Reproduce: 1. Label one node as egress node 2. Create egressIP object $ oc get egressip -o yaml apiVersion: v1 items: - apiVersion: k8s.ovn.org/v1 kind: EgressIP metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"k8s.ovn.org/v1","kind":"EgressIP","metadata":{"annotations":{},"name":"egressip-47021"},"spec":{"egressIPs":["10.0.56.19","10.0.59.169"],"namespaceSelector":{"matchLabels":{"name":"test"}}}} creationTimestamp: "2022-07-07T07:12:21Z" generation: 11 name: egressip-47021 resourceVersion: "146672" uid: 9d6f75ac-3aaf-46c9-827c-fdebaeaa5313 spec: egressIPs: - 10.0.56.19 - 10.0.59.169 namespaceSelector: matchLabels: name: test status: items: - egressIP: 10.0.56.19 node: ip-10-0-51-130.us-east-2.compute.internal kind: List metadata: resourceVersion: "" selfLink: "" 3. Created a namespace and test pods in it. The test pods scaled to 10. added name=test label to it. 4. Scale down CNO to 0 5. Delete ovnkube-master pods 6. Scaled test pods to 1 $ oc get pods -n e2e-test-networking-pyi2tbyh-trtc7 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-rc-jfpxm 1/1 Running 0 23m 10.128.2.19 ip-10-0-70-185.us-east-2.compute.internal <none> <none> $ oc get ns e2e-test-networking-pyi2tbyh-trtc7 --show-labels NAME STATUS AGE LABELS e2e-test-networking-pyi2tbyh-trtc7 Active 7m16s kubernetes.io/metadata.name=e2e-test-networking-pyi2tbyh-trtc7,name=test,pod-security.kubernetes.io/audit-version=v1.24,pod-security.kubernetes.io/audit=restricted,pod-security.kubernetes.io/warn-version=v1.24,pod-security.kubernetes.io/warn=restricted 7. Scale up CNO to 1 8. Check lr-policy-list and snat Actual results: No lr-policy-list for egressip but it had snat here $ oc exec ovnkube-master-rz5vs -n openshift-ovn-kubernetes -- ovn-nbctl lr-policy-list ovn_cluster_router Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker Routing Policies 1004 inport == "rtos-ip-10-0-51-130.us-east-2.compute.internal" && ip4.dst == 10.0.51.130 /* ip-10-0-51-130.us-east-2.compute.internal */ reroute 10.129.2.2 1004 inport == "rtos-ip-10-0-53-201.us-east-2.compute.internal" && ip4.dst == 10.0.53.201 /* ip-10-0-53-201.us-east-2.compute.internal */ reroute 10.129.0.2 1004 inport == "rtos-ip-10-0-55-181.us-east-2.compute.internal" && ip4.dst == 10.0.55.181 /* ip-10-0-55-181.us-east-2.compute.internal */ reroute 10.131.0.2 1004 inport == "rtos-ip-10-0-58-64.us-east-2.compute.internal" && ip4.dst == 10.0.58.64 /* ip-10-0-58-64.us-east-2.compute.internal */ reroute 10.128.0.2 1004 inport == "rtos-ip-10-0-69-225.us-east-2.compute.internal" && ip4.dst == 10.0.69.225 /* ip-10-0-69-225.us-east-2.compute.internal */ reroute 10.130.0.2 1004 inport == "rtos-ip-10-0-70-185.us-east-2.compute.internal" && ip4.dst == 10.0.70.185 /* ip-10-0-70-185.us-east-2.compute.internal */ reroute 10.128.2.2 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.0.51.130/32 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.0.53.201/32 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.0.55.181/32 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.0.58.64/32 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.0.69.225/32 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.0.70.185/32 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 10.128.0.0/14 allow 101 ip4.src == 10.128.0.0/14 && ip4.dst == 100.64.0.0/16 allow $ oc exec ovnkube-master-rz5vs -n openshift-ovn-kubernetes -- ovn-nbctl --format=csv --no-heading find nat external_ids:name=egressip-47021 Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker eac2635c-b8af-425f-89ee-68a00699486e,[],[],{name=egressip-47021},"""10.0.56.19""",[],"""""",[],"""10.128.2.19""",k8s-ip-10-0-51-130.us-east-2.compute.internal,"{stateless=""false""}",snat The source IP of egress traffic was not EgressIP $ oc rsh -n e2e-test-networking-pyi2tbyh-trtc7 test-rc-jfpxm ~ $ curl 10.0.5.230:9095 10.0.70.185~ Expected results: EgressIP lr-policy-list should be updated correctly. Additional info: