Bug 2027244

Hi

A couple of things on this:

1. There has been a "performance improvement" for egress IP. It's currently being back-ported to 4.9 https://github.com/openshift/ovn-kubernetes/pull/884/commits/8e6ef439426bd471151548c43832d9c9284e2d44 so 4.8 is coming after. Previously the egress IP setup needed to wait for pod.Status.podIPs to be set before we could proceed with the setup, that takes a lot longer than just reacting to the annotation ovn sets when it has allocated an IP address.

2. 1. doesn't matter because egress IP can only work once the pod has received an IP address and is running. I.e: just reading logs and making a determination of whether this has performance impacts is moot point for egress IP. The egress IP performance is dependent on the CRI, Kubelet, ovnkube-master's IP allocation for the pod, ovnkube-node's networking setup for the pod, etc. Even if the logs would have indicated a quicker time, it still wouldn't have matter for what concerns the question: "__can__ the pod use egress IP?" because it can only start sending traffic once __all__ of that is done, not once those logs have been written.

3. Given 2. we also need to know what conditions you're cluster has. How many pods are you running? How many nodes? How many namespaces? Performance is impacted by all of this, and sometimes things are just slow because there's a lot of things going on. 

I am thinking about closing this as a NOTABUG. But I'll keep it open for posterity until answers and more information come in w.r.t the cluster setup. 

/Alex.

Please follow https://bugzilla.redhat.com/show_bug.cgi?id=2034668 for an answer as to when https://github.com/openshift/ovn-kubernetes/pull/884/commits/8e6ef439426bd471151548c43832d9c9284e2d44 is back-ported to 4.8. But like I said in https://bugzilla.redhat.com/show_bug.cgi?id=2027244#c1 this is not really a bug in that the egress IP code doesn't completely decide how fast a pod starts using an egress IP. If there's a big delta between the time a pod matching an egress IP is started and running and the time its egress traffic gets the egress IP assigned, then that's a bug. But those log lines are not an indication of something conclusive and necessarily wrong.

Thank you very much for your reply. I have taken a look at the "performance improvement" for the egress IP. I think this change fixes this problem. Unfortunately, I'm stuck on okd 4.8.0-0.okd-2021-11-14-052418 right now and can't update for other reasons.
After updating to okd 4.9 with your patch, I will let you know if the problem has been fixed.

I have updated my pre prod environment to OKD 4.9.0-0.okd-2022-01-29-035536. A review of the log shows a significant speed improvement. Everything is done in milliseconds. In my eyes, the bug can be closed.

I0210 10:25:32.262224       1 egressip.go:416] EgressIP: vault has matched on pod: kubernetes-external-secrets-567c97cf48-5xfgm in namespace: external-secrets
I0210 10:25:32.275289       1 egressip.go:427] EgressIP: vault update for pod: kubernetes-external-secrets-567c97cf48-5xfgm in namespace: external-secrets
I0210 10:25:32.277769       1 pods.go:293] Creating logical port for external-secrets_kubernetes-external-secrets-567c97cf48-5xfgm on switch worker3-cl1-dc99.s-ocp.cloud.avm.de [66eb4bba-709b-4a17-9a5a-2df29978d65c]
I0210 10:25:32.279553       1 pods.go:491] Annotation values: ip=[10.129.4.7/23] ; mac=0a:58:0a:81:04:07 ; gw=[10.129.4.1]
Annotation=map[k8s.ovn.org/pod-networks:{"default":{"ip_addresses":["10.129.4.7/23"],"mac_address":"0a:58:0a:81:04:07","gateway_ips":["10.129.4.1"],"ip_address":"10.129.4.7/23","gateway_ip":"10.129.4.1"}}]
I0210 10:25:32.281066       1 kube.go:67] Setting annotations map[k8s.ovn.org/pod-networks:{"default":{"ip_addresses":["10.129.4.7/23"],"mac_address":"0a:58:0a:81:04:07","gateway_ips":["10.129.4.1"],"ip_address":"10.129.4.7/23","gateway_ip":"10.129.4.1"}}] on pod external-secrets/kubernetes-external-secrets-567c97cf48-5xfgm
I0210 10:25:32.367885       1 port_cache.go:55] port-cache(external-secrets_kubernetes-external-secrets-567c97cf48-5xfgm): added port &{name:external-secrets_kubernetes-external-secrets-567c97cf48-5xfgm uuid:8e5fd222-f9d8-4d28-a889-53a98cab581c logicalSwitch:worker3-cl1-dc99.s-ocp.cloud.avm.de ips:[0xc001f9a750] mac:[10 88 10 129 4 7] expires:{wall:0 ext:0 loc:<nil>}}
I0210 10:25:32.369126       1 pods.go:282] [external-secrets/kubernetes-external-secrets-567c97cf48-5xfgm] addLogicalPort took 91.370815ms, OVN Execute time 14.5123ms, pod Annotation time: 72.271737ms
I0210 10:25:32.369222       1 egressip.go:427] EgressIP: vault update for pod: kubernetes-external-secrets-567c97cf48-5xfgm in namespace: external-secrets
I0210 10:25:32.369288       1 egressip.go:427] EgressIP: vault update for pod: kubernetes-external-secrets-567c97cf48-5xfgm in namespace: external-secrets
I0210 10:25:32.369413       1 ovs.go:209] exec(1071): /usr/bin/ovn-nbctl --timeout=15 --format=csv --data=bare --no-heading --columns=_uuid find logical_router_policy match="ip4.src == 10.129.4.7" priority=100 external_ids:name=vault nexthops=["100.64.0.8"]
I0210 10:25:32.395977       1 ovs.go:209] exec(1072): /usr/bin/ovn-nbctl --timeout=15 --id=@lr-policy create logical_router_policy action=reroute match="ip4.src == 10.129.4.7" priority=100 nexthops=["100.64.0.8"] external_ids:name=vault -- add logical_router ovn_cluster_router policies @lr-policy
2022-02-10T10:25:32.401Z|00737|ovn_dbctl|INFO|Running command run --id=@lr-policy -- create logical_router_policy action=reroute "match=\"ip4.src == 10.129.4.7\"" priority=100 "nexthops=[\"100.64.0.8\"]" external_ids:name=vault -- add logical_router ovn_cluster_router policies @lr-policy

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.32 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0559

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days