Bug 2041681

Summary: [OVN] IBM-Cloud IPsec cluster cannot provision workers, machine-api-controller gets dial tcp: i/o timeout
Product: OpenShift Container Platform Reporter: Ross Brattain <rbrattai>
Component: NetworkingAssignee: Andreas Karis <akaris>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: medium CC: akaris, anbhat, mifiedle
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-23 08:40:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2038774    
Bug Blocks:    

Description Ross Brattain 2022-01-18 05:50:18 UTC 
Description of problem:

IBM-Cloud OVN IPsec deployment cannot create workers.
The machine-controller fails when trying to provision workers.  

Specifically the machine-controller is unable to access the IBM API.

Initial failures were DNS

sh-4.4$ curl -v https://iam.cloud.ibm.com/identity/token                                                                                                                                                                                                                       * Could not resolve host: iam.cloud.ibm.com

Then other http timeouts

  Error Message:           error creating ibm client: Get "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06": dial tcp: i/o timeout



Version-Release number of selected component (if applicable):

4.10.0-0.nightly-2022-01-17-023213


How reproducible:

Always


Steps to Reproduce:
1. Deploy IBM-Cloud OVN-IPsec cluster with manual UDP port 500,4500 security group workaround as described in BZ 2038774
2. All three master deploy
3. No workers deploy
4. Test curl from inside machine-controller pod

oc exec -n openshift-machine-api -c machine-controller machine-api-controllers-c44c76c84-7xr76 -- curl -v https://iam.cloud.ibm.com/identity/token

oc exec -n openshift-machine-api -c machine-controller machine-api-controllers-c44c76c84-z7dvn -- curl -v "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06"


Actual results:

curl: (6) Could not resolve host: us-south.iaas.cloud.ibm.com

error creating ibm client: Get "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06": dial tcp: i/o timeout

Status:
  Conditions:
    Last Transition Time:  2022-01-18T03:24:27Z
    Message:               Instance has not been created
    Reason:                InstanceNotCreated
    Severity:              Warning
    Status:                False
    Type:                  InstanceExists
  Error Message:           error creating ibm client: Get "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06": dial tcp: i/o timeout
  Error Reason:            InvalidConfiguration
  Last Updated:            2022-01-18T03:25:07Z
  Phase:                   Failed


Expected results:

Workers are provisioned.


Initial failure was a DNS failure

 oc rsh -n openshift-machine-api -c machine-controller machine-api-controllers-c44c76c84-7xr76
sh-4.4$ curl https://iam.cloud.ibm.com/identity/token
curl: (6) Could not resolve host: iam.cloud.ibm.com
sh-4.4$ curl -v https://iam.cloud.ibm.com/identity/token
* Could not resolve host: iam.cloud.ibm.com
* Closing connection 0
curl: (6) Could not resolve host: iam.cloud.ibm.com
sh-4.4$
Comment 8 Andreas Karis 2022-01-19 16:28:57 UTC 
I had a look at the cluster that you shared with me.

I checked the tunnels status for a worker and a master node. If you look at the "Traffic: ESPin"/out lines, you can see that the nodes receive no ESP traffic:

Worker:
~~~
[akaris@linux 2041681]$ oc get pods -n openshift-ovn-kubernetes -o wide | grep rbrattai-i410i32-c5lwl-worker-3-wqtj9
ovn-ipsec-h6lgk        1/1     Running   0             61m   10.242.128.4   rbrattai-i410i32-c5lwl-worker-3-wqtj9   <none>           <none>
ovnkube-node-sf2d8     5/5     Running   0             61m   10.242.128.4   rbrattai-i410i32-c5lwl-worker-3-wqtj9   <none>           <none>
[akaris@linux 2041681]$ oc rsh -n openshift-ovn-kubernetes ovn-ipsec-h6lgk
Defaulted container "ovn-ipsec" out of: ovn-ipsec, ovn-keys (init)
sh-4.4# 
sh-4.4# 
sh-4.4# 
sh-4.4#  ovs-appctl -t ovs-monitor-ipsec tunnels/show
Interface name: ovn-e19934-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.129.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    e19934d5-cc17-4d2d-9677-513d1d158190
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         1
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081
  sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081
  sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081
  sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081
IPsec connections that are active:
  000 #6: "ovn-e19934-0-in-1" esp.a7d1942a.129.4 esp.d43520f1.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #8: "ovn-e19934-0-out-1" esp.2c14043.129.4 esp.76f9751f.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B

Interface name: ovn-d2ee6a-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.0.7
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         4
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
IPsec connections that are active:
  000 #9: "ovn-d2ee6a-0-in-1" esp.c1c1747f.0.7 esp.df9d7449.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #16: "ovn-d2ee6a-0-out-1" esp.43ef92b6.0.7 esp.501e394d.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B

Interface name: ovn-29c4ab-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.64.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    29c4aba3-da37-4b72-8820-fb3b28aa496a
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         3
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B

Interface name: ovn-ece7f2-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.1.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    ece7f209-8283-48d7-9060-6a1c1c7ca905
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         13
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #24: "ovn-ece7f2-0-in-1" esp.5c0268d2.1.4 esp.52144f65.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #22: "ovn-ece7f2-0-out-1" esp.e8d98c5.1.4 esp.f7d51d81.128.4 Traffic: ESPin=0B ESPout=7KB! ESPmax=0B

Interface name: ovn-b7253f-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.65.6
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    b7253f0d-0a4e-4cb4-8c26-aedb77d2567e
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         2
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.65.6/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.65.6/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.65.6/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081
  sel src 10.242.65.6/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081
  sel src 10.242.65.6/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.65.6/32 proto udp sport 6081
IPsec connections that are active:
  000 #13: "ovn-b7253f-0-in-1" esp.25602e91.65.6 esp.f535896e.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #18: "ovn-b7253f-0-out-1" esp.2b2d05e4.65.6 esp.fcc9b0d3.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B

sh-4.4#  ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep 10.242.64.4
  Remote IP:      10.242.64.4
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
  000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B
sh-4.4#  ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep 10.242.64.4 -C20
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
IPsec connections that are active:
  000 #9: "ovn-d2ee6a-0-in-1" esp.c1c1747f.0.7 esp.df9d7449.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #16: "ovn-d2ee6a-0-out-1" esp.43ef92b6.0.7 esp.501e394d.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B

Interface name: ovn-29c4ab-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.64.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    29c4aba3-da37-4b72-8820-fb3b28aa496a
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         3
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B

Interface name: ovn-ece7f2-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.1.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    ece7f209-8283-48d7-9060-6a1c1c7ca905
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         13
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
sh-4.4# reset
sh: reset: command not found
sh-4.4#  ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep 10.242.64.4 -C40
  sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081
IPsec connections that are active:
  000 #6: "ovn-e19934-0-in-1" esp.a7d1942a.129.4 esp.d43520f1.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #8: "ovn-e19934-0-out-1" esp.2c14043.129.4 esp.76f9751f.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B

Interface name: ovn-d2ee6a-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.0.7
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         4
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
IPsec connections that are active:
  000 #9: "ovn-d2ee6a-0-in-1" esp.c1c1747f.0.7 esp.df9d7449.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #16: "ovn-d2ee6a-0-out-1" esp.43ef92b6.0.7 esp.501e394d.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B

Interface name: ovn-29c4ab-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.64.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    29c4aba3-da37-4b72-8820-fb3b28aa496a
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         3
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B

Interface name: ovn-ece7f2-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.1.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    ece7f209-8283-48d7-9060-6a1c1c7ca905
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         13
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
  src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081
  sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081
  sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #24: "ovn-ece7f2-0-in-1" esp.5c0268d2.1.4 esp.52144f65.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #22: "ovn-ece7f2-0-out-1" esp.e8d98c5.1.4 esp.f7d51d81.128.4 Traffic: ESPin=0B ESPout=7KB! ESPmax=0B

Interface name: ovn-b7253f-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.128.4
  Remote IP:      10.242.65.6
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     6564d5cb-b7cc-4118-9628-116273be4c05
~~~

Master:
~~~
[akaris@linux 2041681]$ oc get pods -n openshift-ovn-kubernetes -o wide | grep master | grep ipsec
ovn-ipsec-4gm54        1/1     Running   0              109m   10.242.0.7     rbrattai-i410i32-c5lwl-master-0         <none>           <none>
ovn-ipsec-4trz5        1/1     Running   0              109m   10.242.129.4   rbrattai-i410i32-c5lwl-master-2         <none>           <none>
ovn-ipsec-mbdcl        1/1     Running   0              109m   10.242.65.6    rbrattai-i410i32-c5lwl-master-1         <none>           <none>
[akaris@linux 2041681]$ oc rsh -n openshift-ovn-kubernetes ovn-ipsec-4gm54
Defaulted container "ovn-ipsec" out of: ovn-ipsec, ovn-keys (init)
sh-4.4# 
sh-4.4# 
sh-4.4#  ovs-appctl -t ovs-monitor-ipsec tunnels/show
Interface name: ovn-b7253f-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.0.7
  Remote IP:      10.242.65.6
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    b7253f0d-0a4e-4cb4-8c26-aedb77d2567e
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         1
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081
  sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081
  sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081
  sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081
IPsec connections that are active:
  000 #8: "ovn-b7253f-0-in-1" esp.fe6b8855.65.6 esp.dffc7a81.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #9: "ovn-b7253f-0-out-1" esp.b793764d.65.6 esp.90e9a31c.0.7 Traffic: ESPin=0B ESPout=6MB! ESPmax=0B

Interface name: ovn-e19934-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.0.7
  Remote IP:      10.242.129.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    e19934d5-cc17-4d2d-9677-513d1d158190
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         3
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081
Kernel security associations installed:
  sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081
  sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081
  sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081
  sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081
IPsec connections that are active:
  000 #11: "ovn-e19934-0-in-1" esp.748a9eb8.129.4 esp.208686e1.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #12: "ovn-e19934-0-out-1" esp.3e8cb223.129.4 esp.d1656111.0.7 Traffic: ESPin=0B ESPout=7MB! ESPmax=0B

Interface name: ovn-29c4ab-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.0.7
  Remote IP:      10.242.64.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    29c4aba3-da37-4b72-8820-fb3b28aa496a
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         67
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.64.4/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.64.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.64.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081
  sel src 10.242.64.4/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.64.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #34: "ovn-29c4ab-0-in-1" esp.63abb726.64.4 esp.19d08a4f.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #32: "ovn-29c4ab-0-out-1" esp.bd8d4745.64.4 esp.bea13255.0.7 Traffic: ESPin=0B ESPout=742KB! ESPmax=0B

Interface name: ovn-6564d5-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.0.7
  Remote IP:      10.242.128.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    6564d5cb-b7cc-4118-9628-116273be4c05
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         71
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081
  sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #21: "ovn-6564d5-0-in-1" esp.501e394d.128.4 esp.43ef92b6.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #19: "ovn-6564d5-0-out-1" esp.df9d7449.128.4 esp.c1c1747f.0.7 Traffic: ESPin=0B ESPout=725KB! ESPmax=0B

Interface name: ovn-ece7f2-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       10.242.0.7
  Remote IP:      10.242.1.4
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     d2ee6a39-13b9-47d8-93a9-7c6375caf5b1
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    ece7f209-8283-48d7-9060-6a1c1c7ca905
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         75
  CFM state:      Disabled
Kernel policies installed:
  src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081
  src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081
  src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081
Kernel security associations installed:
  sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081
  sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081
  sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp dport 6081
  sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081
  sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp sport 6081
  sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081
IPsec connections that are active:
  000 #30: "ovn-ece7f2-0-in-1" esp.3cae9761.1.4 esp.f0a36212.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
  000 #29: "ovn-ece7f2-0-out-1" esp.11806853.1.4 esp.9a378fdb.0.7 Traffic: ESPin=0B ESPout=22KB! ESPmax=0B
~~~

I ran tcpdumps on br-ex once, and also once on ens3, on 2 of your worker nodes. You can see that ESP traffic only passes in the outbound direction, it is not received:
~~~
[akaris@linux 2041681]$ tshark -t ad -nn -r brattai-i410i32-c5lwl-worker-2-rmjxv.br-ex.pcap
    1 2022-01-19 16:07:18.448110 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    2 2022-01-19 16:07:23.447542 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    3 2022-01-19 16:07:28.447705 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
[akaris@linux 2041681]$ tshark -t ad -nn -r rbrattai-i410i32-c5lwl-worker-3-wqtj9.br-ex.pcap
    1 2022-01-19 16:07:10.344317 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
    2 2022-01-19 16:07:15.343699 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
    3 2022-01-19 16:07:20.343927 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
    4 2022-01-19 16:07:25.353964 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
    5 2022-01-19 16:07:30.353752 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
    6 2022-01-19 16:07:35.353849 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
[akaris@linux 2041681]$ tshark -t ad -nn -r rbrattai-i410i32-c5lwl-worker-2-rmjxv.ens3.pcap
    1 2022-01-19 16:20:18.747957 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    2 2022-01-19 16:20:23.747610 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    3 2022-01-19 16:20:28.747870 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    4 2022-01-19 16:21:43.779100 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    5 2022-01-19 16:21:48.778549 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
    6 2022-01-19 16:21:53.778824 02:00:01:30:ee:01 02:00:00:30:ee:01  10.242.64.4 → 10.242.128.4   ESP 234 ESP (SPI=0x13aadeb4)    
[akaris@linux 2041681]$ tshark -t ad -nn -r rbrattai-i410i32-c5lwl-worker-3-wqtj9.ens3x.pcap
    1 2022-01-19 16:19:00.627303 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
    2 2022-01-19 16:19:05.627416 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4    ESP 234 ESP (SPI=0xbbd09776)    
[akaris@linux 2041681]$ 
~~~

From what we talked about, you only unblocked UDP 500 and UDP 4500, but ESP is not unblocked.

I have strong reasons to believe that you *must* unblock ESP explicitly:

a) ovs-monitor-ipsec does not enforce NAT-T when setting up the ipsec tunnels:
https://github.com/openvswitch/ovs/blob/master/ipsec/ovs-monitor-ipsec.in#L167

b) If NAT-T is not enforced, then the default mode is NAT detection:
https://libreswan.org/man/ipsec.conf.5.html
~~~
encapsulation

In some cases, for example when ESP packets are filtered or when a broken IPsec peer does not properly recognise NAT, it can be useful to force RFC-3948 encapsulation. In other cases, where IKE is NAT'ed but ESP packets can or should flow without encapsulation, it can be useful to ignore the NAT-Traversal auto-detection. encapsulation=yes forces the NAT detection code to lie and tell the remote peer that RFC-3948 encapsulation (ESP in port 4500 packets) is required. encapsulation=no ignores the NAT detection causing ESP packets to send send without encapsulation. The default value of encapsulation=auto follows the regular outcome of the NAT auto-detection code performed in IKE. This option replaced the obsoleted forceencaps option. 
~~~

c) If libresawn behaves according to RFCs (which it should), then NAT detection works by hashing IP addresses and ports and comparing the transmitted hash to the receive IP. Given that the communication between your nodes is not NATed, NAT detection will determine that there is no NAT, and thus IPsec will use ESP instead of the UDP ports.

https://datatracker.ietf.org/doc/html/rfc3947#section-3.2
~~~

3.2.  Detecting the Presence of NAT

   The NAT-D payload not only detects the presence of NAT between the
   two IKE peers, but also detects where the NAT is.  The location of
   the NAT device is important, as the keepalives have to initiate from
   the peer "behind" the NAT.

   To detect NAT between the two hosts, we have to detect whether the IP
   address or the port changes along the path.  This is done by sending
   the hashes of the IP addresses and ports of both IKE peers from each
   end to the other.  If both ends calculate those hashes and get same
   result, they know there is no NAT between.  If the hashes do not
   match, somebody has translated the address or port.  This means that
   we have to do NAT-Traversal to get IPsec packets through.

   If the sender of the packet does not know his own IP address (in case
   of multiple interfaces, and the implementation does not know which IP
   address is used to route the packet out), the sender can include
   multiple local hashes to the packet (as separate NAT-D payloads).  In
   this case, NAT is detected if and only if none of the hashes match.

   The hashes are sent as a series of NAT-D (NAT discovery) payloads.
   Each payload contains one hash, so in case of multiple hashes,
   multiple NAT-D payloads are sent.  In the normal case there are only
   two NAT-D payloads.

   The NAT-D payloads are included in the third and fourth packets of
   Main Mode, and in the second and third packets in the Aggressive
   Mode.
~~~
Comment 9 Andreas Karis 2022-01-19 16:53:15 UTC 
Note that host networked communication is not encrypted: https://docs.openshift.com/container-platform/4.9/networking/ovn_kubernetes_network_provider/about-ipsec-ovn.html
Which is why host networked traffic is working.

What's broken though is pod to pod and pod to host communication.

In this patch here, you only unblock UDP 500/4500: https://github.com/openshift/installer/pull/5539

Given that ESP does not seem to make it through, we should shift the bugzilla to the platform side of things.

ESP traffic passing from node to node is a requirement for our current implementation, unless you'd be passing through NAT where NAT-T can be auto-configured.
Comment 10 Andreas Karis 2022-01-19 18:06:06 UTC 
I just hacked this and forced NAT-T:
~~~
oc patch clusterversion version --type json -p '[{"op":"add","path":"/spec/overrides","value":[{"kind":"Deployment","group":"apps","name":"network-operator","namespace":"openshift-network-operator","unmanaged":true}]}]'
oc scale -n openshift-network-operator deployment.apps/network-operator --replicas=0
~~~

Then:
~~~
oc edit ds -n openshift-ovn-kubernetes ovn-ipsec 
~~~

And add the line marked with '+' below:
~~~
          # Environment variables are for workaround for https://mail.openvswitch.org/pipermail/ovs-dev/2020-October/375734.html
          # We now start ovs-monitor-ipsec which will monitor for changes in the ovs
          # tunnelling configuration (for example addition of a node) and configures
          # libreswan appropriately.
+          if ! grep -q encapsulation=yes /usr/share/openvswitch/scripts/ovs-monitor-ipsec ; then sed -i 's/    auto=route/    auto=route\n    encapsulation=yes/' /usr/share/openvswitch/scripts/ovs-monitor-ipsec ; fi
          OVS_LOGDIR=/var/log/openvswitch OVS_RUNDIR=/var/run/openvswitch OVS_PKGDATADIR=/usr/share/openvswitch /usr/share/openvswitch/scripts/ovs-ctl --ike-daemon=libreswan --no-restart-ike-daemon start-ovs-ipsec
~~~

That will add encapsulation=yes to /etc/ipsec.conf.

~~~
18:02:42.665689 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 686: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x4e), length 644
18:02:42.665780 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x3e), length 124
18:02:42.668022 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 1470: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x3f), length 1428
18:02:42.668045 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 1450: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x40), length 1408
18:02:42.669973 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x4f), length 124
18:02:42.670264 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 230: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x50), length 188
18:02:42.670349 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x51), length 124
18:02:43.012103 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 174: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x52), length 132
18:02:43.012649 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 174: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x41), length 132
18:02:43.014719 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x53), length 12
~~~

Then, IPsec actually works and the cluster comes up cleanly:
~~~
[akaris@linux 2041681]$ oc get co
NAME                                       VERSION                                                   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      2m9s    
baremetal                                  4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
cloud-controller-manager                   4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h33m   
cloud-credential                           4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
cluster-autoscaler                         4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
config-operator                            4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h27m   
console                                    4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      5m3s    
csi-snapshot-controller                    4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h26m   
dns                                        4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
etcd                                       4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h24m   
image-registry                             4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h10m   
ingress                                    4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      106m    
insights                                   4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h10m   
kube-apiserver                             4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h8m    
kube-controller-manager                    4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h23m   
kube-scheduler                             4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h23m   
kube-storage-version-migrator              4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h26m   
machine-api                                4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h22m   
machine-approver                           4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
machine-config                             4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h26m   
marketplace                                4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
monitoring                                 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      7m59s   
network                                    4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h24m   
node-tuning                                4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
openshift-apiserver                        4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      8m40s   
openshift-controller-manager               4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h10m   
openshift-samples                          4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      8m32s   
operator-lifecycle-manager                 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
operator-lifecycle-manager-catalog         4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h25m   
operator-lifecycle-manager-packageserver   4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      38m     
service-ca                                 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      4h27m   
storage                                    4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         False      3h53m  
[akaris@linux 2041681]$ oc get clusterversion
NAME      VERSION                                                   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest   True        False         3m14s   Cluster version is 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest
~~~
Comment 14 Andreas Karis 2022-01-20 14:50:44 UTC 
For the missing OVS bits:
https://bugzilla.redhat.com/show_bug.cgi?id=2043057
Comment 17 Ross Brattain 2022-04-12 04:45:46 UTC 
BZ 2038774 is verified, UDP Ports 500 and 4500 are added to the security group during install.

ESP is still required.
Comment 22 Andreas Karis 2022-06-23 08:40:30 UTC 
Closing this as a duplicate of: https://issues.redhat.com/browse/SDN-2629