Description of problem: IBM-Cloud OVN IPsec deployment cannot create workers. The machine-controller fails when trying to provision workers. Specifically the machine-controller is unable to access the IBM API. Initial failures were DNS sh-4.4$ curl -v https://iam.cloud.ibm.com/identity/token * Could not resolve host: iam.cloud.ibm.com Then other http timeouts Error Message: error creating ibm client: Get "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06": dial tcp: i/o timeout Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-17-023213 How reproducible: Always Steps to Reproduce: 1. Deploy IBM-Cloud OVN-IPsec cluster with manual UDP port 500,4500 security group workaround as described in BZ 2038774 2. All three master deploy 3. No workers deploy 4. Test curl from inside machine-controller pod oc exec -n openshift-machine-api -c machine-controller machine-api-controllers-c44c76c84-7xr76 -- curl -v https://iam.cloud.ibm.com/identity/token oc exec -n openshift-machine-api -c machine-controller machine-api-controllers-c44c76c84-z7dvn -- curl -v "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06" Actual results: curl: (6) Could not resolve host: us-south.iaas.cloud.ibm.com error creating ibm client: Get "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06": dial tcp: i/o timeout Status: Conditions: Last Transition Time: 2022-01-18T03:24:27Z Message: Instance has not been created Reason: InstanceNotCreated Severity: Warning Status: False Type: InstanceExists Error Message: error creating ibm client: Get "https://us-south.iaas.cloud.ibm.com/v1/regions/eu-gb?generation=2&version=2021-05-06": dial tcp: i/o timeout Error Reason: InvalidConfiguration Last Updated: 2022-01-18T03:25:07Z Phase: Failed Expected results: Workers are provisioned. Initial failure was a DNS failure oc rsh -n openshift-machine-api -c machine-controller machine-api-controllers-c44c76c84-7xr76 sh-4.4$ curl https://iam.cloud.ibm.com/identity/token curl: (6) Could not resolve host: iam.cloud.ibm.com sh-4.4$ curl -v https://iam.cloud.ibm.com/identity/token * Could not resolve host: iam.cloud.ibm.com * Closing connection 0 curl: (6) Could not resolve host: iam.cloud.ibm.com sh-4.4$
I had a look at the cluster that you shared with me. I checked the tunnels status for a worker and a master node. If you look at the "Traffic: ESPin"/out lines, you can see that the nodes receive no ESP traffic: Worker: ~~~ [akaris@linux 2041681]$ oc get pods -n openshift-ovn-kubernetes -o wide | grep rbrattai-i410i32-c5lwl-worker-3-wqtj9 ovn-ipsec-h6lgk 1/1 Running 0 61m 10.242.128.4 rbrattai-i410i32-c5lwl-worker-3-wqtj9 <none> <none> ovnkube-node-sf2d8 5/5 Running 0 61m 10.242.128.4 rbrattai-i410i32-c5lwl-worker-3-wqtj9 <none> <none> [akaris@linux 2041681]$ oc rsh -n openshift-ovn-kubernetes ovn-ipsec-h6lgk Defaulted container "ovn-ipsec" out of: ovn-ipsec, ovn-keys (init) sh-4.4# sh-4.4# sh-4.4# sh-4.4# ovs-appctl -t ovs-monitor-ipsec tunnels/show Interface name: ovn-e19934-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.129.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: e19934d5-cc17-4d2d-9677-513d1d158190 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 1 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081 sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081 sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp dport 6081 sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081 IPsec connections that are active: 000 #6: "ovn-e19934-0-in-1" esp.a7d1942a.129.4 esp.d43520f1.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #8: "ovn-e19934-0-out-1" esp.2c14043.129.4 esp.76f9751f.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B Interface name: ovn-d2ee6a-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.0.7 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 4 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 IPsec connections that are active: 000 #9: "ovn-d2ee6a-0-in-1" esp.c1c1747f.0.7 esp.df9d7449.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #16: "ovn-d2ee6a-0-out-1" esp.43ef92b6.0.7 esp.501e394d.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B Interface name: ovn-29c4ab-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.64.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: 29c4aba3-da37-4b72-8820-fb3b28aa496a CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 3 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 IPsec connections that are active: 000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B Interface name: ovn-ece7f2-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.1.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: ece7f209-8283-48d7-9060-6a1c1c7ca905 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 13 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 IPsec connections that are active: 000 #24: "ovn-ece7f2-0-in-1" esp.5c0268d2.1.4 esp.52144f65.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #22: "ovn-ece7f2-0-out-1" esp.e8d98c5.1.4 esp.f7d51d81.128.4 Traffic: ESPin=0B ESPout=7KB! ESPmax=0B Interface name: ovn-b7253f-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.65.6 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: b7253f0d-0a4e-4cb4-8c26-aedb77d2567e CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 2 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.65.6/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.65.6/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.65.6/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081 sel src 10.242.65.6/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.65.6/32 proto udp dport 6081 sel src 10.242.65.6/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.65.6/32 proto udp sport 6081 IPsec connections that are active: 000 #13: "ovn-b7253f-0-in-1" esp.25602e91.65.6 esp.f535896e.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #18: "ovn-b7253f-0-out-1" esp.2b2d05e4.65.6 esp.fcc9b0d3.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B sh-4.4# ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep 10.242.64.4 Remote IP: 10.242.64.4 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B sh-4.4# ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep 10.242.64.4 -C20 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 IPsec connections that are active: 000 #9: "ovn-d2ee6a-0-in-1" esp.c1c1747f.0.7 esp.df9d7449.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #16: "ovn-d2ee6a-0-out-1" esp.43ef92b6.0.7 esp.501e394d.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B Interface name: ovn-29c4ab-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.64.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: 29c4aba3-da37-4b72-8820-fb3b28aa496a CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 3 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 IPsec connections that are active: 000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B Interface name: ovn-ece7f2-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.1.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: ece7f209-8283-48d7-9060-6a1c1c7ca905 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 13 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 sh-4.4# reset sh: reset: command not found sh-4.4# ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep 10.242.64.4 -C40 sel src 10.242.129.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.129.4/32 proto udp sport 6081 IPsec connections that are active: 000 #6: "ovn-e19934-0-in-1" esp.a7d1942a.129.4 esp.d43520f1.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #8: "ovn-e19934-0-out-1" esp.2c14043.129.4 esp.76f9751f.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B Interface name: ovn-d2ee6a-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.0.7 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 4 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 IPsec connections that are active: 000 #9: "ovn-d2ee6a-0-in-1" esp.c1c1747f.0.7 esp.df9d7449.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #16: "ovn-d2ee6a-0-out-1" esp.43ef92b6.0.7 esp.501e394d.128.4 Traffic: ESPin=0B ESPout=9KB! ESPmax=0B Interface name: ovn-29c4ab-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.64.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: 29c4aba3-da37-4b72-8820-fb3b28aa496a CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 3 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.64.4/32 proto udp dport 6081 IPsec connections that are active: 000 #28: "ovn-29c4ab-0-in-1" esp.2aa68a6c.64.4 esp.13aadeb4.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #26: "ovn-29c4ab-0-out-1" esp.bbd09776.64.4 esp.43b0a77c.128.4 Traffic: ESPin=0B ESPout=15KB! ESPmax=0B Interface name: ovn-ece7f2-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.1.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: ece7f209-8283-48d7-9060-6a1c1c7ca905 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 13 CFM state: Disabled Kernel policies installed: src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp dport 6081 sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp sport 6081 sel src 10.242.1.4/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.1.4/32 proto udp dport 6081 IPsec connections that are active: 000 #24: "ovn-ece7f2-0-in-1" esp.5c0268d2.1.4 esp.52144f65.128.4 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #22: "ovn-ece7f2-0-out-1" esp.e8d98c5.1.4 esp.f7d51d81.128.4 Traffic: ESPin=0B ESPout=7KB! ESPmax=0B Interface name: ovn-b7253f-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.128.4 Remote IP: 10.242.65.6 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: 6564d5cb-b7cc-4118-9628-116273be4c05 ~~~ Master: ~~~ [akaris@linux 2041681]$ oc get pods -n openshift-ovn-kubernetes -o wide | grep master | grep ipsec ovn-ipsec-4gm54 1/1 Running 0 109m 10.242.0.7 rbrattai-i410i32-c5lwl-master-0 <none> <none> ovn-ipsec-4trz5 1/1 Running 0 109m 10.242.129.4 rbrattai-i410i32-c5lwl-master-2 <none> <none> ovn-ipsec-mbdcl 1/1 Running 0 109m 10.242.65.6 rbrattai-i410i32-c5lwl-master-1 <none> <none> [akaris@linux 2041681]$ oc rsh -n openshift-ovn-kubernetes ovn-ipsec-4gm54 Defaulted container "ovn-ipsec" out of: ovn-ipsec, ovn-keys (init) sh-4.4# sh-4.4# sh-4.4# ovs-appctl -t ovs-monitor-ipsec tunnels/show Interface name: ovn-b7253f-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.0.7 Remote IP: 10.242.65.6 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: b7253f0d-0a4e-4cb4-8c26-aedb77d2567e CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 1 CFM state: Disabled Kernel policies installed: src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081 sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081 sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp dport 6081 sel src 10.242.65.6/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.65.6/32 proto udp sport 6081 IPsec connections that are active: 000 #8: "ovn-b7253f-0-in-1" esp.fe6b8855.65.6 esp.dffc7a81.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #9: "ovn-b7253f-0-out-1" esp.b793764d.65.6 esp.90e9a31c.0.7 Traffic: ESPin=0B ESPout=6MB! ESPmax=0B Interface name: ovn-e19934-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.0.7 Remote IP: 10.242.129.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: e19934d5-cc17-4d2d-9677-513d1d158190 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 3 CFM state: Disabled Kernel policies installed: src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081 Kernel security associations installed: sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081 sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081 sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp dport 6081 sel src 10.242.129.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.129.4/32 proto udp sport 6081 IPsec connections that are active: 000 #11: "ovn-e19934-0-in-1" esp.748a9eb8.129.4 esp.208686e1.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #12: "ovn-e19934-0-out-1" esp.3e8cb223.129.4 esp.d1656111.0.7 Traffic: ESPin=0B ESPout=7MB! ESPmax=0B Interface name: ovn-29c4ab-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.0.7 Remote IP: 10.242.64.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: 29c4aba3-da37-4b72-8820-fb3b28aa496a CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 67 CFM state: Disabled Kernel policies installed: src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.64.4/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.64.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.64.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.64.4/32 proto udp sport 6081 sel src 10.242.64.4/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.64.4/32 proto udp dport 6081 IPsec connections that are active: 000 #34: "ovn-29c4ab-0-in-1" esp.63abb726.64.4 esp.19d08a4f.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #32: "ovn-29c4ab-0-out-1" esp.bd8d4745.64.4 esp.bea13255.0.7 Traffic: ESPin=0B ESPout=742KB! ESPmax=0B Interface name: ovn-6564d5-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.0.7 Remote IP: 10.242.128.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: 6564d5cb-b7cc-4118-9628-116273be4c05 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 71 CFM state: Disabled Kernel policies installed: src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp sport 6081 sel src 10.242.128.4/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.128.4/32 proto udp dport 6081 IPsec connections that are active: 000 #21: "ovn-6564d5-0-in-1" esp.501e394d.128.4 esp.43ef92b6.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #19: "ovn-6564d5-0-out-1" esp.df9d7449.128.4 esp.c1c1747f.0.7 Traffic: ESPin=0B ESPout=725KB! ESPmax=0B Interface name: ovn-ece7f2-0 v1 (CONFIGURED) Tunnel Type: geneve Local IP: 10.242.0.7 Remote IP: 10.242.1.4 Address Family: IPv4 SKB mark: None Local cert: /etc/openvswitch/keys/ipsec-cert.pem Local name: d2ee6a39-13b9-47d8-93a9-7c6375caf5b1 Local key: /etc/openvswitch/keys/ipsec-privkey.pem Remote cert: None Remote name: ece7f209-8283-48d7-9060-6a1c1c7ca905 CA cert: /etc/openvswitch/keys/ipsec-cacert.pem PSK: None Ofport: 75 CFM state: Disabled Kernel policies installed: src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081 src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081 src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081 Kernel security associations installed: sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081 sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081 sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp dport 6081 sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp sport 6081 sel src 10.242.1.4/32 dst 10.242.0.7/32 proto udp sport 6081 sel src 10.242.0.7/32 dst 10.242.1.4/32 proto udp dport 6081 IPsec connections that are active: 000 #30: "ovn-ece7f2-0-in-1" esp.3cae9761.1.4 esp.f0a36212.0.7 Traffic: ESPin=0B ESPout=0B! ESPmax=0B 000 #29: "ovn-ece7f2-0-out-1" esp.11806853.1.4 esp.9a378fdb.0.7 Traffic: ESPin=0B ESPout=22KB! ESPmax=0B ~~~ I ran tcpdumps on br-ex once, and also once on ens3, on 2 of your worker nodes. You can see that ESP traffic only passes in the outbound direction, it is not received: ~~~ [akaris@linux 2041681]$ tshark -t ad -nn -r brattai-i410i32-c5lwl-worker-2-rmjxv.br-ex.pcap 1 2022-01-19 16:07:18.448110 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 2 2022-01-19 16:07:23.447542 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 3 2022-01-19 16:07:28.447705 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) [akaris@linux 2041681]$ tshark -t ad -nn -r rbrattai-i410i32-c5lwl-worker-3-wqtj9.br-ex.pcap 1 2022-01-19 16:07:10.344317 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) 2 2022-01-19 16:07:15.343699 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) 3 2022-01-19 16:07:20.343927 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) 4 2022-01-19 16:07:25.353964 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) 5 2022-01-19 16:07:30.353752 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) 6 2022-01-19 16:07:35.353849 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) [akaris@linux 2041681]$ tshark -t ad -nn -r rbrattai-i410i32-c5lwl-worker-2-rmjxv.ens3.pcap 1 2022-01-19 16:20:18.747957 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 2 2022-01-19 16:20:23.747610 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 3 2022-01-19 16:20:28.747870 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 4 2022-01-19 16:21:43.779100 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 5 2022-01-19 16:21:48.778549 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) 6 2022-01-19 16:21:53.778824 02:00:01:30:ee:01 02:00:00:30:ee:01 10.242.64.4 → 10.242.128.4 ESP 234 ESP (SPI=0x13aadeb4) [akaris@linux 2041681]$ tshark -t ad -nn -r rbrattai-i410i32-c5lwl-worker-3-wqtj9.ens3x.pcap 1 2022-01-19 16:19:00.627303 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) 2 2022-01-19 16:19:05.627416 02:00:01:30:69:fe 02:00:00:30:69:fe 10.242.128.4 → 10.242.64.4 ESP 234 ESP (SPI=0xbbd09776) [akaris@linux 2041681]$ ~~~ From what we talked about, you only unblocked UDP 500 and UDP 4500, but ESP is not unblocked. I have strong reasons to believe that you *must* unblock ESP explicitly: a) ovs-monitor-ipsec does not enforce NAT-T when setting up the ipsec tunnels: https://github.com/openvswitch/ovs/blob/master/ipsec/ovs-monitor-ipsec.in#L167 b) If NAT-T is not enforced, then the default mode is NAT detection: https://libreswan.org/man/ipsec.conf.5.html ~~~ encapsulation In some cases, for example when ESP packets are filtered or when a broken IPsec peer does not properly recognise NAT, it can be useful to force RFC-3948 encapsulation. In other cases, where IKE is NAT'ed but ESP packets can or should flow without encapsulation, it can be useful to ignore the NAT-Traversal auto-detection. encapsulation=yes forces the NAT detection code to lie and tell the remote peer that RFC-3948 encapsulation (ESP in port 4500 packets) is required. encapsulation=no ignores the NAT detection causing ESP packets to send send without encapsulation. The default value of encapsulation=auto follows the regular outcome of the NAT auto-detection code performed in IKE. This option replaced the obsoleted forceencaps option. ~~~ c) If libresawn behaves according to RFCs (which it should), then NAT detection works by hashing IP addresses and ports and comparing the transmitted hash to the receive IP. Given that the communication between your nodes is not NATed, NAT detection will determine that there is no NAT, and thus IPsec will use ESP instead of the UDP ports. https://datatracker.ietf.org/doc/html/rfc3947#section-3.2 ~~~ 3.2. Detecting the Presence of NAT The NAT-D payload not only detects the presence of NAT between the two IKE peers, but also detects where the NAT is. The location of the NAT device is important, as the keepalives have to initiate from the peer "behind" the NAT. To detect NAT between the two hosts, we have to detect whether the IP address or the port changes along the path. This is done by sending the hashes of the IP addresses and ports of both IKE peers from each end to the other. If both ends calculate those hashes and get same result, they know there is no NAT between. If the hashes do not match, somebody has translated the address or port. This means that we have to do NAT-Traversal to get IPsec packets through. If the sender of the packet does not know his own IP address (in case of multiple interfaces, and the implementation does not know which IP address is used to route the packet out), the sender can include multiple local hashes to the packet (as separate NAT-D payloads). In this case, NAT is detected if and only if none of the hashes match. The hashes are sent as a series of NAT-D (NAT discovery) payloads. Each payload contains one hash, so in case of multiple hashes, multiple NAT-D payloads are sent. In the normal case there are only two NAT-D payloads. The NAT-D payloads are included in the third and fourth packets of Main Mode, and in the second and third packets in the Aggressive Mode. ~~~
Note that host networked communication is not encrypted: https://docs.openshift.com/container-platform/4.9/networking/ovn_kubernetes_network_provider/about-ipsec-ovn.html Which is why host networked traffic is working. What's broken though is pod to pod and pod to host communication. In this patch here, you only unblock UDP 500/4500: https://github.com/openshift/installer/pull/5539 Given that ESP does not seem to make it through, we should shift the bugzilla to the platform side of things. ESP traffic passing from node to node is a requirement for our current implementation, unless you'd be passing through NAT where NAT-T can be auto-configured.
I just hacked this and forced NAT-T: ~~~ oc patch clusterversion version --type json -p '[{"op":"add","path":"/spec/overrides","value":[{"kind":"Deployment","group":"apps","name":"network-operator","namespace":"openshift-network-operator","unmanaged":true}]}]' oc scale -n openshift-network-operator deployment.apps/network-operator --replicas=0 ~~~ Then: ~~~ oc edit ds -n openshift-ovn-kubernetes ovn-ipsec ~~~ And add the line marked with '+' below: ~~~ # Environment variables are for workaround for https://mail.openvswitch.org/pipermail/ovs-dev/2020-October/375734.html # We now start ovs-monitor-ipsec which will monitor for changes in the ovs # tunnelling configuration (for example addition of a node) and configures # libreswan appropriately. + if ! grep -q encapsulation=yes /usr/share/openvswitch/scripts/ovs-monitor-ipsec ; then sed -i 's/ auto=route/ auto=route\n encapsulation=yes/' /usr/share/openvswitch/scripts/ovs-monitor-ipsec ; fi OVS_LOGDIR=/var/log/openvswitch OVS_RUNDIR=/var/run/openvswitch OVS_PKGDATADIR=/usr/share/openvswitch /usr/share/openvswitch/scripts/ovs-ctl --ike-daemon=libreswan --no-restart-ike-daemon start-ovs-ipsec ~~~ That will add encapsulation=yes to /etc/ipsec.conf. ~~~ 18:02:42.665689 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 686: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x4e), length 644 18:02:42.665780 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x3e), length 124 18:02:42.668022 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 1470: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x3f), length 1428 18:02:42.668045 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 1450: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x40), length 1408 18:02:42.669973 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x4f), length 124 18:02:42.670264 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 230: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x50), length 188 18:02:42.670349 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x51), length 124 18:02:43.012103 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 174: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x52), length 132 18:02:43.012649 02:00:01:30:ee:01 > 02:00:00:30:ee:01, ethertype IPv4 (0x0800), length 174: 10.242.64.4.4500 > 10.242.128.4.4500: UDP-encap: ESP(spi=0xe6114b91,seq=0x41), length 132 18:02:43.014719 02:00:00:30:ee:01 > 02:00:01:30:ee:01, ethertype IPv4 (0x0800), length 166: 10.242.128.4.4500 > 10.242.64.4.4500: UDP-encap: ESP(spi=0xf1b87090,seq=0x53), length 12 ~~~ Then, IPsec actually works and the cluster comes up cleanly: ~~~ [akaris@linux 2041681]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 2m9s baremetal 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m cloud-controller-manager 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h33m cloud-credential 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m cluster-autoscaler 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m config-operator 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h27m console 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 5m3s csi-snapshot-controller 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h26m dns 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m etcd 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h24m image-registry 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h10m ingress 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 106m insights 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h10m kube-apiserver 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h8m kube-controller-manager 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h23m kube-scheduler 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h23m kube-storage-version-migrator 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h26m machine-api 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h22m machine-approver 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m machine-config 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h26m marketplace 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m monitoring 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 7m59s network 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h24m node-tuning 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m openshift-apiserver 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 8m40s openshift-controller-manager 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h10m openshift-samples 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 8m32s operator-lifecycle-manager 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m operator-lifecycle-manager-catalog 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h25m operator-lifecycle-manager-packageserver 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 38m service-ca 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 4h27m storage 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False False 3h53m [akaris@linux 2041681]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest True False 3m14s Cluster version is 4.10.0-0.ci.test-2022-01-19-130718-ci-ln-1bs30mt-latest ~~~
For the missing OVS bits: https://bugzilla.redhat.com/show_bug.cgi?id=2043057
BZ 2038774 is verified, UDP Ports 500 and 4500 are added to the security group during install. ESP is still required.
Closing this as a duplicate of: https://issues.redhat.com/browse/SDN-2629