Version: 4.10.0-0.nightly-2022-01-08-215919 Platform: IPI IBM Cloud What happened? Installation fails with OVN IPsec. It looks like the IPsec ports are not in the security group. see https://github.com/openshift/installer/pull/4552 for Ports UDP port 500 IPsec IKE UDP port 4500 IPsec NAT-T IP proto 50 ESP I don't see the ports in the TF https://github.com/openshift/installer/blob/master/data/data/ibmcloud/network/vpc/security-groups.tf Also not in the terraform.network.tfstate What did you expect to happen? Cluster deploys successfully, OVN traffic uses IPsec How to reproduce it (as minimally and precisely as possible)? create manifests edit manifests/cluster-network-03-config.yml apiVersion: operator.openshift.io/v1 kind: Network metadata: name: cluster spec: defaultNetwork: ovnKubernetesConfig: ipsecConfig: {} See https://docs.openshift.com/container-platform/4.9/installing/installing_aws/installing-aws-network-customizations.html#modifying-nwoperator-config-startup_installing-aws-network-customizations for examples.
TF from AWS https://github.com/openshift/installer/blob/master/data/data/aws/cluster/vpc/sg-master.tf#L127
As this only effects installations to IBM Cloud using OVN IPSec and there is a workaround of the user manually opening the ports in the security group, I am marking this as not a blocker.
Attempting manual workaround. In the web-console I don't see an option to specify IP protocol 50 (ESP) The command line doesn't seem to allow ESP protocol. ibmcloud is security-group-rule-add z8h-sg-cluster-wide inbound esp --remote z8h-sg-cluster-wide FAILED Incorrect Usage: Invalid protocol. valid values are all |icmp |tcp | udp. NAME: security-group-rule-add - Add a rule to a security group USAGE: ibmcloud is security-group-rule-add GROUP DIRECTION PROTOCOL [--vpc VPC] [--remote REMOTE_ADDRESS | CIDR_BLOCK | SECURITY_GROUP] [--icmp-type ICMP_TYPE [--icmp-code ICMP_CODE]] [--port-min PORT_MIN] [--port-max PORT_MAX] [--output JSON] [-q, --quiet] GROUP: ID or name of the security group. DIRECTION: Direction of traffic to enforce. One of: inbound, outbound. PROTOCOL: Protocol to enforce. One of: all, icmp, tcp, udp. EXAMPLE: ibmcloud is security-group-rule-add 72b27b5c-f4b0-48bb-b954-5becc7c1dcb3 inbound all ibmcloud is security-group-rule-add my-sg inbound all ibmcloud is security-group-rule-add my-sg inbound all --vpc my-vpc ibmcloud is security-group-rule-add 72b27b5c-f4b0-48bb-b954-5becc7c1dcb3 inbound icmp --icmp-type 8 --icmp-code 0 ibmcloud is security-group-rule-add 72b27b5c-f4b0-48bb-b954-5becc7c1dcb3 inbound all --remote 12.2.2.3 ibmcloud is security-group-rule-add 72b27b5c-f4b0-48bb-b954-5becc7c1dcb3 inbound all --remote 72b27b5c-f4b0-48bb-b954-5becc7c1dcb3 ibmcloud is security-group-rule-add my-sg inbound all --remote my-sg ibmcloud is security-group-rule-add 72b27b5c-f4b0-48bb-b954-5becc7c1dcb3 inbound tcp --port-min 4 --port-max 22 --output JSON OPTIONS: --vpc value ID or name of the VPC. It is only required to specify the unique resource by name inside this VPC. --remote value The set of network interfaces from which this rule allows traffic. It can be specified as either a REMOTE_ADDRESS, CIDR_BLOCK or SECURITY_GROUP. If unspecified, then traffic is allowed from any source (or to any source, for outbound rules) --icmp-type value ICMP traffic type to allow. Valid values from 0 to 254. This option is specified only when protocol is set to icmp. If unspecified, all types are allowed --icmp-code value ICMP traffic code to allow. Valid values from 0 to 255. This option is specified only when protocol is set to icmp. If unspecified, all codes are allowed --port-min value Minimum port number. Valid values are from 1 to 65535. This option is specified only when protocol is set to tcp or udp. If unspecified, all ports are allowed (default: 1) --port-max value Maximum port number. Valid values are from 1 to 65535. This option is specified only when protocol is set to tcp or udp. If unspecified, all ports are allowed (default: 65535) --output value Specify output format, only JSON is supported. One of: JSON. -q, --quiet Suppress verbose output
Partially verified on 4.11.0-0.nightly-2022-04-11-200046 Port 500, 4500 UDP added but ESP is not supported. Will create a new BZ for ESP. Listing rules of security group o411i15-pbpm8-sg-cluster-wide ID Direction IP version Protocol Remote r018-267defaf-7e43-4ca7-9549-6dbf3733f159 inbound ipv4 tcp Ports:Min=22,Max=22 o411i15-pbpm8-sg-cluster-wide r018-14c310a6-9128-42bc-8eb3-8814a142a67f outbound ipv4 all 0.0.0.0/0 r018-62b4b5b0-1fca-4596-a732-007508d10a4a inbound ipv4 icmp o411i15-pbpm8-sg-cluster-wide r018-00b58973-3fad-49a8-bd80-5087c0f5a3a7 inbound ipv4 udp Ports:Min=4789,Max=4789 o411i15-pbpm8-sg-cluster-wide r018-24729f21-de81-46b7-b4f1-7f7a2783fe33 inbound ipv4 udp Ports:Min=6081,Max=6081 o411i15-pbpm8-sg-cluster-wide Listing rules of security group o411i15-pbpm8-sg-control-plane ID Direction IP version Protocol Remote r018-af65d42c-39b4-4561-9a33-c512be57fadf inbound ipv4 tcp Ports:Min=6443,Max=6443 o411i15-pbpm8-sg-kube-api-lb r018-073a35a4-8e37-4baf-852a-8ca0efe78584 inbound ipv4 tcp Ports:Min=22623,Max=22623 o411i15-pbpm8-sg-kube-api-lb r018-42ae545e-8eea-4298-8583-bb0d0a43ac6e inbound ipv4 tcp Ports:Min=6443,Max=6443 o411i15-pbpm8-sg-cluster-wide Listing rules of security group o411i15-pbpm8-sg-cp-internal ID Direction IP version Protocol Remote r018-f697645e-1b43-4ee5-98d9-46a170d6b8b9 inbound ipv4 tcp Ports:Min=2379,Max=2380 o411i15-pbpm8-sg-cp-internal r018-0eef722a-3933-4c4b-a09f-8d59869ea01e inbound ipv4 tcp Ports:Min=10257,Max=10259 o411i15-pbpm8-sg-cp-internal Listing rules of security group o411i15-pbpm8-sg-kube-api-lb ID Direction IP version Protocol Remote r018-9e94b111-3804-4f56-bf64-5f73f7feb684 outbound ipv4 tcp Ports:Min=22623,Max=22623 o411i15-pbpm8-sg-control-plane r018-1a5cdb2e-de29-4b8f-885f-b6f5e335a2b1 outbound ipv4 tcp Ports:Min=6443,Max=6443 o411i15-pbpm8-sg-control-plane r018-bcaa374d-30c3-4c59-b3f4-d56e8d81b198 inbound ipv4 tcp Ports:Min=22623,Max=22623 o411i15-pbpm8-sg-cluster-wide r018-e72c8b8d-55bf-4480-9040-ec973f24dc03 inbound ipv4 tcp Ports:Min=6443,Max=6443 0.0.0.0/0 Listing rules of security group o411i15-pbpm8-sg-openshift-net ID Direction IP version Protocol Remote r018-5de10c58-f84d-4672-98e9-fc72fd7b7f2a inbound ipv4 tcp Ports:Min=10250,Max=10250 o411i15-pbpm8-sg-openshift-net r018-a37f8df2-adfc-43bc-b25a-11fef3fa1435 inbound ipv4 udp Ports:Min=9000,Max=9999 o411i15-pbpm8-sg-openshift-net r018-80759546-15f9-4931-8495-ae9501ea3c71 inbound ipv4 udp Ports:Min=500,Max=500 o411i15-pbpm8-sg-openshift-net r018-6d2f58f6-a4b6-4493-bce5-a9b766365c80 inbound ipv4 tcp Ports:Min=9000,Max=9999 o411i15-pbpm8-sg-openshift-net r018-a59f2381-f15c-4db7-83bc-27dbcb6479b8 inbound ipv4 udp Ports:Min=4500,Max=4500 o411i15-pbpm8-sg-openshift-net r018-0d9b1c7c-3858-4cbc-92ea-cb7af9cda311 inbound ipv4 tcp Ports:Min=30000,Max=32767 10.242.129.0/24 r018-11216ed3-40ee-426f-8f00-7913fc8d0dfd inbound ipv4 tcp Ports:Min=30000,Max=32767 10.242.64.0/24 r018-62453dbb-4b7f-4521-9cfa-0b063bfffbec inbound ipv4 udp Ports:Min=30000,Max=32767 10.242.1.0/24 r018-943a7270-b992-44a8-89ef-348339250e6c inbound ipv4 tcp Ports:Min=30000,Max=32767 10.242.0.0/24 r018-685c934a-1dcf-4c4a-a8ab-4b90e68d813c inbound ipv4 tcp Ports:Min=30000,Max=32767 10.242.1.0/24 r018-7a88fd34-19ca-44a7-90e8-56178b074ba7 inbound ipv4 tcp Ports:Min=30000,Max=32767 10.242.128.0/24 r018-8c360c75-2f3e-461f-b48b-791c85585229 inbound ipv4 tcp Ports:Min=30000,Max=32767 10.242.65.0/24 r018-b9bce344-8253-4f2e-80cb-f9b02372dbd8 inbound ipv4 udp Ports:Min=30000,Max=32767 10.242.128.0/24 r018-75d700ba-8862-49b2-95e3-b890efec6756 inbound ipv4 udp Ports:Min=30000,Max=32767 10.242.64.0/24 r018-54cab605-cb8a-4a0d-b6ec-b1f6bb798527 inbound ipv4 udp Ports:Min=30000,Max=32767 10.242.129.0/24 r018-70588661-42de-4f19-86c9-cd8c08324c4c inbound ipv4 udp Ports:Min=30000,Max=32767 10.242.65.0/24 r018-ef4e7ad4-9eb7-44c2-9abb-89a3e303f090 inbound ipv4 udp Ports:Min=30000,Max=32767 10.242.0.0/24
BZ 204168 is the overall IBM-Cloud OVN IPsec install issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069