Hide Forgot
Created attachment 1847602 [details] must-gather Description of problem: The OCP installation on top of Openstack is failing as the workers are no attached to the cluster. The workers have an extra network that is attached to all of them (for integration with Manila) and it is provoking a wrong routing table on the workers, so they are not able to connect to internet an download the images to get configured. The install-config.yaml looks like below: $ cat install-config.yaml # This file is autogenerated by infrared openshift plugin apiVersion: v1 baseDomain: "shiftstack.com" compute: - name: worker platform: openstack: zones: [] additionalNetworkIDs: ['c1185e5e-ac6d-4548-bafc-d7326b261c5c'] replicas: 3 controlPlane: name: master platform: openstack: zones: [] replicas: 3 metadata: name: "ostest" networking: clusterNetworks: - cidr: 10.128.0.0/14 hostSubnetLength: 9 serviceNetwork: - 172.30.0.0/16 machineNetwork: - cidr: "10.196.0.0/16" networkType: "OVNKubernetes" [...] where: $ openstack subnet list | grep c1185e5e-ac6d-4548-bafc-d7326b261c5c | 3d24e0d0-05ca-4659-ada7-86f83b2eceb1 | StorageNFSSubnet | c1185e5e-ac6d-4548-bafc-d7326b261c5c | 172.17.5.0/24 | Before installation we are manipulating the manifest to enable the externalCCM on it: apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: annotations: include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" release.openshift.io/create-only: "true" name: cluster spec: customNoUpgrade: enabled: - ExternalCloudProvider - CSIMigrationOpenStack featureSet: CustomNoUpgrade The installation is failing due to the machines remaining on provisioned status, and the node resource is not created: (shiftstack) [stack@undercloud-0 ~]$ oc get machines -n openshift-machine-api NAME PHASE TYPE REGION ZONE AGE ostest-td7wn-master-0 Running 19h ostest-td7wn-master-1 Running 19h ostest-td7wn-master-2 Running 19h ostest-td7wn-worker-0-64prw Provisioned 19h ostest-td7wn-worker-0-78pvd Provisioned m4.xlarge regionOne nova 4h33m ostest-td7wn-worker-0-rxccv Provisioned m4.xlarge regionOne nova 19h (shiftstack) [stack@undercloud-0 ~]$ oc get nodes NAME STATUS ROLES AGE VERSION ostest-td7wn-master-0 Ready master 19h v1.22.1+6859754 ostest-td7wn-master-1 Ready master 19h v1.22.1+6859754 ostest-td7wn-master-2 Ready master 19h v1.22.1+6859754 (shiftstack) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+-----------------------------+--------+--------------------------------------------------------------+--------------------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------------+--------+--------------------------------------------------------------+--------------------+--------+ | 0c792e79-481e-4653-953f-d3b4e451834c | ostest-td7wn-worker-0-78pvd | ACTIVE | StorageNFS=172.17.5.164; ostest-td7wn-openshift=10.196.2.210 | ostest-td7wn-rhcos | | | 835d86d4-16ce-4d72-82cd-658aa61ff354 | ostest-td7wn-worker-0-rxccv | ACTIVE | StorageNFS=172.17.5.209; ostest-td7wn-openshift=10.196.2.187 | ostest-td7wn-rhcos | | | 16bf0272-aa74-4c0c-8769-990756163201 | ostest-td7wn-worker-0-64prw | ACTIVE | StorageNFS=172.17.5.193; ostest-td7wn-openshift=10.196.2.119 | ostest-td7wn-rhcos | | | c7f64b3a-48a9-489c-bab7-1e1356e149fd | ostest-td7wn-master-2 | ACTIVE | ostest-td7wn-openshift=10.196.1.25 | ostest-td7wn-rhcos | | | f1e579f3-8457-47c3-9b7b-8fa302c8e2d5 | ostest-td7wn-master-1 | ACTIVE | ostest-td7wn-openshift=10.196.2.93 | ostest-td7wn-rhcos | | | 8415da41-4c86-4d48-ac5f-6def6b30bea2 | ostest-td7wn-master-0 | ACTIVE | ostest-td7wn-openshift=10.196.0.150 | ostest-td7wn-rhcos | | +--------------------------------------+-----------------------------+--------+--------------------------------------------------------------+--------------------+--------+ The workers have no access to internet due to a routing issue: (shiftstack) [stack@undercloud-0 ~]$ ssh -J core.43.181 core.2.187 Warning: Permanently added '10.46.43.181' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.2.187' (ECDSA) to the list of known hosts. Red Hat Enterprise Linux CoreOS 410.84.202112201602-0 Part of OpenShift 4.10, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.10/architecture/architecture-rhcos.html --- [core@ostest-td7wn-worker-0-rxccv ~]$ sudo netstat -nrv Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 172.17.5.1 0.0.0.0 UG 0 0 0 ens4 0.0.0.0 10.196.0.1 0.0.0.0 UG 0 0 0 br-ex 10.196.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-ex 169.254.169.254 172.17.5.150 255.255.255.255 UGH 0 0 0 ens4 169.254.169.254 10.196.0.10 255.255.255.255 UGH 0 0 0 br-ex 172.17.5.0 0.0.0.0 255.255.255.0 U 0 0 0 ens4 [core@ostest-td7wn-worker-0-rxccv ~]$ ip r default via 172.17.5.1 dev ens4 proto dhcp metric 100 default via 10.196.0.1 dev br-ex proto dhcp metric 100 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.2.187 metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.209 metric 100 [core@ostest-td7wn-worker-0-rxccv ~]$ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ^C --- 8.8.8.8 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1032ms The first entry on the routing table is sending the default traffic to the manila network, instead of the machine network. Removing the ens4 default route fix the issue: [core@ostest-td7wn-worker-0-rxccv ~]$ sudo ip route delete default dev ens4 [core@ostest-td7wn-worker-0-rxccv ~]$ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=41.5 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=40.3 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=39.8 ms ^C --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 39.785/40.544/41.536/0.733 ms The same type of installation is working using: 1. the in-tree Cloud Config Manager with OVNKubernetes 2. ExternalCloudProvider and OpenShiftSDN NetworkType. In the second case, the routes on the worker looks like this: [core@ostest-46nr7-worker-0-xmlcw ~]$ ip r default via 10.196.0.1 dev ens3 proto dhcp metric 100 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 dev tun0 scope link 10.196.0.0/16 dev ens3 proto kernel scope link src 10.196.0.65 metric 100 169.254.169.254 via 10.196.0.10 dev ens3 proto dhcp metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.164 metric 101 172.30.0.0/16 dev tun0 where the defaults are created with different metrics. Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2021-12-21-130047 on top of RHOS-16.2-RHEL-8-20211129.n.1 with Manila and OVN enabled. How reproducible: Always. Steps to Reproduce: 1. Install as detailed on description. 2. 3. Actual results: Installation fails. Expected results: Installation OK. Additional info: - must-gather. - install-config.yaml
Hi there, Can you provide the `openstack network show`, `openstack subnet show`, `openstack router show` command for the networks, subnets, routers in question? Please also add the `openstack port show` for all worker and master node ports, as well as the `openstack server show` for all master and worker nodes. How are you telling OpenStack / OpenShift which of the 2 default routes should have a better metric than the other? Which version of OpenStack are you using underneath OpenShift? Thanks! - Andreas
I managed to grab an OpenStack 16.2 system (all virtual, so it's not powerful enough for an OpenShift deployment) and I created an instance with (incomplete reproducer steps): ~~~ openstack network create akaris1 openstack network create akaris2 openstack subnet create akaris1-subnet --network akaris1 --subnet-range 192.168.20.0/24 --dns-nameserver 10.5.30.45 openstack subnet create akaris2-subnet --network akaris2 --subnet-range 192.168.21.0/24 --dns-nameserver 10.5.30.45 openstack router add subnet test-router-andrea akaris1-subnet openstack router add subnet test-router-andrea akaris2-subnet openstack server create --image rhel8 --flavor nano --nic net-id=akaris1 --nic net-id=akaris2 rhel8 openstack floating ip set --port 9891d857-6c6e-4e05-a090-c5d58d291c7a 10.0.0.172 ~~~ It turns out that is actually a NetworkManager feature and neither one from OpenStack nor form OpenShift or OVN-Kubernetes. Nothing in the interface configuration or the DHCP leases which are handed out from dnsmasq actually sets a metric. Instead: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/managing-the-default-gateway-setting_configuring-and-managing-networking ~~~ 21.8. How NetworkManager manages multiple default gateways In certain situations, for example for fallback reasons, you set multiple default gateways on a host. However, to avoid asynchronous routing issues, each default gateway of the same protocol requires a separate metric value. Note that RHEL only uses the connection to the default gateway that has the lowest metric set. You can set the metric for both the IPv4 and IPv6 gateway of a connection using the following command: # nmcli connection modify connection-name ipv4.route-metric value ipv6.route-metric value Important Do not set the same metric value for the same protocol in multiple connection profiles to avoid routing issues. If you set a default gateway without a metric value, NetworkManager automatically sets the metric value based on the interface type. For that, NetworkManager assigns the default value of this network type to the first connection that is activated, and sets an incremented value to each other connection of the same type in the order they are activated. For example, if two Ethernet connections with a default gateway exist, NetworkManager sets a metric of 100 on the route to the default gateway of the connection that you activate first. For the second connection, NetworkManager sets 101. ~~~ For OpenShiftSDN, this works because both interfaces are of the same type Ethernet. The first attached network is the machine network so it gets 100, and the second attached network is the manila network, so it gets a route metric of 101. However, there's really no guarantee that this works. It's just thanks to order of operations that the correct metric is assigned to the respective interfaces. For OVNKubernetes this fails because br-ex and ens4 are of 2 different types. The conversion happens here https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml#L511 We force a metric of 100 for br-ex here in the conversion process: https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml#L183 That's the same as the default ethernet route metric. Given that ens4 is now the only ethernet type interface, it's metric is lowered from 101 to 100, and we suddently have 2 routes with the same metric of value 100. To test this further, I installed openvswitch2.16, NetworkManager-ovs and I converted the interfaces on my RHEL instance with the same script that's used by OpenShift: https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml In order to reset the interfaces after each test, I used the following procedure: ~~~ # delete all connections nmcli conn delete <...> cloud-init clean cloud-init init # if needed copy ifcfg-eth0 script to ifcfg-eth1 systemctl restart NetworkManager ~~~ ~~~ [root@rhel8 ~]# ip r default via 192.168.20.1 dev eth0 proto dhcp metric 100 default via 192.168.21.1 dev eth1 proto dhcp metric 101 169.254.169.254 via 192.168.20.2 dev eth0 proto dhcp metric 100 169.254.169.254 via 192.168.21.2 dev eth1 proto dhcp metric 101 192.168.20.0/24 dev eth0 proto kernel scope link src 192.168.20.80 metric 100 192.168.21.0/24 dev eth1 proto kernel scope link src 192.168.21.167 metric 101 ~~~ ~~~ /usr/local/bin/ovs-configure.sh ~~~ Right after executing the script: ~~~ # ip r default via 192.168.21.1 dev eth1 proto dhcp metric 100 default via 192.168.20.1 dev br-ex proto dhcp metric 100 169.254.169.254 via 192.168.21.2 dev eth1 proto dhcp metric 100 169.254.169.254 via 192.168.20.2 dev br-ex proto dhcp metric 100 192.168.20.0/24 dev br-ex proto kernel scope link src 192.168.20.80 metric 100 192.168.21.0/24 dev eth1 proto kernel scope link src 192.168.21.167 metric 100 ~~~ ~~~ # nmcli conn show br-ex (...) connection.type: ovs-bridge (...) ~~~ ~~~ # nmcli conn show 'System eth1' (...) connection.type: 802-3-ethernet (...) ~~~ A very similar issue was already addressed here: https://bugzilla.redhat.com/show_bug.cgi?id=1880259 Fixes to that BZ which set the metric to 100: https://github.com/openshift/machine-config-operator/commit/974bb6fae2cb3bc0beff8f43e3879cf22d95c148 https://github.com/openshift/machine-config-operator/commit/d6326cd110bcfacfb52912f4696335ccefc193b7 Without the above fixes, the routing table had a metric of 800: ~~~ # ip r default via 192.168.21.1 dev eth1 proto dhcp metric 100 default via 192.168.20.1 dev br-ex proto dhcp metric 800 169.254.169.254 via 192.168.21.2 dev eth1 proto dhcp metric 100 169.254.169.254 via 192.168.20.2 dev br-ex proto dhcp metric 800 192.168.20.0/24 dev br-ex proto kernel scope link src 192.168.20.80 metric 800 192.168.21.0/24 dev eth1 proto kernel scope link src 192.168.21.167 metric 100 ~~~ Simply lowering the metric to e.g. 50 should be the easiest fix for this and would be in line with the intention behind the above fixes.
*** Bug 2036112 has been marked as a duplicate of this bug. ***
@rlobillo Could you help verify this bug ,thanks.
Verified on OCP4.10.0-0.nightly-2022-01-25-023600 on top of OSP16.1 (RHOS-16.1-RHEL-8-20210903.n.0) Regular IPI installation with OVNKubernetes Network types finished successfully: $ tail ~/ostest/.openshift_install.log time="2022-01-25T08:14:26Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.ostest.shiftstack.com" time="2022-01-25T08:14:26Z" level=info msg="Login to the console with user: \"kubeadmin\", and password: \"TUMeD-NnoG2-547Nr-MZrp7\"" time="2022-01-25T08:14:26Z" level=debug msg="Time elapsed per stage:" time="2022-01-25T08:14:26Z" level=debug msg=" masters: 2m17s" time="2022-01-25T08:14:26Z" level=debug msg=" bootstrap: 44s" time="2022-01-25T08:14:26Z" level=debug msg="Bootstrap Complete: 16m0s" time="2022-01-25T08:14:26Z" level=debug msg=" API: 2m50s" time="2022-01-25T08:14:26Z" level=debug msg=" Bootstrap Destroy: 34s" time="2022-01-25T08:14:26Z" level=debug msg=" Cluster Operators: 21m16s" time="2022-01-25T08:14:26Z" level=info msg="Time elapsed: 47m13s" $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.10.0-0.nightly-2022-01-25-023600 True False False 76m baremetal 4.10.0-0.nightly-2022-01-25-023600 True False False 100m cloud-controller-manager 4.10.0-0.nightly-2022-01-25-023600 True False False 105m cloud-credential 4.10.0-0.nightly-2022-01-25-023600 True False False 107m cluster-autoscaler 4.10.0-0.nightly-2022-01-25-023600 True False False 100m config-operator 4.10.0-0.nightly-2022-01-25-023600 True False False 101m console 4.10.0-0.nightly-2022-01-25-023600 True False False 75m csi-snapshot-controller 4.10.0-0.nightly-2022-01-25-023600 True False False 100m dns 4.10.0-0.nightly-2022-01-25-023600 True False False 100m etcd 4.10.0-0.nightly-2022-01-25-023600 True False False 99m image-registry 4.10.0-0.nightly-2022-01-25-023600 True False False 82m ingress 4.10.0-0.nightly-2022-01-25-023600 True False False 82m insights 4.10.0-0.nightly-2022-01-25-023600 True False False 94m kube-apiserver 4.10.0-0.nightly-2022-01-25-023600 True False False 80m kube-controller-manager 4.10.0-0.nightly-2022-01-25-023600 True False False 98m kube-scheduler 4.10.0-0.nightly-2022-01-25-023600 True False False 98m kube-storage-version-migrator 4.10.0-0.nightly-2022-01-25-023600 True False False 101m machine-api 4.10.0-0.nightly-2022-01-25-023600 True False False 96m machine-approver 4.10.0-0.nightly-2022-01-25-023600 True False False 100m machine-config 4.10.0-0.nightly-2022-01-25-023600 True False False 99m marketplace 4.10.0-0.nightly-2022-01-25-023600 True False False 100m monitoring 4.10.0-0.nightly-2022-01-25-023600 True False False 75m network 4.10.0-0.nightly-2022-01-25-023600 True False False 102m node-tuning 4.10.0-0.nightly-2022-01-25-023600 True False False 100m openshift-apiserver 4.10.0-0.nightly-2022-01-25-023600 True False False 83m openshift-controller-manager 4.10.0-0.nightly-2022-01-25-023600 True False False 91m openshift-samples 4.10.0-0.nightly-2022-01-25-023600 True False False 78m operator-lifecycle-manager 4.10.0-0.nightly-2022-01-25-023600 True False False 100m operator-lifecycle-manager-catalog 4.10.0-0.nightly-2022-01-25-023600 True False False 100m operator-lifecycle-manager-packageserver 4.10.0-0.nightly-2022-01-25-023600 True False False 96m service-ca 4.10.0-0.nightly-2022-01-25-023600 True False False 101m storage 4.10.0-0.nightly-2022-01-25-023600 True False False 82m $ oc get nodes NAME STATUS ROLES AGE VERSION ostest-48ng8-master-0 Ready master 106m v1.23.0+06791f6 ostest-48ng8-master-1 Ready master 106m v1.23.0+06791f6 ostest-48ng8-master-2 Ready master 106m v1.23.0+06791f6 ostest-48ng8-worker-0-gqwjt Ready worker 85m v1.23.0+06791f6 ostest-48ng8-worker-0-qblvj Ready worker 77m v1.23.0+06791f6 ostest-48ng8-worker-0-zxjvg Ready worker 77m v1.23.0+06791f6 $ oc get pods -n openshift-ovn-kubernetes NAME READY STATUS RESTARTS AGE ovnkube-master-b6dld 6/6 Running 7 (91m ago) 105m ovnkube-master-f7cdx 6/6 Running 2 (93m ago) 105m ovnkube-master-jxln2 6/6 Running 6 (104m ago) 105m ovnkube-node-56qs6 5/5 Running 0 77m ovnkube-node-9wv6v 5/5 Running 3 (84m ago) 85m ovnkube-node-9x7pq 5/5 Running 0 105m ovnkube-node-h66j5 5/5 Running 0 105m ovnkube-node-j82zs 5/5 Running 0 105m ovnkube-node-llmsz 5/5 Running 0 78m and the workers show the proper metrics: [core@ostest-48ng8-worker-0-qblvj ~]$ ip r default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 100 10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0 10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.156 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.3 via 10.128.2.1 dev ovn-k8s-mp0 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.245 metric 100 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 Furthermore, manila integration is working as expected: $ oc get pods -n manila-test -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES demo-59c59d6d55-ct28v 1/1 Running 0 31s 10.129.2.18 ostest-48ng8-worker-0-zxjvg <none> <none> demo-59c59d6d55-wpm8j 1/1 Running 0 31s 10.128.2.13 ostest-48ng8-worker-0-qblvj <none> <none> $ oc get pvc -n manila-test -o wide NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE pvc-manila Bound pvc-7b698c68-beb3-446d-af4a-70b030e3500c 1Gi RWO csi-manila-default 40s Filesystem (venv_shade) (shiftstack) [stack@undercloud-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056