Description of problem: In the procedure detailed on https://github.com/openshift/installer/tree/release-4.8/docs/user/openstack#enabling-octavia-for-load-balancer-services (that will be backported to 4.7), it is required to perform a change on the cloud provider config. On a cluster using OVN-Kubernetes NetworkType when one extra network is attached to the workers for using manila integration, this action is leaving the cluster unusable. Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2022-02-16-124118 RHOS-16.2-RHEL-8-20211129.n.1 How reproducible: Always Steps to Reproduce: 1. IPI installation with multiple networks connected to the worker worked fine: apiVersion: v1 baseDomain: "shiftstack.com" compute: - name: worker platform: openstack: zones: [] additionalNetworkIDs: ['d44f723c-a504-4bdc-9660-66b1eb2bc94b'] replicas: 3 controlPlane: name: master platform: openstack: zones: [] replicas: 3 networking: clusterNetworks: - cidr: 10.128.0.0/14 hostSubnetLength: 9 serviceNetwork: - 172.30.0.0/16 machineNetwork: - cidr: "10.196.0.0/16" networkType: "OVNKubernetes" DEBUG Time elapsed per stage: DEBUG Infrastructure: 2m31s DEBUG Bootstrap Complete: 14m51s DEBUG API: 2m40s DEBUG Bootstrap Destroy: 37s DEBUG Cluster Operators: 21m18s INFO Time elapsed: 40m8s and the routes in the workers are shown below: $ openstack server list -c Name -c Networks +-----------------------------+--------------------------------------------------------------+ | Name | Networks | +-----------------------------+--------------------------------------------------------------+ | ostest-fqm4m-worker-0-ft9wd | StorageNFS=172.17.5.227; ostest-fqm4m-openshift=10.196.2.172 | | ostest-fqm4m-worker-0-d8gwb | StorageNFS=172.17.5.204; ostest-fqm4m-openshift=10.196.1.165 | | ostest-fqm4m-worker-0-ff22t | StorageNFS=172.17.5.211; ostest-fqm4m-openshift=10.196.0.147 | | ostest-fqm4m-master-2 | ostest-fqm4m-openshift=10.196.1.94 | | ostest-fqm4m-master-1 | ostest-fqm4m-openshift=10.196.2.89 | | ostest-fqm4m-master-0 | ostest-fqm4m-openshift=10.196.3.179 | +-----------------------------+--------------------------------------------------------------+ $ ssh -J core.shiftstack.com core.2.172 ip r default via 10.196.0.1 dev br-ex proto dhcp metric 100 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0 10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.2.172 metric 100 169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.227 metric 101 172.30.0.0/16 via 10.128.2.1 dev ovn-k8s-mp0 $ ssh -J core.shiftstack.com core.1.165 ip r default via 10.196.0.1 dev br-ex proto dhcp metric 100 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.1.165 metric 100 169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.204 metric 101 172.30.0.0/16 via 10.129.2.1 dev ovn-k8s-mp0 $ ssh -J core.shiftstack.com core.0.147 ip r default via 10.196.0.1 dev br-ex proto dhcp metric 100 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 via 10.131.0.1 dev ovn-k8s-mp0 10.131.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.131.0.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.147 metric 100 169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.211 metric 101 172.30.0.0/16 via 10.131.0.1 dev ovn-k8s-mp0 where: $ openstack subnet list +--------------------------------------+--------------------+--------------------------------------+---------------+ | ID | Name | Network | Subnet | +--------------------------------------+--------------------+--------------------------------------+---------------+ | 9719cc05-104b-4604-9330-13c161bd707e | ostest-fqm4m-nodes | ef8f2570-4ce5-465c-ad06-e768be6f7289 | 10.196.0.0/16 | | e4f51f81-c06b-468e-aa18-0fec98a4f54e | StorageNFSSubnet | d44f723c-a504-4bdc-9660-66b1eb2bc94b | 172.17.5.0/24 | +--------------------------------------+--------------------+--------------------------------------+---------------+ 2. changing the cloud-provider-config ($ oc edit cm cloud-provider-config -n openshift-config) from: config: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem to: config: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem [LoadBalancer] use-octavia = True and wait until all nodes are to unschedulable and then moved to schedulable again (to confirm that the change has been applied): NAME STATUS ROLES AGE VERSION ostest-fqm4m-master-0 Ready,SchedulingDisabled master 88m v1.20.11+e880017 ostest-fqm4m-master-1 Ready master 88m v1.20.11+e880017 ostest-fqm4m-master-2 Ready master 88m v1.20.11+e880017 ostest-fqm4m-worker-0-d8gwb Ready,SchedulingDisabled worker 71m v1.20.11+e880017 ostest-fqm4m-worker-0-ff22t Ready worker 71m v1.20.11+e880017 ostest-fqm4m-worker-0-ft9wd Ready worker 71m v1.20.11+e880017 The routes are changed in the workers: $ ssh -J core.shiftstack.com core.2.172 ip r default via 172.17.5.1 dev ens4 proto dhcp metric 100 default via 10.196.0.1 dev br-ex proto dhcp metric 100 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.2.172 metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.227 metric 100 $ ssh -J core.shiftstack.com core.1.165 ip r default via 172.17.5.1 dev ens4 proto dhcp metric 100 default via 10.196.0.1 dev br-ex proto dhcp metric 100 10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.1.165 metric 100 169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.204 metric 100 172.30.0.0/16 via 10.129.2.1 dev ovn-k8s-mp0 $ ssh -J core.shiftstack.com core.0.147 ip r default via 172.17.5.1 dev ens4 proto dhcp metric 100 default via 10.196.0.1 dev br-ex proto dhcp metric 100 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.147 metric 100 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.211 metric 100 Now the default routes' metrics are equal to 100, and the default route used to reach the outside world is the one used for integration with ManilaNFS, so the cluster become inoperative: $ oc get pods -n demo NAME READY STATUS RESTARTS AGE demo-7897db69cc-grs9h 0/1 ImagePullBackOff 0 41m demo-7897db69cc-kwp7l 0/1 ImagePullBackOff 0 41m demo-7897db69cc-p6z77 0/1 ImagePullBackOff 0 41m Actual results: cluster becomes unoperative Expected results: The cluster should remain operative Additional info: must-gather on private comment.
Looks like we'll have to backport https://github.com/openshift/machine-config-operator/commit/e6a673db9a2931906cf00832684f5d132ca333ee all the way to 4.7
I'll hold this until 2057160 merges as I believe that those changes are important here.
Verified on OCP4.7.51 on top of RHOS-16.2-RHEL-8-20220311.n.1 Given a cluster under below conditions: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.51 True False 14h Cluster version is 4.7.51 $ oc get nodes NAME STATUS ROLES AGE VERSION ostest-8dfs4-master-0 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-master-1 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-master-2 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-5nlz5 Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-h2rsq Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready worker 14h v1.20.15+98b2293 $ oc get network/cluster -o json | jq .spec.networkType "OVNKubernetes" $ openstack server list -c Name -c Networks +-----------------------------+------------------------------------------------------------+ | Name | Networks | +-----------------------------+------------------------------------------------------------+ | ostest-8dfs4-worker-0-5nlz5 | StorageNFS=10.0.0.228; ostest-8dfs4-openshift=10.196.0.179 | | ostest-8dfs4-worker-0-s622c | StorageNFS=10.0.0.126; ostest-8dfs4-openshift=10.196.1.205 | | ostest-8dfs4-worker-0-h2rsq | StorageNFS=10.0.0.253; ostest-8dfs4-openshift=10.196.3.206 | | ostest-8dfs4-master-2 | ostest-8dfs4-openshift=10.196.1.196 | | ostest-8dfs4-master-1 | ostest-8dfs4-openshift=10.196.2.112 | | ostest-8dfs4-master-0 | ostest-8dfs4-openshift=10.196.2.192 | +-----------------------------+------------------------------------------------------------+ $ for i in $(openstack server list -f json | jq -r .[].Name); do echo "# $i"; ssh -J core.shiftstack.com core@$i ip r show default; echo ; done # ostest-8dfs4-worker-0-5nlz5 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-worker-0-5nlz5' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 10.0.0.1 dev ens4 proto dhcp metric 101 # ostest-8dfs4-worker-0-s622c Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-worker-0-s622c' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 10.0.0.1 dev ens4 proto dhcp metric 101 # ostest-8dfs4-worker-0-h2rsq Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-worker-0-h2rsq' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 10.0.0.1 dev ens4 proto dhcp metric 101 # ostest-8dfs4-master-2 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-master-2' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 # ostest-8dfs4-master-1 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-master-1' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 # ostest-8dfs4-master-0 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-master-0' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 ################################### changing the cloud-provider-config ($ oc edit cm cloud-provider-config -n openshift-config) from: config: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem to: config: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem [LoadBalancer] use-octavia = True $ oc get nodes -w NAME STATUS ROLES AGE VERSION ostest-8dfs4-master-0 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-master-1 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-master-2 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-5nlz5 Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-h2rsq Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-master-0 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-master-0 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-master-0 Ready,SchedulingDisabled master 14h v1.20.15+98b2293 ostest-8dfs4-master-0 Ready,SchedulingDisabled master 14h v1.20.15+98b2293 ostest-8dfs4-master-1 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-h2rsq Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-master-1 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c NotReady,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c NotReady,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-master-0 Ready,SchedulingDisabled master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c NotReady,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-h2rsq Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-master-1 Ready master 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c NotReady,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c NotReady,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready,SchedulingDisabled worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready,SchedulingDisabled worker 14h v1.20.15+98b2293 [...] $ oc get nodes NAME STATUS ROLES AGE VERSION ostest-8dfs4-master-0 Ready master 15h v1.20.15+98b2293 ostest-8dfs4-master-1 Ready master 15h v1.20.15+98b2293 ostest-8dfs4-master-2 Ready master 15h v1.20.15+98b2293 ostest-8dfs4-worker-0-5nlz5 Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-h2rsq Ready worker 14h v1.20.15+98b2293 ostest-8dfs4-worker-0-s622c Ready worker 14h v1.20.15+98b2293 ################################### Routes are set correctly: $ for i in $(openstack server list -f json | jq -r .[].Name); do echo "# $i"; ssh -J core.shiftstack.com core@$i ip r show default; echo ; done # ostest-8dfs4-worker-0-5nlz5 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-worker-0-5nlz5' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 10.0.0.1 dev ens4 proto dhcp metric 100 # ostest-8dfs4-worker-0-s622c Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-worker-0-s622c' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 10.0.0.1 dev ens4 proto dhcp metric 100 # ostest-8dfs4-worker-0-h2rsq Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-worker-0-h2rsq' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 10.0.0.1 dev ens4 proto dhcp metric 100 # ostest-8dfs4-master-2 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-master-2' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 # ostest-8dfs4-master-1 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-master-1' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 # ostest-8dfs4-master-0 Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts. Warning: Permanently added 'ostest-8dfs4-master-0' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 Cluster is operative: $ oc apply -f demo.yaml deployment.apps/demo created service/demo created $ oc get pods -n demo -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES demo-7897db69cc-cfhkj 1/1 Running 0 4m5s 10.131.0.7 ostest-8dfs4-worker-0-5nlz5 <none> <none> demo-7897db69cc-fmnlw 1/1 Running 0 4m5s 10.128.2.7 ostest-8dfs4-worker-0-s622c <none> <none> demo-7897db69cc-sgxr2 1/1 Running 0 4m6s 10.129.2.12 ostest-8dfs4-worker-0-h2rsq <none> <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.51 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:2268