Bug 2057526

Summary: cloud provider config change breaks the cluster
Product: OpenShift Container Platform Reporter: rlobillo
Component: NetworkingAssignee: Andreas Karis <akaris>
Networking sub component: ovn-kubernetes QA Contact: rlobillo
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: medium CC: anusaxen, ffernand, jechen, juriarte, scuppett
Version: 4.7Keywords: TestBlocker
Target Milestone: ---Flags: jechen: needinfo-
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-25 12:03:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2059330    
Bug Blocks:    

Description rlobillo 2022-02-23 15:11:19 UTC 
Description of problem:

In the procedure detailed on https://github.com/openshift/installer/tree/release-4.8/docs/user/openstack#enabling-octavia-for-load-balancer-services (that will be backported to 4.7), it is required to perform a change on the cloud provider config. On a cluster using OVN-Kubernetes NetworkType when one extra network is attached to the workers for using manila integration, this action is leaving the cluster unusable.


Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2022-02-16-124118
RHOS-16.2-RHEL-8-20211129.n.1


How reproducible: Always


Steps to Reproduce:
1. IPI installation with multiple networks connected to the worker worked fine:

apiVersion: v1                                                                                                                                                                               
baseDomain: "shiftstack.com"                                                                                                                                                                 
compute:                                                                                                                                                                                     
- name: worker                                                                                                                                                                               
  platform:                                                                                                                                                                                  
    openstack:                                                                                                                                                                               
      zones: []                                                                                                                                                                              
      additionalNetworkIDs: ['d44f723c-a504-4bdc-9660-66b1eb2bc94b']                                                                                                                         
  replicas: 3                                                                                                                                                                                
controlPlane:                                                                                                                                                                                
  name: master                                                                                                                                                                               
  platform:
    openstack:                                                                                                                                                                               
      zones: []                                                                                                                                                                              
  replicas: 3                                                                               
networking:
  clusterNetworks:
  - cidr:             10.128.0.0/14
    hostSubnetLength: 9
  serviceNetwork:
    - 172.30.0.0/16
  machineNetwork:
    - cidr: "10.196.0.0/16"
  networkType: "OVNKubernetes"


DEBUG Time elapsed per stage:
DEBUG     Infrastructure: 2m31s
DEBUG Bootstrap Complete: 14m51s
DEBUG                API: 2m40s
DEBUG  Bootstrap Destroy: 37s
DEBUG  Cluster Operators: 21m18s
INFO Time elapsed: 40m8s

and the routes in the workers are shown below:

$ openstack server list -c Name -c Networks
+-----------------------------+--------------------------------------------------------------+
| Name                        | Networks                                                     |
+-----------------------------+--------------------------------------------------------------+
| ostest-fqm4m-worker-0-ft9wd | StorageNFS=172.17.5.227; ostest-fqm4m-openshift=10.196.2.172 |
| ostest-fqm4m-worker-0-d8gwb | StorageNFS=172.17.5.204; ostest-fqm4m-openshift=10.196.1.165 |
| ostest-fqm4m-worker-0-ff22t | StorageNFS=172.17.5.211; ostest-fqm4m-openshift=10.196.0.147 |
| ostest-fqm4m-master-2       | ostest-fqm4m-openshift=10.196.1.94                           |
| ostest-fqm4m-master-1       | ostest-fqm4m-openshift=10.196.2.89                           |
| ostest-fqm4m-master-0       | ostest-fqm4m-openshift=10.196.3.179                          |
+-----------------------------+--------------------------------------------------------------+

$ ssh -J core.shiftstack.com core.2.172 ip r
default via 10.196.0.1 dev br-ex proto dhcp metric 100
default via 172.17.5.1 dev ens4 proto dhcp metric 101
10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0
10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.2.172 metric 100
169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.227 metric 101
172.30.0.0/16 via 10.128.2.1 dev ovn-k8s-mp0

$ ssh -J core.shiftstack.com core.1.165 ip r
default via 10.196.0.1 dev br-ex proto dhcp metric 100
default via 172.17.5.1 dev ens4 proto dhcp metric 101
10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0
10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.1.165 metric 100
169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.204 metric 101
172.30.0.0/16 via 10.129.2.1 dev ovn-k8s-mp0

$ ssh -J core.shiftstack.com core.0.147 ip r
default via 10.196.0.1 dev br-ex proto dhcp metric 100
default via 172.17.5.1 dev ens4 proto dhcp metric 101
10.128.0.0/14 via 10.131.0.1 dev ovn-k8s-mp0
10.131.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.131.0.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.147 metric 100
169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.211 metric 101
172.30.0.0/16 via 10.131.0.1 dev ovn-k8s-mp0

where: 

$ openstack subnet list
+--------------------------------------+--------------------+--------------------------------------+---------------+
| ID                                   | Name               | Network                              | Subnet        |
+--------------------------------------+--------------------+--------------------------------------+---------------+
| 9719cc05-104b-4604-9330-13c161bd707e | ostest-fqm4m-nodes | ef8f2570-4ce5-465c-ad06-e768be6f7289 | 10.196.0.0/16 |
| e4f51f81-c06b-468e-aa18-0fec98a4f54e | StorageNFSSubnet   | d44f723c-a504-4bdc-9660-66b1eb2bc94b | 172.17.5.0/24 |
+--------------------------------------+--------------------+--------------------------------------+---------------+


2. changing the cloud-provider-config ($ oc edit cm cloud-provider-config -n openshift-config) from:

  config: |
    [Global]
    secret-name = openstack-credentials
    secret-namespace = kube-system
    region = regionOne
    ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem

to:

  config: |
    [Global]
    secret-name = openstack-credentials
    secret-namespace = kube-system
    region = regionOne
    ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
    [LoadBalancer]
    use-octavia = True

and wait until all nodes are to unschedulable and then moved to schedulable again (to confirm that the change has been applied):

NAME                          STATUS                     ROLES    AGE   VERSION
ostest-fqm4m-master-0         Ready,SchedulingDisabled   master   88m   v1.20.11+e880017
ostest-fqm4m-master-1         Ready                      master   88m   v1.20.11+e880017
ostest-fqm4m-master-2         Ready                      master   88m   v1.20.11+e880017
ostest-fqm4m-worker-0-d8gwb   Ready,SchedulingDisabled   worker   71m   v1.20.11+e880017
ostest-fqm4m-worker-0-ff22t   Ready                      worker   71m   v1.20.11+e880017
ostest-fqm4m-worker-0-ft9wd   Ready                      worker   71m   v1.20.11+e880017


The routes are changed in the workers:

$ ssh -J core.shiftstack.com core.2.172 ip r
default via 172.17.5.1 dev ens4 proto dhcp metric 100 
default via 10.196.0.1 dev br-ex proto dhcp metric 100 
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.2.172 metric 100 
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.227 metric 100 

$ ssh -J core.shiftstack.com core.1.165 ip r
default via 172.17.5.1 dev ens4 proto dhcp metric 100 
default via 10.196.0.1 dev br-ex proto dhcp metric 100 
10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 
10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.1.165 metric 100 
169.254.0.0/20 dev ovn-k8s-gw0 proto kernel scope link src 169.254.0.1 
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.204 metric 100 
172.30.0.0/16 via 10.129.2.1 dev ovn-k8s-mp0 

$ ssh -J core.shiftstack.com core.0.147 ip r
default via 172.17.5.1 dev ens4 proto dhcp metric 100 
default via 10.196.0.1 dev br-ex proto dhcp metric 100 
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.147 metric 100 
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.211 metric 100 

Now the default routes' metrics are equal to 100, and the default route used to reach the outside world is the one used for integration with ManilaNFS, so the cluster become inoperative:

$ oc get pods -n demo
NAME                    READY   STATUS             RESTARTS   AGE
demo-7897db69cc-grs9h   0/1     ImagePullBackOff   0          41m
demo-7897db69cc-kwp7l   0/1     ImagePullBackOff   0          41m
demo-7897db69cc-p6z77   0/1     ImagePullBackOff   0          41m

Actual results: cluster becomes unoperative
Expected results: The cluster should remain operative
Additional info: must-gather on private comment.
Comment 2 Andreas Karis 2022-02-28 16:39:03 UTC 
Looks like we'll have to backport https://github.com/openshift/machine-config-operator/commit/e6a673db9a2931906cf00832684f5d132ca333ee all the way to 4.7
Comment 4 Andreas Karis 2022-03-11 17:49:22 UTC 
I'll hold this until 2057160 merges as I believe that those changes are important here.
Comment 15 rlobillo 2022-05-24 09:43:13 UTC 
Verified on OCP4.7.51 on top of RHOS-16.2-RHEL-8-20220311.n.1


Given a cluster under below conditions: 

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.51    True        False         14h     Cluster version is 4.7.51

$ oc get nodes
NAME                          STATUS   ROLES    AGE   VERSION
ostest-8dfs4-master-0         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-master-1         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-master-2         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-5nlz5   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-h2rsq   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready    worker   14h   v1.20.15+98b2293

$ oc get network/cluster -o json | jq .spec.networkType
"OVNKubernetes"

$ openstack server list -c Name -c Networks
+-----------------------------+------------------------------------------------------------+
| Name                        | Networks                                                   |
+-----------------------------+------------------------------------------------------------+
| ostest-8dfs4-worker-0-5nlz5 | StorageNFS=10.0.0.228; ostest-8dfs4-openshift=10.196.0.179 |
| ostest-8dfs4-worker-0-s622c | StorageNFS=10.0.0.126; ostest-8dfs4-openshift=10.196.1.205 |
| ostest-8dfs4-worker-0-h2rsq | StorageNFS=10.0.0.253; ostest-8dfs4-openshift=10.196.3.206 |
| ostest-8dfs4-master-2       | ostest-8dfs4-openshift=10.196.1.196                        |
| ostest-8dfs4-master-1       | ostest-8dfs4-openshift=10.196.2.112                        |
| ostest-8dfs4-master-0       | ostest-8dfs4-openshift=10.196.2.192                        |
+-----------------------------+------------------------------------------------------------+

$ for i in $(openstack server list -f json | jq -r .[].Name); do echo "# $i"; ssh -J core.shiftstack.com core@$i ip r show default; echo ; done
# ostest-8dfs4-worker-0-5nlz5                        
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-worker-0-5nlz5' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 10.0.0.1 dev ens4 proto dhcp metric 101                                                                                                                                           
     
# ostest-8dfs4-worker-0-s622c
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-worker-0-s622c' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 10.0.0.1 dev ens4 proto dhcp metric 101
                             
# ostest-8dfs4-worker-0-h2rsq                                                                         
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-worker-0-h2rsq' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 10.0.0.1 dev ens4 proto dhcp metric 101
                                                                                                      
# ostest-8dfs4-master-2                                                                     
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-master-2' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
                                                                                                      
# ostest-8dfs4-master-1                                                               
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-master-1' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49                                                 
                                                                                      
# ostest-8dfs4-master-0  
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-master-0' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49         


###################################

changing the cloud-provider-config ($ oc edit cm cloud-provider-config -n openshift-config) from:

  config: |
    [Global]
    secret-name = openstack-credentials
    secret-namespace = kube-system
    region = regionOne
    ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem

to:

  config: |
    [Global]
    secret-name = openstack-credentials
    secret-namespace = kube-system
    region = regionOne
    ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
    [LoadBalancer]
    use-octavia = True
    

$ oc get nodes -w                                                                                                                             
NAME                          STATUS   ROLES    AGE   VERSION                                                                                                                                 
ostest-8dfs4-master-0         Ready    master   14h   v1.20.15+98b2293                                                                                                                        
ostest-8dfs4-master-1         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-master-2         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-5nlz5   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-h2rsq   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-master-0         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-master-0         Ready    master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-master-0         Ready,SchedulingDisabled   master   14h   v1.20.15+98b2293
ostest-8dfs4-master-0         Ready,SchedulingDisabled   master   14h   v1.20.15+98b2293
ostest-8dfs4-master-1         Ready                      master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-h2rsq   Ready                      worker   14h   v1.20.15+98b2293
ostest-8dfs4-master-1         Ready                      master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   NotReady,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   NotReady,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-master-0         Ready,SchedulingDisabled      master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   NotReady,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-h2rsq   Ready                         worker   14h   v1.20.15+98b2293
ostest-8dfs4-master-1         Ready                         master   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   NotReady,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   NotReady,SchedulingDisabled   worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready,SchedulingDisabled      worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready,SchedulingDisabled      worker   14h   v1.20.15+98b2293
[...]
$ oc get nodes 
NAME                          STATUS   ROLES    AGE   VERSION
ostest-8dfs4-master-0         Ready    master   15h   v1.20.15+98b2293
ostest-8dfs4-master-1         Ready    master   15h   v1.20.15+98b2293
ostest-8dfs4-master-2         Ready    master   15h   v1.20.15+98b2293
ostest-8dfs4-worker-0-5nlz5   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-h2rsq   Ready    worker   14h   v1.20.15+98b2293
ostest-8dfs4-worker-0-s622c   Ready    worker   14h   v1.20.15+98b2293

###################################

Routes are set correctly:

$ for i in $(openstack server list -f json | jq -r .[].Name); do echo "# $i"; ssh -J core.shiftstack.com core@$i ip r show default; echo ; done
# ostest-8dfs4-worker-0-5nlz5
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-worker-0-5nlz5' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 10.0.0.1 dev ens4 proto dhcp metric 100

# ostest-8dfs4-worker-0-s622c
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-worker-0-s622c' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 10.0.0.1 dev ens4 proto dhcp metric 100

# ostest-8dfs4-worker-0-h2rsq
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-worker-0-h2rsq' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 10.0.0.1 dev ens4 proto dhcp metric 100

# ostest-8dfs4-master-2
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-master-2' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49

# ostest-8dfs4-master-1
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-master-1' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49

# ostest-8dfs4-master-0
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.23.46' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added 'ostest-8dfs4-master-0' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49

Cluster is operative:

$ oc apply -f demo.yaml 
deployment.apps/demo created
service/demo created

$ oc get pods -n demo -o wide
NAME                    READY   STATUS    RESTARTS   AGE    IP            NODE                          NOMINATED NODE   READINESS GATES
demo-7897db69cc-cfhkj   1/1     Running   0          4m5s   10.131.0.7    ostest-8dfs4-worker-0-5nlz5   <none>           <none>
demo-7897db69cc-fmnlw   1/1     Running   0          4m5s   10.128.2.7    ostest-8dfs4-worker-0-s622c   <none>           <none>
demo-7897db69cc-sgxr2   1/1     Running   0          4m6s   10.129.2.12   ostest-8dfs4-worker-0-h2rsq   <none>           <none>
Comment 17 errata-xmlrpc 2022-05-25 12:03:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.51 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:2268