2059330 – [ExternalCloudProvider] installation with additional network on workers fails

Bug 2059330 - [ExternalCloudProvider] installation with additional network on workers fails

Summary: [ExternalCloudProvider] installation with additional network on workers fails

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	low
Target Milestone:	---
Target Release:	4.8.z
Assignee:	Andreas Karis
QA Contact:	rlobillo
Docs Contact:
URL:
Whiteboard:
Depends On:	2044503
Blocks:	2057526
TreeView+	depends on / blocked

Reported:	2022-02-28 19:09 UTC by Andreas Karis
Modified:	2022-03-16 11:30 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	2044503
Environment:
Last Closed:	2022-03-16 11:30:10 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift machine-config-operator pull 2974	0	None	open	[release-4.8] Bug 2059330: ovs-configuration: use lower than NM default ethernet route metric	2022-02-28 19:52:34 UTC
Red Hat Product Errata	RHBA-2022:0795	0	None	None	None	2022-03-16 11:30:24 UTC

Comment 5 rlobillo 2022-03-07 14:13:05 UTC

verified on 4.8.0-0.nightly-2022-03-05-052841 on top of RHOS-16.2-RHEL-8-20211129.n.1

installation with OVNKubernetes network type worked fine and the metrics for the default routes are the expected ones in the workers:

(shiftstack) [stack@undercloud-0 ~]$ oc get nodes -o wide                                                                                                                                    
NAME                          STATUS   ROLES    AGE   VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION           
     CONTAINER-RUNTIME
ostest-hzb45-master-0         Ready    master   58m   v1.21.8+8a3bf4a   10.196.1.101   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_
64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-master-1         Ready    master   58m   v1.21.8+8a3bf4a   10.196.2.213   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_
64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8                                                                                                                                               
ostest-hzb45-master-2         Ready    master   58m   v1.21.8+8a3bf4a   10.196.2.1     <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_
64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-worker-0-82jvp   Ready    worker   40m   v1.21.8+8a3bf4a   10.196.3.202   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_
64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-worker-0-8dctf   Ready    worker   41m   v1.21.8+8a3bf4a   10.196.3.45    <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_
64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-worker-0-nrcbp   Ready    worker   39m   v1.21.8+8a3bf4a   10.196.0.199   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_
64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8


$ ssh -J core.shiftstack.com core.3.202 ip r                                                                                            
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts.                                                                                       
Warning: Permanently added '10.196.3.202' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 172.17.5.1 dev ens4 proto dhcp metric 101
10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0                                                                                                                                                 
10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.202 metric 49                                                                                                                   
169.254.169.0/30 via 10.196.0.1 dev br-ex
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.188 metric 101
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342

$ ssh -J core.shiftstack.com core.3.45 ip r
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.3.45' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 172.17.5.1 dev ens4 proto dhcp metric 101
10.128.0.0/14 via 10.131.0.1 dev ovn-k8s-mp0
10.131.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.131.0.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.45 metric 49
169.254.169.0/30 via 10.196.0.1 dev br-ex
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.228 metric 101
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342

$ ssh -J core.shiftstack.com core.0.199 ip r
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.0.199' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 172.17.5.1 dev ens4 proto dhcp metric 101
10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0
10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.199 metric 49
169.254.169.0/30 via 10.196.0.1 dev br-ex
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.197 metric 101
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342


After performing a change on the cloud-config, the nodes are reconfigured and the metrics change to a valid values that allow the cluster to work correctly:

$ oc replace --force --wait --filename cloud_cm.yaml
configmap "cloud-provider-config" deleted
configmap/cloud-provider-config replaced


$ oc get nodes
NAME                          STATUS                     ROLES    AGE    VERSION
ostest-hzb45-master-0         Ready                      master   127m   v1.21.8+8a3bf4a
ostest-hzb45-master-1         Ready,SchedulingDisabled   master   127m   v1.21.8+8a3bf4a
ostest-hzb45-master-2         Ready                      master   127m   v1.21.8+8a3bf4a
ostest-hzb45-worker-0-82jvp   Ready                      worker   110m   v1.21.8+8a3bf4a
ostest-hzb45-worker-0-8dctf   Ready,SchedulingDisabled   worker   110m   v1.21.8+8a3bf4a
ostest-hzb45-worker-0-nrcbp   Ready                      worker   109m   v1.21.8+8a3bf4a

$ oc get nodes
NAME                          STATUS   ROLES    AGE     VERSION
ostest-hzb45-master-0         Ready    master   4h59m   v1.21.8+8a3bf4a
ostest-hzb45-master-1         Ready    master   4h59m   v1.21.8+8a3bf4a
ostest-hzb45-master-2         Ready    master   4h59m   v1.21.8+8a3bf4a
ostest-hzb45-worker-0-82jvp   Ready    worker   4h41m   v1.21.8+8a3bf4a
ostest-hzb45-worker-0-8dctf   Ready    worker   4h42m   v1.21.8+8a3bf4a
ostest-hzb45-worker-0-nrcbp   Ready    worker   4h40m   v1.21.8+8a3bf4a


$ oc get nodes -o wide
NAME                          STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME                       
ostest-hzb45-master-0         Ready    master   4h56m   v1.21.8+8a3bf4a   10.196.1.101   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-master-1         Ready    master   4h56m   v1.21.8+8a3bf4a   10.196.2.213   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-master-2         Ready    master   4h56m   v1.21.8+8a3bf4a   10.196.2.1     <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-worker-0-82jvp   Ready    worker   4h38m   v1.21.8+8a3bf4a   10.196.3.202   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-worker-0-8dctf   Ready    worker   4h39m   v1.21.8+8a3bf4a   10.196.3.45    <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8
ostest-hzb45-worker-0-nrcbp   Ready    worker   4h37m   v1.21.8+8a3bf4a   10.196.0.199   <none>        Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa)   4.18.0-305.34.2.el8_4.x86_64   cri-o://1.21.4-12.rhaos4.8.git030755e.el8

$ ssh -J core.shiftstack.com core.3.202 ip r
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.3.202' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 172.17.5.1 dev ens4 proto dhcp metric 100
10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0
10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.202 metric 49
169.254.169.0/30 via 10.196.0.1 dev br-ex
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.188 metric 100
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342

$ ssh -J core.shiftstack.com core.3.45 ip r
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.3.45' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 172.17.5.1 dev ens4 proto dhcp metric 100
10.128.0.0/14 via 10.131.0.1 dev ovn-k8s-mp0
10.131.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.131.0.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.45 metric 49
169.254.169.0/30 via 10.196.0.1 dev br-ex
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.228 metric 100
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342

$ ssh -J core.shiftstack.com core.0.199 ip r
Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.0.199' (ECDSA) to the list of known hosts.
default via 10.196.0.1 dev br-ex proto dhcp metric 49
default via 172.17.5.1 dev ens4 proto dhcp metric 100
10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0
10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.199 metric 49
169.254.169.0/30 via 10.196.0.1 dev br-ex
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.197 metric 100
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342


$ oc get all -n demo
NAME                        READY   STATUS    RESTARTS   AGE
pod/demo-7897db69cc-4rxnv   1/1     Running   0          163m
pod/demo-7897db69cc-bxldp   1/1     Running   0          169m
pod/demo-7897db69cc-f4gf4   1/1     Running   0          169m

NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/demo   ClusterIP   172.30.1.242   <none>        80/TCP    4h19m

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo   3/3     3            3           4h19m

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-7897db69cc   3         3         3       4h19m

$ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242
demo-7897db69cc-4rxnv: HELLO! I AM ALIVE!!!

$ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242
demo-7897db69cc-f4gf4: HELLO! I AM ALIVE!!!

$ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242
demo-7897db69cc-4rxnv: HELLO! I AM ALIVE!!!

$ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242
demo-7897db69cc-bxldp: HELLO! I AM ALIVE!!!

Comment 8 errata-xmlrpc 2022-03-16 11:30:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.34 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0795

Note You need to log in before you can comment on or make changes to this bug.