Bug 2059330
| Summary: | [ExternalCloudProvider] installation with additional network on workers fails | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Andreas Karis <akaris> |
| Component: | Networking | Assignee: | Andreas Karis <akaris> |
| Networking sub component: | ovn-kubernetes | QA Contact: | rlobillo |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | low | ||
| Priority: | medium | CC: | akaris, anusaxen, itbrown, mifiedle, rlobillo |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2044503 | Environment: | |
| Last Closed: | 2022-03-16 11:30:10 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2044503 | ||
| Bug Blocks: | 2057526 | ||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.8.34 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0795 |
verified on 4.8.0-0.nightly-2022-03-05-052841 on top of RHOS-16.2-RHEL-8-20211129.n.1 installation with OVNKubernetes network type worked fine and the metrics for the default routes are the expected ones in the workers: (shiftstack) [stack@undercloud-0 ~]$ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ostest-hzb45-master-0 Ready master 58m v1.21.8+8a3bf4a 10.196.1.101 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_ 64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-master-1 Ready master 58m v1.21.8+8a3bf4a 10.196.2.213 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_ 64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-master-2 Ready master 58m v1.21.8+8a3bf4a 10.196.2.1 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_ 64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-worker-0-82jvp Ready worker 40m v1.21.8+8a3bf4a 10.196.3.202 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_ 64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-worker-0-8dctf Ready worker 41m v1.21.8+8a3bf4a 10.196.3.45 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_ 64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-worker-0-nrcbp Ready worker 39m v1.21.8+8a3bf4a 10.196.0.199 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_ 64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 $ ssh -J core.shiftstack.com core.3.202 ip r Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.3.202' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0 10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.202 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.188 metric 101 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 $ ssh -J core.shiftstack.com core.3.45 ip r Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.3.45' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 via 10.131.0.1 dev ovn-k8s-mp0 10.131.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.131.0.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.45 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.228 metric 101 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 $ ssh -J core.shiftstack.com core.0.199 ip r Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.0.199' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 101 10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.199 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.197 metric 101 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 After performing a change on the cloud-config, the nodes are reconfigured and the metrics change to a valid values that allow the cluster to work correctly: $ oc replace --force --wait --filename cloud_cm.yaml configmap "cloud-provider-config" deleted configmap/cloud-provider-config replaced $ oc get nodes NAME STATUS ROLES AGE VERSION ostest-hzb45-master-0 Ready master 127m v1.21.8+8a3bf4a ostest-hzb45-master-1 Ready,SchedulingDisabled master 127m v1.21.8+8a3bf4a ostest-hzb45-master-2 Ready master 127m v1.21.8+8a3bf4a ostest-hzb45-worker-0-82jvp Ready worker 110m v1.21.8+8a3bf4a ostest-hzb45-worker-0-8dctf Ready,SchedulingDisabled worker 110m v1.21.8+8a3bf4a ostest-hzb45-worker-0-nrcbp Ready worker 109m v1.21.8+8a3bf4a $ oc get nodes NAME STATUS ROLES AGE VERSION ostest-hzb45-master-0 Ready master 4h59m v1.21.8+8a3bf4a ostest-hzb45-master-1 Ready master 4h59m v1.21.8+8a3bf4a ostest-hzb45-master-2 Ready master 4h59m v1.21.8+8a3bf4a ostest-hzb45-worker-0-82jvp Ready worker 4h41m v1.21.8+8a3bf4a ostest-hzb45-worker-0-8dctf Ready worker 4h42m v1.21.8+8a3bf4a ostest-hzb45-worker-0-nrcbp Ready worker 4h40m v1.21.8+8a3bf4a $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ostest-hzb45-master-0 Ready master 4h56m v1.21.8+8a3bf4a 10.196.1.101 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-master-1 Ready master 4h56m v1.21.8+8a3bf4a 10.196.2.213 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-master-2 Ready master 4h56m v1.21.8+8a3bf4a 10.196.2.1 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-worker-0-82jvp Ready worker 4h38m v1.21.8+8a3bf4a 10.196.3.202 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-worker-0-8dctf Ready worker 4h39m v1.21.8+8a3bf4a 10.196.3.45 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 ostest-hzb45-worker-0-nrcbp Ready worker 4h37m v1.21.8+8a3bf4a 10.196.0.199 <none> Red Hat Enterprise Linux CoreOS 48.84.202203040913-0 (Ootpa) 4.18.0-305.34.2.el8_4.x86_64 cri-o://1.21.4-12.rhaos4.8.git030755e.el8 $ ssh -J core.shiftstack.com core.3.202 ip r Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.3.202' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 100 10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0 10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.202 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.188 metric 100 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 $ ssh -J core.shiftstack.com core.3.45 ip r Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.3.45' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 100 10.128.0.0/14 via 10.131.0.1 dev ovn-k8s-mp0 10.131.0.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.131.0.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.3.45 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.228 metric 100 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 $ ssh -J core.shiftstack.com core.0.199 ip r Warning: Permanently added 'api.ostest.shiftstack.com,10.46.43.96' (ECDSA) to the list of known hosts. Warning: Permanently added '10.196.0.199' (ECDSA) to the list of known hosts. default via 10.196.0.1 dev br-ex proto dhcp metric 49 default via 172.17.5.1 dev ens4 proto dhcp metric 100 10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.199 metric 49 169.254.169.0/30 via 10.196.0.1 dev br-ex 169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.197 metric 100 172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 $ oc get all -n demo NAME READY STATUS RESTARTS AGE pod/demo-7897db69cc-4rxnv 1/1 Running 0 163m pod/demo-7897db69cc-bxldp 1/1 Running 0 169m pod/demo-7897db69cc-f4gf4 1/1 Running 0 169m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/demo ClusterIP 172.30.1.242 <none> 80/TCP 4h19m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/demo 3/3 3 3 4h19m NAME DESIRED CURRENT READY AGE replicaset.apps/demo-7897db69cc 3 3 3 4h19m $ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242 demo-7897db69cc-4rxnv: HELLO! I AM ALIVE!!! $ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242 demo-7897db69cc-f4gf4: HELLO! I AM ALIVE!!! $ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242 demo-7897db69cc-4rxnv: HELLO! I AM ALIVE!!! $ oc rsh -n demo pod/demo-7897db69cc-4rxnv curl 172.30.1.242 demo-7897db69cc-bxldp: HELLO! I AM ALIVE!!!