Description of problem: default ipv6 route on openshift nodes is set via bootstrap vm or node where metal3 pod is running due to router advertisements enabled by dnsmasq. As a result all external ipv6 traffic is routed through the node that is running the metal3 pod, over the provisioning network. Checking routing tables during deployment while bootstrap VM is still running: [core@master-0 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::62 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd2e:6f44:5dd8:c956::10 dev enp5s0 proto kernel metric 256 pref medium fd2e:6f44:5dd8:c956::107 dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium default via fe80::8b62:cb84:62e0:735f dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium [core@master-1 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::63 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd2e:6f44:5dd8:c956::10d dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium default via fe80::8b62:cb84:62e0:735f dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium [core@master-2 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::14 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd2e:6f44:5dd8:c956::121 dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium default via fe80::8b62:cb84:62e0:735f dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium fe80::8b62:cb84:62e0:735f is the link local IP address of the provisioning interface of the boostrap VM After the boostrap VM has been destroyed and the metal3 pod was created we can see the following routing tables: oc -n openshift-machine-api get pods/metal3-6bf6fdbd65-vvzwg -o yaml | grep nodeName nodeName: master-0.ocp-edge-cluster.qe.lab.redhat.com master-0 is the node running the metal3 pod [core@master-0 ~]$ ip a s dev enp4s0 2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:c8:47:99 brd ff:ff:ff:ff:ff:ff inet6 fd00:1101::3/64 scope global dynamic valid_lft 9sec preferred_lft 9sec inet6 fe80::44f1:17a0:a00f:2a5d/64 scope link noprefixroute valid_lft forever preferred_lft forever [core@master-0 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::62 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd00:1101::/64 dev enp4s0 proto kernel metric 256 expires 5sec pref medium fd01:0:0:2::/64 dev k8s-master-0.oc proto kernel metric 256 pref medium fd01::/48 via fd01:0:0:2::1 dev k8s-master-0.oc metric 1024 pref medium fd02::/112 via fd01:0:0:2::1 dev k8s-master-0.oc metric 1024 pref medium fd2e:6f44:5dd8:c956::5 dev enp5s0 proto kernel metric 256 pref medium fd2e:6f44:5dd8:c956::10 dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::10 dev enp5s0 proto kernel metric 256 pref medium fd2e:6f44:5dd8:c956::107 dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fd99::/64 dev br-nexthop proto kernel metric 256 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium fe80::/64 dev br-local proto kernel metric 256 pref medium fe80::/64 dev fe842d7ee7ea0c6 proto kernel metric 256 pref medium fe80::/64 dev 31b9f57ba76bc40 proto kernel metric 256 pref medium fe80::/64 dev b5e1ff654e65d25 proto kernel metric 256 pref medium fe80::/64 dev e21733449b0560a proto kernel metric 256 pref medium fe80::/64 dev 5ed89f179633d2b proto kernel metric 256 pref medium fe80::/64 dev ed31353892b89a2 proto kernel metric 256 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium [core@master-1 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::63 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd01:0:0:1::/64 dev k8s-master-1.oc proto kernel metric 256 pref medium fd01::/48 via fd01:0:0:1::1 dev k8s-master-1.oc metric 1024 pref medium fd02::/112 via fd01:0:0:1::1 dev k8s-master-1.oc metric 1024 pref medium fd2e:6f44:5dd8:c956::2 dev enp5s0 proto kernel metric 256 pref medium fd2e:6f44:5dd8:c956::10d dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fd99::/64 dev br-nexthop proto kernel metric 256 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium fe80::/64 dev br-local proto kernel metric 256 pref medium fe80::/64 dev 1593d53806bcbd1 proto kernel metric 256 pref medium fe80::/64 dev 30ba70200adb733 proto kernel metric 256 pref medium fe80::/64 dev a3543ff01278239 proto kernel metric 256 pref medium fe80::/64 dev af47181644178db proto kernel metric 256 pref medium default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium [core@master-2 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::14 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd01:0:0:3::/64 dev k8s-master-2.oc proto kernel metric 256 pref medium fd01::/48 via fd01:0:0:3::1 dev k8s-master-2.oc metric 1024 pref medium fd02::/112 via fd01:0:0:3::1 dev k8s-master-2.oc metric 1024 pref medium fd2e:6f44:5dd8:c956::121 dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fd99::/64 dev br-nexthop proto kernel metric 256 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium fe80::/64 dev br-local proto kernel metric 256 pref medium fe80::/64 dev 2ea1fe1f24a8305 proto kernel metric 256 pref medium fe80::/64 dev 21288b1f7e153c0 proto kernel metric 256 pref medium fe80::/64 dev 54730d44a48d91a proto kernel metric 256 pref medium fe80::/64 dev ebad5cd7dc36bf7 proto kernel metric 256 pref medium fe80::/64 dev 37d0ed44970e690 proto kernel metric 256 pref medium fe80::/64 dev 33b2806d233a1ab proto kernel metric 256 pref medium fe80::/64 dev c079218f097c991 proto kernel metric 256 pref medium fe80::/64 dev 96e6e8393e3ee17 proto kernel metric 256 pref medium fe80::/64 dev 9582f2678237e4f proto kernel metric 256 pref medium fe80::/64 dev 451f5f36dfb3a3d proto kernel metric 256 pref medium fe80::/64 dev 9eb3d6c51c0720d proto kernel metric 256 pref medium fe80::/64 dev 87111a34a098d15 proto kernel metric 256 pref medium fe80::/64 dev 7325cc2e2aeb4c2 proto kernel metric 256 pref medium fe80::/64 dev d36d12014b8cb00 proto kernel metric 256 pref medium fe80::/64 dev 088817b59cc8f6a proto kernel metric 256 pref medium fe80::/64 dev 419fef5ca083d83 proto kernel metric 256 pref medium fe80::/64 dev da52cd945550f8f proto kernel metric 256 pref medium fe80::/64 dev 7cb723beb053a48 proto kernel metric 256 pref medium fe80::/64 dev 73beb0860a1c557 proto kernel metric 256 pref medium fe80::/64 dev 6ad57518ef0524d proto kernel metric 256 pref medium fe80::/64 dev b8605ccded122bf proto kernel metric 256 pref medium fe80::/64 dev e467dba2f7a7586 proto kernel metric 256 pref medium fe80::/64 dev c717155c8dc0ad0 proto kernel metric 256 pref medium default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium [core@worker-0 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::24 dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd2e:6f44:5dd8:c956::13f dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium [core@worker-1 ~]$ ip -6 r ::1 dev lo proto kernel metric 256 pref medium fd00:1101::2f dev enp4s0 proto kernel metric 100 pref medium fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium fd2e:6f44:5dd8:c956::14d dev enp5s0 proto kernel metric 101 pref medium fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium fe80::/64 dev enp4s0 proto kernel metric 100 pref medium fe80::/64 dev enp5s0 proto kernel metric 101 pref medium default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium Version-Release number of selected component (if applicable): 4.4.0-0.nightly-2020-03-18-102708 How reproducible: 100% Steps to Reproduce: 1. Deploy IPv6 baremetal cluster 2. Check openshift nodes IPv6 routing tables Actual results: Preferred default IPv6 route goes via the node which runs the metal3 pod over the provisioning network. Expected results: Default IPv6 route is set via the baremetal network Additional info: metal3 pod runs dnsmasq binding on the provisioning network which acts as a RA source and provides a default route. We should probably adjust it so that it doesn't provide a default route since the provisioning network is supposed to be isolated.
metal3 dnsmasq.conf [root@master-1 /]# cat /etc/dnsmasq.conf interface=enp4s0 except-interface=lo bind-dynamic enable-tftp tftp-root=/shared/tftpboot # Disable listening for DNS port=0 log-dhcp dhcp-range=fd00:1101::a,fd00:1101::64 # Disable default router(s) and DNS over provisioning network dhcp-option=3 dhcp-option=6 # IPv6 Configuration: enable-ra ra-param=enp4s0,10 dhcp-vendorclass=set:pxe6,enterprise:343,PXEClient dhcp-userclass=set:ipxe6,iPXE dhcp-option=tag:pxe6,option6:bootfile-url,tftp://[fd00:1101::3]/snponly.efi dhcp-option=tag:ipxe6,option6:bootfile-url,http://[fd00:1101::3]:6180/dualboot.ipxe # Disable default router(s) and DNS over provisioning network dhcp-option=3 dhcp-option=6
This mailing list post seems to have more information: https://www.redhat.com/archives/libvir-list/2016-June/msg02203.html My understanding based on reading this is that ra-params is incorrect, and should be 0,0 - not 10 to ensure we don't send any default gateway.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409