Bug 1814901 - In a baremetal IPv6 deployment egress ipv6 traffic is routed via the node that is running the metal3 pod over the provisioning network
Summary: In a baremetal IPv6 deployment egress ipv6 traffic is routed via the node tha...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.5.0
Assignee: Stephen Benjamin
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks: 1771572 1821443
TreeView+ depends on / blocked
 
Reported: 2020-03-19 00:51 UTC by Marius Cornea
Modified: 2020-07-13 17:23 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1821443 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:22:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ironic-image pull 71 0 None closed Bug 1814901: Disable default gateway for provisioning network in IPv6 2020-06-03 11:49:37 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:23:02 UTC

Description Marius Cornea 2020-03-19 00:51:51 UTC
Description of problem:

default ipv6 route on openshift nodes is set via bootstrap vm or node where metal3 pod is running due to router advertisements enabled by dnsmasq.

As a result all external ipv6 traffic is routed through the node that is running the metal3 pod, over the provisioning network.

Checking routing tables during deployment while bootstrap VM is still running:

[core@master-0 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::62 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd2e:6f44:5dd8:c956::10 dev enp5s0 proto kernel metric 256 pref medium
fd2e:6f44:5dd8:c956::107 dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
default via fe80::8b62:cb84:62e0:735f dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium

[core@master-1 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::63 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd2e:6f44:5dd8:c956::10d dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
default via fe80::8b62:cb84:62e0:735f dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium


[core@master-2 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::14 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd2e:6f44:5dd8:c956::121 dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
default via fe80::8b62:cb84:62e0:735f dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium

fe80::8b62:cb84:62e0:735f is the link local IP address of the provisioning interface of the boostrap VM

After the boostrap VM has been destroyed and the metal3 pod was created we can see the following routing tables:


oc -n openshift-machine-api get pods/metal3-6bf6fdbd65-vvzwg -o yaml | grep nodeName
  nodeName: master-0.ocp-edge-cluster.qe.lab.redhat.com


master-0 is the node running the metal3 pod

[core@master-0 ~]$ ip a s dev enp4s0
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:c8:47:99 brd ff:ff:ff:ff:ff:ff
    inet6 fd00:1101::3/64 scope global dynamic 
       valid_lft 9sec preferred_lft 9sec
    inet6 fe80::44f1:17a0:a00f:2a5d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[core@master-0 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::62 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto kernel metric 256 expires 5sec pref medium
fd01:0:0:2::/64 dev k8s-master-0.oc proto kernel metric 256 pref medium
fd01::/48 via fd01:0:0:2::1 dev k8s-master-0.oc metric 1024 pref medium
fd02::/112 via fd01:0:0:2::1 dev k8s-master-0.oc metric 1024 pref medium
fd2e:6f44:5dd8:c956::5 dev enp5s0 proto kernel metric 256 pref medium
fd2e:6f44:5dd8:c956::10 dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::10 dev enp5s0 proto kernel metric 256 pref medium
fd2e:6f44:5dd8:c956::107 dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fd99::/64 dev br-nexthop proto kernel metric 256 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
fe80::/64 dev br-local proto kernel metric 256 pref medium
fe80::/64 dev fe842d7ee7ea0c6 proto kernel metric 256 pref medium
fe80::/64 dev 31b9f57ba76bc40 proto kernel metric 256 pref medium
fe80::/64 dev b5e1ff654e65d25 proto kernel metric 256 pref medium
fe80::/64 dev e21733449b0560a proto kernel metric 256 pref medium
fe80::/64 dev 5ed89f179633d2b proto kernel metric 256 pref medium
fe80::/64 dev ed31353892b89a2 proto kernel metric 256 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium


[core@master-1 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::63 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd01:0:0:1::/64 dev k8s-master-1.oc proto kernel metric 256 pref medium
fd01::/48 via fd01:0:0:1::1 dev k8s-master-1.oc metric 1024 pref medium
fd02::/112 via fd01:0:0:1::1 dev k8s-master-1.oc metric 1024 pref medium
fd2e:6f44:5dd8:c956::2 dev enp5s0 proto kernel metric 256 pref medium
fd2e:6f44:5dd8:c956::10d dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fd99::/64 dev br-nexthop proto kernel metric 256 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
fe80::/64 dev br-local proto kernel metric 256 pref medium
fe80::/64 dev 1593d53806bcbd1 proto kernel metric 256 pref medium
fe80::/64 dev 30ba70200adb733 proto kernel metric 256 pref medium
fe80::/64 dev a3543ff01278239 proto kernel metric 256 pref medium
fe80::/64 dev af47181644178db proto kernel metric 256 pref medium
default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium

[core@master-2 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::14 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd01:0:0:3::/64 dev k8s-master-2.oc proto kernel metric 256 pref medium
fd01::/48 via fd01:0:0:3::1 dev k8s-master-2.oc metric 1024 pref medium
fd02::/112 via fd01:0:0:3::1 dev k8s-master-2.oc metric 1024 pref medium
fd2e:6f44:5dd8:c956::121 dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fd99::/64 dev br-nexthop proto kernel metric 256 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
fe80::/64 dev br-local proto kernel metric 256 pref medium
fe80::/64 dev 2ea1fe1f24a8305 proto kernel metric 256 pref medium
fe80::/64 dev 21288b1f7e153c0 proto kernel metric 256 pref medium
fe80::/64 dev 54730d44a48d91a proto kernel metric 256 pref medium
fe80::/64 dev ebad5cd7dc36bf7 proto kernel metric 256 pref medium
fe80::/64 dev 37d0ed44970e690 proto kernel metric 256 pref medium
fe80::/64 dev 33b2806d233a1ab proto kernel metric 256 pref medium
fe80::/64 dev c079218f097c991 proto kernel metric 256 pref medium
fe80::/64 dev 96e6e8393e3ee17 proto kernel metric 256 pref medium
fe80::/64 dev 9582f2678237e4f proto kernel metric 256 pref medium
fe80::/64 dev 451f5f36dfb3a3d proto kernel metric 256 pref medium
fe80::/64 dev 9eb3d6c51c0720d proto kernel metric 256 pref medium
fe80::/64 dev 87111a34a098d15 proto kernel metric 256 pref medium
fe80::/64 dev 7325cc2e2aeb4c2 proto kernel metric 256 pref medium
fe80::/64 dev d36d12014b8cb00 proto kernel metric 256 pref medium
fe80::/64 dev 088817b59cc8f6a proto kernel metric 256 pref medium
fe80::/64 dev 419fef5ca083d83 proto kernel metric 256 pref medium
fe80::/64 dev da52cd945550f8f proto kernel metric 256 pref medium
fe80::/64 dev 7cb723beb053a48 proto kernel metric 256 pref medium
fe80::/64 dev 73beb0860a1c557 proto kernel metric 256 pref medium
fe80::/64 dev 6ad57518ef0524d proto kernel metric 256 pref medium
fe80::/64 dev b8605ccded122bf proto kernel metric 256 pref medium
fe80::/64 dev e467dba2f7a7586 proto kernel metric 256 pref medium
fe80::/64 dev c717155c8dc0ad0 proto kernel metric 256 pref medium
default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium

[core@worker-0 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::24 dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd2e:6f44:5dd8:c956::13f dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium

[core@worker-1 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
fd00:1101::2f dev enp4s0 proto kernel metric 100 pref medium
fd00:1101::/64 dev enp4s0 proto ra metric 100 pref medium
fd2e:6f44:5dd8:c956::14d dev enp5s0 proto kernel metric 101 pref medium
fd2e:6f44:5dd8:c956::/64 dev enp5s0 proto ra metric 101 pref medium
fe80::/64 dev enp4s0 proto kernel metric 100 pref medium
fe80::/64 dev enp5s0 proto kernel metric 101 pref medium
default via fe80::44f1:17a0:a00f:2a5d dev enp4s0 proto ra metric 100 pref medium
default via fe80::5054:ff:fe8f:e34a dev enp5s0 proto ra metric 101 pref medium


Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-03-18-102708

How reproducible:
100%

Steps to Reproduce:
1. Deploy IPv6 baremetal cluster
2. Check openshift nodes IPv6 routing tables

Actual results:
Preferred default IPv6 route goes via the node which runs the metal3 pod over the provisioning network.

Expected results:
Default IPv6 route is set via the baremetal network

Additional info:
metal3 pod runs dnsmasq binding on the provisioning network which acts as a RA source and provides a default route. 

We should probably adjust it so that it doesn't provide a default route since the provisioning network is supposed to be isolated.

Comment 1 Marius Cornea 2020-03-19 01:51:44 UTC
metal3 dnsmasq.conf


[root@master-1 /]# cat /etc/dnsmasq.conf
interface=enp4s0
except-interface=lo
bind-dynamic
enable-tftp
tftp-root=/shared/tftpboot

# Disable listening for DNS
port=0
log-dhcp
dhcp-range=fd00:1101::a,fd00:1101::64

# Disable default router(s) and DNS over provisioning network
dhcp-option=3
dhcp-option=6


# IPv6 Configuration:
enable-ra
ra-param=enp4s0,10

dhcp-vendorclass=set:pxe6,enterprise:343,PXEClient
dhcp-userclass=set:ipxe6,iPXE
dhcp-option=tag:pxe6,option6:bootfile-url,tftp://[fd00:1101::3]/snponly.efi
dhcp-option=tag:ipxe6,option6:bootfile-url,http://[fd00:1101::3]:6180/dualboot.ipxe

# Disable default router(s) and DNS over provisioning network
dhcp-option=3
dhcp-option=6

Comment 2 Stephen Benjamin 2020-03-19 13:13:59 UTC
This mailing list post seems to have more information: https://www.redhat.com/archives/libvir-list/2016-June/msg02203.html

My understanding based on reading this is that ra-params is incorrect, and should be 0,0 - not 10 to ensure we don't send any default gateway.

Comment 6 errata-xmlrpc 2020-07-13 17:22:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.