Bug 1823602 - Active-Active bonding configuration fails - UPI Bare Metal Installation
Summary: Active-Active bonding configuration fails - UPI Bare Metal Installation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.3.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Dusty Mabe
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-14 01:59 UTC by umesh_sunnapu
Modified: 2020-10-27 15:58 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 15:57:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
active-active bonding entry in PXE using DHCP (51.31 KB, image/png)
2020-04-14 01:59 UTC, umesh_sunnapu
no flags Details
active-active bonding entry in PXE using static IP assignment (39.58 KB, image/png)
2020-04-14 02:00 UTC, umesh_sunnapu
no flags Details
dhcp port turned off in switch (10.22 KB, image/png)
2020-04-16 21:11 UTC, umesh_sunnapu
no flags Details
dhcp port turned on in switch (16.28 KB, image/png)
2020-04-16 21:11 UTC, umesh_sunnapu
no flags Details
RHEL 8.1 IP address check with DHCP port turned on in switch (222.91 KB, image/png)
2020-04-20 18:04 UTC, umesh_sunnapu
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:58:06 UTC

Description umesh_sunnapu 2020-04-14 01:59:40 UTC
Created attachment 1678604 [details]
active-active bonding entry in PXE using DHCP

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
[root@csah csi]# rpm -q openshift-ansible
openshift-ansible-4.3.3-202002142331.git.173.bb0b5a1.el7.noarch

rpm -q ansible
[root@csah csi]# rpm -q ansible
ansible-2.9.5-1.el7ae.noarch

ansible --version
[root@csah csi]# ansible --version
ansible 2.9.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]


How reproducible:

Steps to Reproduce:
When installing bootstrap, control and worker nodes, we try to provide bonding configuration using the interface names manually. This is causing a concern when the following scenario occurs

- consider two interfaces are available for bonding (ens2f0, ens2f1)
- Bonding configuration is provided in the PXE config file. Reference the attachments.
- Things work fine and installation is successful
- IPs are assigned as expected.

Problem Scenario:

- (ens2f0) MAC address used in DHCP to assign IP address goes down
- Access to the node does not get disconnected
- However, if node (say worker node) is rebooted, then IP address is not assigned automatically.
- Scenario is same when assigning IP address through DHCP or static
(Please refer to attached images)


Actual results:
Even if the interface (ens2f0) is down or switch connected to the port is down, IP address should be intact considering bonding is used. But that is not happening and when node reboot happens, IP address is not assigned

Expected results:
IP address should be assigned through DHCP or static when active-active bond is used.

Comment 1 umesh_sunnapu 2020-04-14 02:00:27 UTC
Created attachment 1678605 [details]
active-active bonding entry in PXE using static IP assignment

Comment 2 Abhinav Dahiya 2020-04-14 02:35:01 UTC
booting the RHCOS machine with correct networking is best helped by OS team and therefore moving

Comment 4 Micah Abbott 2020-04-16 14:25:25 UTC
Is it possible to confirm that this configuration is working with RHEL8?


I don't have access to a switch that supports LACP, but I did a quick test using a VM with two NICs configured in a bonded interface.  I installed via the 4.3.8 ISO with the following parameters:

`coreos.inst=yes coreos.inst.install_dev=vda coreos.inst.ignition_url=http://192.168.124.1:9001/ignition.json coreos.inst.image_url=http://192.168.124.1:9001/rhcos-4.3.8-x86_64-metal.x86_64.raw.gz ip=bond0:dhcp bond=bond0:enp1s0,enp2s0:mode=802.3ad,lacp_rate=1,xmit_hash_policy=layer3+4,miimon=100`

I confirmed the bonded interface came up with an IP address on first boot and subsequent boots.  When I disabled the `enp1s0` interface and rebooted, the bonded interface was still able to come up with a DHCP address (though it was a different addresss).

It's not an exact test since I don't have a proper LACP switch, but it seems to indicate the bond is minimally functioning.  Seeing how a RHEL8 host operates in this configuration/environment would be very useful.


## First boot

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.103/24 brd 192.168.124.255 scope global dynamic noprefixroute bond0
       valid_lft 3483sec preferred_lft 3483sec
    inet6 fe80::5054:ff:fef6:f341/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
$ cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: enp1s0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 52:54:00:f6:f3:41
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0

Slave Interface: enp2s0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 52:54:00:de:44:7c
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0


## Second boot

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.103/24 brd 192.168.124.255 scope global dynamic noprefixroute bond0
       valid_lft 3557sec preferred_lft 3557sec
    inet6 fe80::5054:ff:fef6:f341/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

$ cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: enp1s0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 52:54:00:f6:f3:41
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0

Slave Interface: enp2s0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 52:54:00:de:44:7c
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0


## Another reboot, disabled enp1s0

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 52:54:00:de:44:7c brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:de:44:7c brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.104/24 brd 192.168.124.255 scope global dynamic noprefixroute bond0
       valid_lft 3584sec preferred_lft 3584sec
    inet6 fe80::5054:ff:fede:447c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

$ cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: enp2s0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 52:54:00:de:44:7c
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0

Comment 5 umesh_sunnapu 2020-04-16 21:10:18 UTC
Hi Abbott, I just did a test on a RHCOS that is running in OCP 4.3 cluster. Please see the below notes. All my work was done by testing worker1.r5b.oss.labs node.

[core@worker1 ~]$ ip a s | grep -A 2 bond
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
4: ens2f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
5: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
    inet 100.82.42.24/24 brd 100.82.42.255 scope global dynamic noprefixroute bond0
       valid_lft 583sec preferred_lft 583sec
    inet6 fe80::9a03:9bff:fe58:aa12/64 scope link noprefixroute


In the DHCP Config file

host worker1{
  hardware ethernet 98:03:9B:58:AA:12;
  fixed-address 100.82.42.24;
}

DHCP Port (Interface) shutdown in switch-1 

R5-S5232F-1(conf-if-eth1/1/4:1)# show configuration
!
interface ethernet1/1/4:1
 description 5bw1->eno1
 shutdown
 channel-group 13 mode active
 no switchport
 mtu 9216
 flowcontrol receive on
 flowcontrol transmit on


after rebooting I still dont see the IP address. So did a oc get nodes and as can be seen, worker1.r5b.oss.labs is in NotReady status.

[core@r5bcsah ~]$ oc get nodes
NAME                   STATUS     ROLES    AGE   VERSION
etcd-0.r5b.oss.labs    Ready      master   76d   v1.16.2
etcd-1.r5b.oss.labs    Ready      master   76d   v1.16.2
etcd-2.r5b.oss.labs    Ready      master   76d   v1.16.2
worker1.r5b.oss.labs   NotReady   worker   76d   v1.16.2
worker2.r5b.oss.labs   Ready      worker   76d   v1.16.2
worker3.r5b.oss.labs   Ready      worker   73d   v1.16.2


DHCP Port (Interface) no shutdown in switch-1

R5-S5232F-1(conf-if-eth1/1/4:1)# show configuration
!
interface ethernet1/1/4:1
 description 5bw1->eno1
 no shutdown
 channel-group 13 mode active
 no switchport
 mtu 9216
 flowcontrol receive on
 flowcontrol transmit on

As can be seen from the system logs, I dont see the DHCP IP (100.82.42.24) assigned to MAC specified in DHCP config file when interface (port) was shutdown in the switch. After turning it on, can see the IP address assigned by DHCP

[root@r5bcsah uefi]# cat /var/log/messages | grep 100.82.42.24 | tail -n 10
Apr 16 16:17:58 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 16 16:24:38 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 from 98:03:9b:58:aa:12 via bond0
Apr 16 16:24:38 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 16 16:31:18 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 from 98:03:9b:58:aa:12 via bond0
Apr 16 16:31:18 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 16 16:37:58 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 from 98:03:9b:58:aa:12 via bond0
Apr 16 16:37:58 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 16 17:01:44 r5bcsah dhcpd: DHCPOFFER on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 16 17:01:44 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 (100.82.42.20) from 98:03:9b:58:aa:12 via bond0
Apr 16 17:01:44 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0

Note: around 16:37, interface was shutdown in switch. 

checking the status of nodes now

[core@r5bcsah ~]$ oc get nodes
NAME                   STATUS   ROLES    AGE   VERSION
etcd-0.r5b.oss.labs    Ready    master   76d   v1.16.2
etcd-1.r5b.oss.labs    Ready    master   76d   v1.16.2
etcd-2.r5b.oss.labs    Ready    master   76d   v1.16.2
worker1.r5b.oss.labs   Ready    worker   76d   v1.16.2
worker2.r5b.oss.labs   Ready    worker   76d   v1.16.2
worker3.r5b.oss.labs   Ready    worker   73d   v1.16.2

My PXE Entry is as follows

[root@r5bcsah uefi]# cat grub.cfg | grep -i worker1 -A 3
menuentry 'worker1' --class fedora --class gnu-linux --class gnu --class os {
  linux rhcos/4.3/rhcos-4.3.0-x86_64-installer-kernel nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.20:8892/rhcos/4.3/rhcos-4.3.0-x86_64-metal.raw.gz coreos.inst.ignition_url=http://100.82.42.20:8892/ignition/worker.ign bond=bond0:eno1,eno2,ens2f0,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 ip=bond0:dhcp
  initrd rhcos/4.3/rhcos-4.3.0-x86_64-installer-initramfs.img
}


Also refer to the images attached screenshot_with_dhcp_port_turned_off_in_switch, screenshot_with_dhcp_port_turned_on_in_switch where it is seen that when dhcp port is turned off, hostname is showing as 'localhost' instead of 'worker1'.

Comment 6 umesh_sunnapu 2020-04-16 21:11:19 UTC
Created attachment 1679490 [details]
dhcp port turned off in switch

Comment 7 umesh_sunnapu 2020-04-16 21:11:48 UTC
Created attachment 1679491 [details]
dhcp port turned on in switch

Comment 8 umesh_sunnapu 2020-04-16 21:12:53 UTC
[core@r5bcsah ~]$ oc get nodes -o wide
NAME                   STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                CONTAINER-RUNTIME
etcd-0.r5b.oss.labs    Ready    master   76d   v1.16.2   100.82.42.21   <none>        Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa)   4.18.0-147.3.1.el8_1.x86_64   cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8
etcd-1.r5b.oss.labs    Ready    master   76d   v1.16.2   100.82.42.22   <none>        Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa)   4.18.0-147.3.1.el8_1.x86_64   cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8
etcd-2.r5b.oss.labs    Ready    master   76d   v1.16.2   100.82.42.23   <none>        Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa)   4.18.0-147.3.1.el8_1.x86_64   cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8
worker1.r5b.oss.labs   Ready    worker   76d   v1.16.2   100.82.42.24   <none>        Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa)   4.18.0-147.3.1.el8_1.x86_64   cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8
worker2.r5b.oss.labs   Ready    worker   76d   v1.16.2   100.82.42.25   <none>        Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa)   4.18.0-147.3.1.el8_1.x86_64   cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8
worker3.r5b.oss.labs   Ready    worker   73d   v1.16.2   100.82.42.26   <none>        Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa)   4.18.0-147.3.1.el8_1.x86_64   cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8

Comment 9 Micah Abbott 2020-04-17 19:18:37 UTC
Does this configuration work with a RHEL 8 host?

i.e.  Can you install RHEL 8 on similar hardware, with similar network cards configured in a bonded interface and observe the DHCP address being assigned when one of the bond members goes down?

Comment 10 umesh_sunnapu 2020-04-20 14:44:03 UTC
yes. I will go ahead and install RHEL 8 on the same node and try to set IP address using DHCP. Will update with my findings. Also I believe the same error is seen when assigning static IP as well with RHCOS. Hopefully I will have some information and update accordingly.

Comment 11 umesh_sunnapu 2020-04-20 18:03:17 UTC
I installed RHEL 8.1 on the same node that was part of the OCP 4.3 cluster before. Please find the info below 

Commands run in RHEL 8.1 to configure bond

nmcli connection add type bond con-name bond0 ifname bond0 bond.options "lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4"
nmcli con add type ethernet ifname eno1 master bond0
nmcli con add type ethernet ifname eno2 master bond0
nmcli con add type ethernet ifname ens2f0 master bond0
nmcli con add type ethernet ifname ens2f1 master bond0
nmcli connection mod bond0 ipv4.method auto
nmcli connection up bond0
nmcli connection modify bond0 connection.autoconnect yes

log messages from DHCP

Apr 20 13:29:02 r5bcsah dhcpd: DHCPDISCOVER from 98:03:9b:58:aa:12 via bond0
Apr 20 13:29:02 r5bcsah dhcpd: DHCPOFFER on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 20 13:29:02 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 (100.82.42.20) from 98:03:9b:58:aa:12 via bond0
Apr 20 13:29:02 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0

For IP address info at this time, please refer to screenshots. 

Port shutdown for eno1 in switch

R5-S5232F-1(conf-if-eth1/1/4:1)# show configuration
!
interface ethernet1/1/4:1
 description 5bw1->eno1
 shutdown
 channel-group 13 mode active
 no switchport
 mtu 9216
 flowcontrol receive on
 flowcontrol transmit on


Performed a reboot of node.

DHCP log messages during the node coming up

Apr 20 13:38:53 r5bcsah dhcpd: DHCPOFFER on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0
Apr 20 13:38:53 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 (100.82.42.20) from 98:03:9b:58:aa:12 via bond0
Apr 20 13:38:53 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0


RHEL 8.1 node came up with IP address. No issues

[root@rhel81 ~]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
4: ens2f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
5: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:9e:b4:3b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:9e:b4:3b brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff
    inet 100.82.42.24/24 brd 100.82.42.255 scope global dynamic noprefixroute bond0
       valid_lft 598sec preferred_lft 598sec
    inet6 fe80::a69d:83e3:be0c:15cf/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Comment 12 umesh_sunnapu 2020-04-20 18:04:56 UTC
Created attachment 1680347 [details]
RHEL 8.1 IP address check with DHCP port turned on in switch

Comment 14 Dusty Mabe 2020-06-10 03:18:30 UTC
This bug has not been selected for work in the current sprint.

Comment 15 Dusty Mabe 2020-06-26 20:39:49 UTC
Hey Umesh,

I'm looking into this issue now. Unfortunately I don't currently have an environment where I can attempt your exact configuration. If possible could you confirm that booting up a RHCOS node with a simple ignition config (like one where it only contains an SSH key) and running the same set of nmcli commands you ran on RHEL in https://bugzilla.redhat.com/show_bug.cgi?id=1823602#c11 gives you differing behavior than RHEL.

Hopefully I can whittle it down when I get back from vacation (so it will be in a future sprint) and get a reproducer and then we can try to find out what the bug is.

Comment 16 Dusty Mabe 2020-06-26 20:46:10 UTC
This bug has not been selected for work in the current sprint.

Comment 17 umesh_sunnapu 2020-07-06 16:02:13 UTC
Hi Dusty, I will see if I can test the things you asked.

Comment 18 Dusty Mabe 2020-07-21 15:48:54 UTC
Thanks Umesh - any updates here?

Comment 19 umesh_sunnapu 2020-07-27 15:05:55 UTC
Hi Mabe, sorry it took some time to get the hardware and things rolling. I am working with OCP 4.5 currently. It seems like when we set ip=dhcp in grub.cfg as below, an IP address is not assigned when we use DHCP to provision ip address using the MAC address

menuentry 'r5aw3' --class fedora --class gnu-linux --class gnu --class os {
  linuxefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.3/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=bond0:dhcp bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90
  initrdefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img
}

as you can see, the MAC address used to obtain the IP address is different that what the dhcpd.conf file has. 

Jul 27 10:45:27 r5acsah dhcpd: DHCPREQUEST for 100.82.42.96 from 3c:fd:fe:d1:b5:81 via bond0: unknown lease 100.82.42.96.
Jul 27 10:45:36 r5acsah dhcpd: DHCPDISCOVER from 3c:fd:fe:d1:b5:81 via bond0: network 100.82.42.64/26: no free leases
Jul 27 10:45:37 r5acsah named[137941]: network unreachable resolving 'elb.apps.telemeter-prod.a5j2.p1.openshiftapps.com/A/IN': 2600:9000:5307:2a00::1#53
Jul 27 10:45:44 r5acsah dhcpd: DHCPDISCOVER from 3c:fd:fe:d1:b5:81 via bond0: network 100.82.42.64/26: no free leases

Another try I did was to set the static IP address 

menuentry 'r5aw3' --class fedora --class gnu-linux --class gnu --class os {
  linuxefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.3/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=100.82.42.96::100.82.42.65:255.255.255.192:r5aw3.oss.labs:bond0:none bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90
  initrdefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img
}

Jul 27 10:50:25 r5acsah dhcpd: DHCPREQUEST for 100.82.42.96 from 98:03:9b:58:ad:62 via bond0
Jul 27 10:50:25 r5acsah dhcpd: DHCPACK on 100.82.42.96 to 98:03:9b:58:ad:62 via bond0


what I would like to know is, is this a normal behavior. And is there any thing else you want me to test in RHCOS.

Comment 20 Dusty Mabe 2020-07-30 19:29:08 UTC
Hey Umesh, I'm not sure if there is a problem with what you are doing. I'm a bit confused why a DHCPREQUEST would be made when you're using static networking.

Let's start simple. If you don't configure a bond (i.e. just use `ip=dhcp`) and just provide a simple Ignition config like:

```
{
    "ignition": {
        "version": "2.2.0"
    },
    "passwd": {
        "users": [
            {
                "groups": [
                    "sudo"
                ],
                "name": "core",
                "sshAuthorizedKeys": [
                    "ssh-rsa AAAA..."
                ]
            }
        ]
    }
}
```

Does the node install and come up and you're able to login (ssh core@host)?

NOTE: You'll need to replace `ssh-rsa AAAA...` with your ssh pubkey.

Comment 21 Dusty Mabe 2020-07-30 19:37:45 UTC
This is being worked on, but is currently awaiting more investigation or more information and won't be completed this sprint.

Comment 22 umesh_sunnapu 2020-07-31 13:32:49 UTC
Hi Mabe, If I try out the scenario that you have mentioned, things work fine with ip=dhcp. Problem is when we are using a bonding (bond0) with two slave interfaces (eno1 and eno2) in active and active mode. Then when eno1 is shutdown, then dhcp is not provisioning any ip address because the dhcpd.conf is having info about the MAC address of eno1. 

I want to understand if this is a known issue or the behavior is not flawed and this is the way it is supposed to work.

Comment 23 Dusty Mabe 2020-08-01 03:37:02 UTC
(In reply to umesh_sunnapu from comment #22)
> Hi Mabe, If I try out the scenario that you have mentioned, things work fine
> with ip=dhcp. Problem is when we are using a bonding (bond0) with two slave
> interfaces (eno1 and eno2) in active and active mode. Then when eno1 is
> shutdown, then dhcp is not provisioning any ip address because the
> dhcpd.conf is having info about the MAC address of eno1.

Thanks. Those details help me understand.

> 
> I want to understand if this is a known issue or the behavior is not flawed
> and this is the way it is supposed to work.

I'll have to look into it.

Comment 24 Dusty Mabe 2020-08-03 20:28:35 UTC
I did some reading about bonding options in https://www.kernel.org/doc/Documentation/networking/bonding.txt. From that I honestly can't tell what the behavior is for 802.3ad (mode 4) with respect to mac address. We might have to find an expert that knows more.

Does anyone know what the behavior is for 802.3ad bonding:

- by default does it use the same MAC address for all outbound traffic?
- when you lose a link on one of your NICs does it continue to use the same MAC for outbound traffic?

Also Umesh, just in case, you may want to make sure your switches are set up correctly. The docs mention:

```

		Prerequisites:

		1. Ethtool support in the base drivers for retrieving
		the speed and duplex of each slave.

		2. A switch that supports IEEE 802.3ad Dynamic link
		aggregation.

		Most switches will require some type of configuration
		to enable 802.3ad mode.
```

Comment 25 Dusty Mabe 2020-08-03 21:59:20 UTC
In the case that the same MAC isn't used you might want to configure your DHCP server to hand out the same IP to multiple MAC addresses. See "Configuring a Multihomed DHCP Server": https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_a_multihomed_dhcp_server

Comment 26 Jarod Wilson 2020-08-05 16:09:11 UTC
(In reply to Dusty Mabe from comment #24)
> I did some reading about bonding options in
> https://www.kernel.org/doc/Documentation/networking/bonding.txt. From that I
> honestly can't tell what the behavior is for 802.3ad (mode 4) with respect
> to mac address. We might have to find an expert that knows more.
> 
> Does anyone know what the behavior is for 802.3ad bonding:
> 
> - by default does it use the same MAC address for all outbound traffic?

Yes. If you cat /proc/net/bonding/bondX with root privs, you'll see additional data that isn't present when cat'd as a normal user, including the lacp "System MAC address:" (output gated by having capable(CAP_NET_ADMIN)). The "System MAC address" refers to the MAC assigned to the LACP link as a whole.

> - when you lose a link on one of your NICs does it continue to use the same
> MAC for outbound traffic?

Yes.
 
> Also Umesh, just in case, you may want to make sure your switches are set up
> correctly. The docs mention:
> 
> ```
> 
> 		Prerequisites:
> 
> 		1. Ethtool support in the base drivers for retrieving
> 		the speed and duplex of each slave.
> 
> 		2. A switch that supports IEEE 802.3ad Dynamic link
> 		aggregation.
> 
> 		Most switches will require some type of configuration
> 		to enable 802.3ad mode.
> ```

You 100% must have a LACP-capable switch with 802.3ad mode enabled. But I don't think that's the problem here. I think all that is necessary is to specify the desired MAC for the bond, rather than relying on roulette, where whatever device wins the race to get enslaved first is the MAC the LACP as a whole gets assigned.

Comment 27 umesh_sunnapu 2020-08-05 16:54:50 UTC
Hi Mabe

performed a PXE install of RHCOS using the pxe menu with following entry

menuentry 'r5aw1' --class fedora --class gnu-linux --class gnu --class os {
  linuxefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.5/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=bond0:dhcp bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90
  initrdefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img
}

Mac address used in the dhcpd.conf file

[root@r5acsah uefi]# cat /etc/dhcp/dhcpd.conf  | grep -i r5aw1 -A 2
host r5aw1{
   hardware ethernet 98:03:9B:58:A9:22;
   fixed-address 100.82.42.94;

Installation was successful.

[root@r5aw1 network-scripts]# cat /etc/redhat-release
Red Hat Enterprise Linux CoreOS release 4.5

[root@r5aw1 network-scripts]# ip a s | grep -i bond0
3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
4: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0

[root@r5aw1 network-scripts]# cat ifcfg-bond0
# Generated by dracut initrd
NAME="bond0"
DEVICE="bond0"
ONBOOT=yes
NETBOOT=yes
UUID="408e745f-7ff0-43dd-96cd-0d5b8c3c0eb9"
IPV6INIT=yes
BOOTPROTO=dhcp
BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4"
NAME="bond0"
TYPE=Bond
DNS1="100.82.42.90"

Assigned a static mac address to bond0. Used the same mac address that is used in the dhcpd.conf

[root@r5aw1 network-scripts]# nmcli con modify bond0 802-3-ethernet.cloned-mac-address 98:03:9b:58:a9:22

[root@r5aw1 network-scripts]# cat ifcfg-bond0
# Generated by dracut initrd
DEVICE=bond0
ONBOOT=yes
NETBOOT=yes
UUID=408e745f-7ff0-43dd-96cd-0d5b8c3c0eb9
IPV6INIT=yes
BOOTPROTO=dhcp
BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4"
NAME=bond0
TYPE=Bond
DNS1=100.82.42.90
BONDING_MASTER=yes
HWADDR=
MACADDR=98:03:9B:58:A9:22
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no

Performed a shutdown of port eno1 in the switch. eno1 mac address is what is referenced in dhcpd.conf

R5-S5232F-1(conf-if-eth1/1/2:1)# show configuration
!
interface ethernet1/1/2:1
 shutdown
 channel-group 5 mode active
 no switchport
 mtu 9216
 flowcontrol receive on
 flowcontrol transmit on
 lacp port-priority 1

port eno1 is down

[root@r5aw1 network-scripts]# ip a s | grep -i bond0
3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
4: eno1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0

rebooted the node

IP address is working 

[core@r5aw1 ~]$ ip a s | grep -E 'eno1|bond0'
3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
4: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0

now, did a no shutdown on the port eno1 at switch side and noticed eno1 is now acting as a slave to bond0

[core@r5aw1 ~]$ ip a s | grep -i bond0
3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
4: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0


So things seems to be working fine with active-active interfaces and ensuring bond0 is set to the mac address that is being used in dhcpd.conf

what I would like to understand is, if this is a supported approach by RedHat and how to set the mac address in the kernel params for RHCOS installations successfully.

Comment 28 Jarod Wilson 2020-08-05 17:08:23 UTC
(In reply to umesh_sunnapu from comment #27)
...
> So things seems to be working fine with active-active interfaces and
> ensuring bond0 is set to the mac address that is being used in dhcpd.conf

Good, that's what I'd expect.

> what I would like to understand is, if this is a supported approach by
> RedHat and how to set the mac address in the kernel params for RHCOS
> installations successfully.

That one I have no idea about, the bonding driver is my realm, I don't really know RHCOS and the installer/bootloader used to get it up and running, but it sure seems like something we ought to have an option for.

Worst-case, I suspect you could just populate your dhcpd.conf with entries for each of the MACs in the bond, so regardless of which one wins the init race, you still get the same IP. :)

Comment 29 umesh_sunnapu 2020-08-05 17:17:30 UTC
(In reply to Jarod Wilson from comment #28)
> (In reply to umesh_sunnapu from comment #27)
> ...
> > So things seems to be working fine with active-active interfaces and
> > ensuring bond0 is set to the mac address that is being used in dhcpd.conf
> 
> Good, that's what I'd expect.
> 
> > what I would like to understand is, if this is a supported approach by
> > RedHat and how to set the mac address in the kernel params for RHCOS
> > installations successfully.
> 
> That one I have no idea about, the bonding driver is my realm, I don't
> really know RHCOS and the installer/bootloader used to get it up and
> running, but it sure seems like something we ought to have an option for.
> 
> Worst-case, I suspect you could just populate your dhcpd.conf with entries
> for each of the MACs in the bond, so regardless of which one wins the init
> race, you still get the same IP. :)

I am not sure how this will work because now you have set multiple mac addresses with same IP. Dont want them to end up assigning the same IP. Something we need to test more. I am guessing there should be a way to set the mac address for the bond interface through kernel params. Hope @Dusty Mabe has an answer for that :)

Comment 30 Dusty Mabe 2020-08-05 17:38:53 UTC
(In reply to umesh_sunnapu from comment #29)
> (In reply to Jarod Wilson from comment #28)
> > (In reply to umesh_sunnapu from comment #27)
> > ...
> > > So things seems to be working fine with active-active interfaces and
> > > ensuring bond0 is set to the mac address that is being used in dhcpd.conf
> > 
> > Good, that's what I'd expect.
> > 
> > > what I would like to understand is, if this is a supported approach by
> > > RedHat and how to set the mac address in the kernel params for RHCOS
> > > installations successfully.
> > 
> > That one I have no idea about, the bonding driver is my realm, I don't
> > really know RHCOS and the installer/bootloader used to get it up and
> > running, but it sure seems like something we ought to have an option for.

RHCOS "is" RHEL, so we should support the same bonding options. Yes it's supported.

> > 
> > Worst-case, I suspect you could just populate your dhcpd.conf with entries
> > for each of the MACs in the bond, so regardless of which one wins the init
> > race, you still get the same IP. :)
> 
> I am not sure how this will work because now you have set multiple mac
> addresses with same IP. Dont want them to end up assigning the same IP.

On the DHCP server side you can do this: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_a_multihomed_dhcp_server

But you won't need to do that if we can get the mac address set for you so let's concentrate on that approach for now.


> Something we need to test more. I am guessing there should be a way to set
> the mac address for the bond interface through kernel params. Hope @Dusty
> Mabe has an answer for that :)

Yep I'll look into if this is possible using `ip=` kargs.

Comment 31 Dusty Mabe 2020-08-06 03:27:27 UTC
(In reply to Dusty Mabe from comment #30)
> (In reply to umesh_sunnapu from comment #29)

> 
> > Something we need to test more. I am guessing there should be a way to set
> > the mac address for the bond interface through kernel params. Hope @Dusty
> > Mabe has an answer for that :)
> 
> Yep I'll look into if this is possible using `ip=` kargs.

You should be able to do this like:

```
ip=bond0:dhcp::98:03:9B:58:A9:22 bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4
```

Please let me know if this works for you and we can close this out.

Comment 32 umesh_sunnapu 2020-08-06 13:17:37 UTC
@Dusty Mabe, It does not look like things worked. Below is the bond network config file and as you can see, the mac address is not setup. 

[root@r5aw1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
# Generated by dracut initrd
NAME="bond0"
DEVICE="bond0"
ONBOOT=yes
NETBOOT=yes
UUID="44194d7b-59b6-42ab-a437-565b8231f8ee"
IPV6INIT=yes
BOOTPROTO=dhcp
BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4"
NAME="bond0"
TYPE=Bond
DNS1="100.82.42.90"

I tried shutting down the port at switch level, rebooted the node and I can see that the mac address was pointing to secondary active slave nic port instead of what we tried to assign bond manually.

Below is what I have in the grub.cfg file

menuentry 'r5aw1' --class fedora --class gnu-linux --class gnu --class os {
  linuxefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.5/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=bond0:dhcp::98:03:9B:58:A9:22 bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90
  initrdefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img
}

Comment 33 Dusty Mabe 2020-08-06 15:25:32 UTC
(In reply to umesh_sunnapu from comment #32)
> @Dusty Mabe, It does not look like things worked. Below is the bond network
> config file and as you can see, the mac address is not setup. 
> 
> [root@r5aw1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
> # Generated by dracut initrd
> NAME="bond0"
> DEVICE="bond0"
> ONBOOT=yes
> NETBOOT=yes
> UUID="44194d7b-59b6-42ab-a437-565b8231f8ee"
> IPV6INIT=yes
> BOOTPROTO=dhcp
> BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4"
> NAME="bond0"
> TYPE=Bond
> DNS1="100.82.42.90"


Fun. OK, this worked when I tested it, but I tested it against RHCOS 4.6.

In 4.6 we switched our initramfs networking (including the thing that parses
the arguments) over to NetworkManager. In 4.5 we still use the legacy
initscripts stack in the initramfs.

So what you're hitting now is a bug in the legacy implementation. I
just opened https://bugzilla.redhat.com/show_bug.cgi?id=1866851 for
that, but it won't matter much (for RHCOS) because it's already "fixed"
in 4.6 with the move to NM.

For now you probably either need to provide files via ignition or run `nmcli`
once on first boot to add the MAC address.

See https://dustymabe.com/2019/11/06/running-a-script-on-bootup-via-ignition/

Comment 34 umesh_sunnapu 2020-08-07 16:16:30 UTC
Thanks Mabe for the info. If that is the case, I will go ahead and put this bug in the guides going forward for OCP 4.5 and mention the need to use static network ip assignment instead of using DHCP. Thats the work around I can think of. 

And for OCP 4.6, we can go with the approach of setting the bond with a specific mac address so that customers can either go with static or dhcp based ip assignment for bond interface.

Thoughts ?

Comment 35 Dusty Mabe 2020-08-07 16:35:55 UTC
You can use static networking. Or you can just not provide networking information via ip= kernel arguments and provide it via ignition instead. During the initramfs it will default to DHCP so you'll still be able to retrieve the config during the initramfs with the defaults (dhcp by default) and then have the appropriate configs applied via files. The files will cause NetworkManager to bring up the networking with bond and all in the real root. So it should be safe.

Comment 36 umesh_sunnapu 2020-08-07 16:46:29 UTC
Agree Mabe. But creating ignition config files and applying them after the RHCOS worker node is installed, is multiple steps. I can point them to you process but again it should be more like, creating a machine config and ensure that is applied to the machine config pools where the worker node belongs to. 

And if the worker nodes in the machine config pool has a combination of RHEL and RHCOS worker nodes, then it becomes a bit more tricky to ensure only RHCOS worker nodes get this ignition config file but for RHEL worker nodes, it is a different process.

Comment 37 Dusty Mabe 2020-08-07 17:18:05 UTC
Yes. Just making sure you know all of the options that are available. Looking forward to this being fixed with the move to NetworkManager in 4.6.

I'll go ahead and mark this as fixed in 4.6 and we'll have someone from the team confirm the behavior I describe.

Comment 38 umesh_sunnapu 2020-08-07 17:24:07 UTC
Cool. Please go ahead with your next steps.

thanks Mabe for all your support.

Comment 39 Dusty Mabe 2020-08-07 17:32:52 UTC
Active-Active bonds will by default use the MAC address of the first enslaved NIC for the bond and all interfaces that belong to the bond. If the first enslaved NIC loses link then the bond will continue to use its MAC address, until a reboot. On reboot the bond will come up and use the first enslaved NIC's MAC address, but in this case it won't be the same as the first boot of the machine because that NIC no longer has link. Thus, the MAC address of the bond has changed.

In order to keep the MAC address of the bond the same we can explicitly set the MAC address of the bond, which will apply to the bond interface and all enslaved NIC interfaces. This is supported today via `ip=` kernel arguments, but there is a bug [1] that causes the MAC address to not get applied to the configuration when you are doing this with a DHCP configuration (i.e., `ip=bond0:dhcp::aa:bb:cc:dd:ff:11`). The bug does not apply to 4.6 because we are using NetworkManager in the initramfs in 4.6, which does not use the same tools to create the networking configuration.

I will mark this as closed in 4.6. 

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1866851

Comment 42 Renata Ravanelli 2020-09-17 18:46:24 UTC
>Active-Active bonds will by default use the MAC address of the first enslaved NIC for the 
>bond and all interfaces that belong to the bond. If the first enslaved NIC loses link then 
>the bond will continue to use its MAC address, until a reboot. On reboot the bond will come 
>up and use the first enslaved NIC's MAC address, but in this case it won't be the same 
>as the first boot of the machine because that NIC no longer has link. Thus, the MAC address
>of the bond has changed.

I have checked it and confirmed this behavior in my tests via libvirt with RHCOS 4.6.

- Using the Kernel parameters: ip=bond0:dhcp

    - On the first boot I have all MACs as following:
        [core@localhost ~]$ ip a | grep link
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        link/ether 52:54:00:d8:53:8b brd ff:ff:ff:ff:ff:ff
        link/ether 52:54:00:d8:53:8b brd ff:ff:ff:ff:ff:ff
        ink/ether 52:54:00:d8:53:8b brd ff:ff:ff:ff:ff:ff
    - Turned NIC down, all MACs still the same. 
      You can use: virsh domif-setlink coreos-nettest 52:54:00:d8:53:8b down
      or even: ip link set vnet0
    - Reboot the server
    - As Dusty explained, when the server came up it updated the bond MAC with the first enslaved
      NIC MAC address available, the second NIC in the bond in this case:
          ens2:  52:54:00:d8:53:8b
          ens3:  52:54:00:dc:29:0e
          bond0: 52:54:00:dc:29:0e

>In order to keep the MAC address of the bond the same we can explicitly set the MAC address
>of the bond, which will apply to the bond interface and all enslaved NIC interfaces.
>This is supported today via `ip=` kernel arguments, but there is a bug [1] that causes the MAC address
>to not get applied to the configuration when you are doing this with a DHCP configuration (i.e., `ip=bond0:dhcp::aa:bb:cc:dd:ff:11`). The bug does not apply to 4.6 because we are using NetworkManager
>in the initramfs in 4.6, which does not use the same tools to create the networking configuration.

- Using the Kernel parameters: ip=bond0:dhcp::52:54:00:00:a6:00
   - NetworkManager updated it:
        NetworkManager/system-connections/bond0.nmconnection:cloned-mac-address=52:54:00:00:A6:00

    - On the first boot I have all MACs as following:
        [core@localhost ~]$ ip a | grep link
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        link/ether 52:54:00:00:a6:00 brd ff:ff:ff:ff:ff:ff
        link/ether 52:54:00:00:a6:00 brd ff:ff:ff:ff:ff:ff
        ink/ether 52:54:00:00:a6:00 brd ff:ff:ff:ff:ff:ff
    - Turned NIC down, all MACs still the same (virsh domif-setlink coreos-nettest 52:54:00:00:a6:00 down).
    - Reboot the server
    - The ens2 was updated in order to reflect the first enslaved NIC MAC address. However, ens3 and bond0 still the same.
        ens2: 52:54:00:ce:ae:92
        ens3: 52:54:00:00:a6:00
        bond0: 52:54:00:00:a6:00

Comment 43 Dusty Mabe 2020-09-18 00:48:42 UTC
Thanks Renata. Moving to verified.

Comment 44 Renata Ravanelli 2020-09-18 01:29:22 UTC
>I have checked it and confirmed this behavior in my tests via libvirt with RHCOS 4.6.
 Just to clarify, the full version I used on the tests is rhcos-46.82.202009150240-0

Comment 46 errata-xmlrpc 2020-10-27 15:57:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.