Hide Forgot
Created attachment 1678604 [details] active-active bonding entry in PXE using DHCP Description of problem: Version-Release number of the following components: rpm -q openshift-ansible [root@csah csi]# rpm -q openshift-ansible openshift-ansible-4.3.3-202002142331.git.173.bb0b5a1.el7.noarch rpm -q ansible [root@csah csi]# rpm -q ansible ansible-2.9.5-1.el7ae.noarch ansible --version [root@csah csi]# ansible --version ansible 2.9.5 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] How reproducible: Steps to Reproduce: When installing bootstrap, control and worker nodes, we try to provide bonding configuration using the interface names manually. This is causing a concern when the following scenario occurs - consider two interfaces are available for bonding (ens2f0, ens2f1) - Bonding configuration is provided in the PXE config file. Reference the attachments. - Things work fine and installation is successful - IPs are assigned as expected. Problem Scenario: - (ens2f0) MAC address used in DHCP to assign IP address goes down - Access to the node does not get disconnected - However, if node (say worker node) is rebooted, then IP address is not assigned automatically. - Scenario is same when assigning IP address through DHCP or static (Please refer to attached images) Actual results: Even if the interface (ens2f0) is down or switch connected to the port is down, IP address should be intact considering bonding is used. But that is not happening and when node reboot happens, IP address is not assigned Expected results: IP address should be assigned through DHCP or static when active-active bond is used.
Created attachment 1678605 [details] active-active bonding entry in PXE using static IP assignment
booting the RHCOS machine with correct networking is best helped by OS team and therefore moving
Is it possible to confirm that this configuration is working with RHEL8? I don't have access to a switch that supports LACP, but I did a quick test using a VM with two NICs configured in a bonded interface. I installed via the 4.3.8 ISO with the following parameters: `coreos.inst=yes coreos.inst.install_dev=vda coreos.inst.ignition_url=http://192.168.124.1:9001/ignition.json coreos.inst.image_url=http://192.168.124.1:9001/rhcos-4.3.8-x86_64-metal.x86_64.raw.gz ip=bond0:dhcp bond=bond0:enp1s0,enp2s0:mode=802.3ad,lacp_rate=1,xmit_hash_policy=layer3+4,miimon=100` I confirmed the bonded interface came up with an IP address on first boot and subsequent boots. When I disabled the `enp1s0` interface and rebooted, the bonded interface was still able to come up with a DHCP address (though it was a different addresss). It's not an exact test since I don't have a proper LACP switch, but it seems to indicate the bond is minimally functioning. Seeing how a RHEL8 host operates in this configuration/environment would be very useful. ## First boot $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff 3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff 5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff inet 192.168.124.103/24 brd 192.168.124.255 scope global dynamic noprefixroute bond0 valid_lft 3483sec preferred_lft 3483sec inet6 fe80::5054:ff:fef6:f341/64 scope link noprefixroute valid_lft forever preferred_lft forever $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Min links: 0 Aggregator selection policy (ad_select): stable Slave Interface: enp1s0 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 52:54:00:f6:f3:41 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: monitoring Partner Churn State: monitoring Actor Churned Count: 0 Partner Churned Count: 0 Slave Interface: enp2s0 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 52:54:00:de:44:7c Slave queue ID: 0 Aggregator ID: 2 Actor Churn State: monitoring Partner Churn State: monitoring Actor Churned Count: 0 Partner Churned Count: 0 ## Second boot $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff 3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff inet 192.168.124.103/24 brd 192.168.124.255 scope global dynamic noprefixroute bond0 valid_lft 3557sec preferred_lft 3557sec inet6 fe80::5054:ff:fef6:f341/64 scope link noprefixroute valid_lft forever preferred_lft forever $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Min links: 0 Aggregator selection policy (ad_select): stable Slave Interface: enp1s0 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 52:54:00:f6:f3:41 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: monitoring Partner Churn State: monitoring Actor Churned Count: 0 Partner Churned Count: 0 Slave Interface: enp2s0 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 52:54:00:de:44:7c Slave queue ID: 0 Aggregator ID: 2 Actor Churn State: monitoring Partner Churn State: monitoring Actor Churned Count: 0 Partner Churned Count: 0 ## Another reboot, disabled enp1s0 $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000 link/ether 52:54:00:f6:f3:41 brd ff:ff:ff:ff:ff:ff 3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 52:54:00:de:44:7c brd ff:ff:ff:ff:ff:ff 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:de:44:7c brd ff:ff:ff:ff:ff:ff inet 192.168.124.104/24 brd 192.168.124.255 scope global dynamic noprefixroute bond0 valid_lft 3584sec preferred_lft 3584sec inet6 fe80::5054:ff:fede:447c/64 scope link noprefixroute valid_lft forever preferred_lft forever $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Min links: 0 Aggregator selection policy (ad_select): stable Slave Interface: enp2s0 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 52:54:00:de:44:7c Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: monitoring Partner Churn State: monitoring Actor Churned Count: 0 Partner Churned Count: 0
Hi Abbott, I just did a test on a RHCOS that is running in OCP 4.3 cluster. Please see the below notes. All my work was done by testing worker1.r5b.oss.labs node. [core@worker1 ~]$ ip a s | grep -A 2 bond 2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 4: ens2f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 5: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff inet 100.82.42.24/24 brd 100.82.42.255 scope global dynamic noprefixroute bond0 valid_lft 583sec preferred_lft 583sec inet6 fe80::9a03:9bff:fe58:aa12/64 scope link noprefixroute In the DHCP Config file host worker1{ hardware ethernet 98:03:9B:58:AA:12; fixed-address 100.82.42.24; } DHCP Port (Interface) shutdown in switch-1 R5-S5232F-1(conf-if-eth1/1/4:1)# show configuration ! interface ethernet1/1/4:1 description 5bw1->eno1 shutdown channel-group 13 mode active no switchport mtu 9216 flowcontrol receive on flowcontrol transmit on after rebooting I still dont see the IP address. So did a oc get nodes and as can be seen, worker1.r5b.oss.labs is in NotReady status. [core@r5bcsah ~]$ oc get nodes NAME STATUS ROLES AGE VERSION etcd-0.r5b.oss.labs Ready master 76d v1.16.2 etcd-1.r5b.oss.labs Ready master 76d v1.16.2 etcd-2.r5b.oss.labs Ready master 76d v1.16.2 worker1.r5b.oss.labs NotReady worker 76d v1.16.2 worker2.r5b.oss.labs Ready worker 76d v1.16.2 worker3.r5b.oss.labs Ready worker 73d v1.16.2 DHCP Port (Interface) no shutdown in switch-1 R5-S5232F-1(conf-if-eth1/1/4:1)# show configuration ! interface ethernet1/1/4:1 description 5bw1->eno1 no shutdown channel-group 13 mode active no switchport mtu 9216 flowcontrol receive on flowcontrol transmit on As can be seen from the system logs, I dont see the DHCP IP (100.82.42.24) assigned to MAC specified in DHCP config file when interface (port) was shutdown in the switch. After turning it on, can see the IP address assigned by DHCP [root@r5bcsah uefi]# cat /var/log/messages | grep 100.82.42.24 | tail -n 10 Apr 16 16:17:58 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 16 16:24:38 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 from 98:03:9b:58:aa:12 via bond0 Apr 16 16:24:38 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 16 16:31:18 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 from 98:03:9b:58:aa:12 via bond0 Apr 16 16:31:18 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 16 16:37:58 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 from 98:03:9b:58:aa:12 via bond0 Apr 16 16:37:58 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 16 17:01:44 r5bcsah dhcpd: DHCPOFFER on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 16 17:01:44 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 (100.82.42.20) from 98:03:9b:58:aa:12 via bond0 Apr 16 17:01:44 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Note: around 16:37, interface was shutdown in switch. checking the status of nodes now [core@r5bcsah ~]$ oc get nodes NAME STATUS ROLES AGE VERSION etcd-0.r5b.oss.labs Ready master 76d v1.16.2 etcd-1.r5b.oss.labs Ready master 76d v1.16.2 etcd-2.r5b.oss.labs Ready master 76d v1.16.2 worker1.r5b.oss.labs Ready worker 76d v1.16.2 worker2.r5b.oss.labs Ready worker 76d v1.16.2 worker3.r5b.oss.labs Ready worker 73d v1.16.2 My PXE Entry is as follows [root@r5bcsah uefi]# cat grub.cfg | grep -i worker1 -A 3 menuentry 'worker1' --class fedora --class gnu-linux --class gnu --class os { linux rhcos/4.3/rhcos-4.3.0-x86_64-installer-kernel nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.20:8892/rhcos/4.3/rhcos-4.3.0-x86_64-metal.raw.gz coreos.inst.ignition_url=http://100.82.42.20:8892/ignition/worker.ign bond=bond0:eno1,eno2,ens2f0,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 ip=bond0:dhcp initrd rhcos/4.3/rhcos-4.3.0-x86_64-installer-initramfs.img } Also refer to the images attached screenshot_with_dhcp_port_turned_off_in_switch, screenshot_with_dhcp_port_turned_on_in_switch where it is seen that when dhcp port is turned off, hostname is showing as 'localhost' instead of 'worker1'.
Created attachment 1679490 [details] dhcp port turned off in switch
Created attachment 1679491 [details] dhcp port turned on in switch
[core@r5bcsah ~]$ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME etcd-0.r5b.oss.labs Ready master 76d v1.16.2 100.82.42.21 <none> Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.18.0-147.3.1.el8_1.x86_64 cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 etcd-1.r5b.oss.labs Ready master 76d v1.16.2 100.82.42.22 <none> Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.18.0-147.3.1.el8_1.x86_64 cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 etcd-2.r5b.oss.labs Ready master 76d v1.16.2 100.82.42.23 <none> Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.18.0-147.3.1.el8_1.x86_64 cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 worker1.r5b.oss.labs Ready worker 76d v1.16.2 100.82.42.24 <none> Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.18.0-147.3.1.el8_1.x86_64 cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 worker2.r5b.oss.labs Ready worker 76d v1.16.2 100.82.42.25 <none> Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.18.0-147.3.1.el8_1.x86_64 cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 worker3.r5b.oss.labs Ready worker 73d v1.16.2 100.82.42.26 <none> Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.18.0-147.3.1.el8_1.x86_64 cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8
Does this configuration work with a RHEL 8 host? i.e. Can you install RHEL 8 on similar hardware, with similar network cards configured in a bonded interface and observe the DHCP address being assigned when one of the bond members goes down?
yes. I will go ahead and install RHEL 8 on the same node and try to set IP address using DHCP. Will update with my findings. Also I believe the same error is seen when assigning static IP as well with RHCOS. Hopefully I will have some information and update accordingly.
I installed RHEL 8.1 on the same node that was part of the OCP 4.3 cluster before. Please find the info below Commands run in RHEL 8.1 to configure bond nmcli connection add type bond con-name bond0 ifname bond0 bond.options "lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4" nmcli con add type ethernet ifname eno1 master bond0 nmcli con add type ethernet ifname eno2 master bond0 nmcli con add type ethernet ifname ens2f0 master bond0 nmcli con add type ethernet ifname ens2f1 master bond0 nmcli connection mod bond0 ipv4.method auto nmcli connection up bond0 nmcli connection modify bond0 connection.autoconnect yes log messages from DHCP Apr 20 13:29:02 r5bcsah dhcpd: DHCPDISCOVER from 98:03:9b:58:aa:12 via bond0 Apr 20 13:29:02 r5bcsah dhcpd: DHCPOFFER on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 20 13:29:02 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 (100.82.42.20) from 98:03:9b:58:aa:12 via bond0 Apr 20 13:29:02 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 For IP address info at this time, please refer to screenshots. Port shutdown for eno1 in switch R5-S5232F-1(conf-if-eth1/1/4:1)# show configuration ! interface ethernet1/1/4:1 description 5bw1->eno1 shutdown channel-group 13 mode active no switchport mtu 9216 flowcontrol receive on flowcontrol transmit on Performed a reboot of node. DHCP log messages during the node coming up Apr 20 13:38:53 r5bcsah dhcpd: DHCPOFFER on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 Apr 20 13:38:53 r5bcsah dhcpd: DHCPREQUEST for 100.82.42.24 (100.82.42.20) from 98:03:9b:58:aa:12 via bond0 Apr 20 13:38:53 r5bcsah dhcpd: DHCPACK on 100.82.42.24 to 98:03:9b:58:aa:12 via bond0 RHEL 8.1 node came up with IP address. No issues [root@rhel81 ~]# ip a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 4: ens2f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 5: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff 6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:9e:b4:3b brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:9e:b4:3b brd ff:ff:ff:ff:ff:ff 8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 98:03:9b:58:aa:12 brd ff:ff:ff:ff:ff:ff inet 100.82.42.24/24 brd 100.82.42.255 scope global dynamic noprefixroute bond0 valid_lft 598sec preferred_lft 598sec inet6 fe80::a69d:83e3:be0c:15cf/64 scope link noprefixroute valid_lft forever preferred_lft forever
Created attachment 1680347 [details] RHEL 8.1 IP address check with DHCP port turned on in switch
This bug has not been selected for work in the current sprint.
Hey Umesh, I'm looking into this issue now. Unfortunately I don't currently have an environment where I can attempt your exact configuration. If possible could you confirm that booting up a RHCOS node with a simple ignition config (like one where it only contains an SSH key) and running the same set of nmcli commands you ran on RHEL in https://bugzilla.redhat.com/show_bug.cgi?id=1823602#c11 gives you differing behavior than RHEL. Hopefully I can whittle it down when I get back from vacation (so it will be in a future sprint) and get a reproducer and then we can try to find out what the bug is.
Hi Dusty, I will see if I can test the things you asked.
Thanks Umesh - any updates here?
Hi Mabe, sorry it took some time to get the hardware and things rolling. I am working with OCP 4.5 currently. It seems like when we set ip=dhcp in grub.cfg as below, an IP address is not assigned when we use DHCP to provision ip address using the MAC address menuentry 'r5aw3' --class fedora --class gnu-linux --class gnu --class os { linuxefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.3/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=bond0:dhcp bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90 initrdefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img } as you can see, the MAC address used to obtain the IP address is different that what the dhcpd.conf file has. Jul 27 10:45:27 r5acsah dhcpd: DHCPREQUEST for 100.82.42.96 from 3c:fd:fe:d1:b5:81 via bond0: unknown lease 100.82.42.96. Jul 27 10:45:36 r5acsah dhcpd: DHCPDISCOVER from 3c:fd:fe:d1:b5:81 via bond0: network 100.82.42.64/26: no free leases Jul 27 10:45:37 r5acsah named[137941]: network unreachable resolving 'elb.apps.telemeter-prod.a5j2.p1.openshiftapps.com/A/IN': 2600:9000:5307:2a00::1#53 Jul 27 10:45:44 r5acsah dhcpd: DHCPDISCOVER from 3c:fd:fe:d1:b5:81 via bond0: network 100.82.42.64/26: no free leases Another try I did was to set the static IP address menuentry 'r5aw3' --class fedora --class gnu-linux --class gnu --class os { linuxefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.3/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=100.82.42.96::100.82.42.65:255.255.255.192:r5aw3.oss.labs:bond0:none bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90 initrdefi rhcos/4.3/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img } Jul 27 10:50:25 r5acsah dhcpd: DHCPREQUEST for 100.82.42.96 from 98:03:9b:58:ad:62 via bond0 Jul 27 10:50:25 r5acsah dhcpd: DHCPACK on 100.82.42.96 to 98:03:9b:58:ad:62 via bond0 what I would like to know is, is this a normal behavior. And is there any thing else you want me to test in RHCOS.
Hey Umesh, I'm not sure if there is a problem with what you are doing. I'm a bit confused why a DHCPREQUEST would be made when you're using static networking. Let's start simple. If you don't configure a bond (i.e. just use `ip=dhcp`) and just provide a simple Ignition config like: ``` { "ignition": { "version": "2.2.0" }, "passwd": { "users": [ { "groups": [ "sudo" ], "name": "core", "sshAuthorizedKeys": [ "ssh-rsa AAAA..." ] } ] } } ``` Does the node install and come up and you're able to login (ssh core@host)? NOTE: You'll need to replace `ssh-rsa AAAA...` with your ssh pubkey.
This is being worked on, but is currently awaiting more investigation or more information and won't be completed this sprint.
Hi Mabe, If I try out the scenario that you have mentioned, things work fine with ip=dhcp. Problem is when we are using a bonding (bond0) with two slave interfaces (eno1 and eno2) in active and active mode. Then when eno1 is shutdown, then dhcp is not provisioning any ip address because the dhcpd.conf is having info about the MAC address of eno1. I want to understand if this is a known issue or the behavior is not flawed and this is the way it is supposed to work.
(In reply to umesh_sunnapu from comment #22) > Hi Mabe, If I try out the scenario that you have mentioned, things work fine > with ip=dhcp. Problem is when we are using a bonding (bond0) with two slave > interfaces (eno1 and eno2) in active and active mode. Then when eno1 is > shutdown, then dhcp is not provisioning any ip address because the > dhcpd.conf is having info about the MAC address of eno1. Thanks. Those details help me understand. > > I want to understand if this is a known issue or the behavior is not flawed > and this is the way it is supposed to work. I'll have to look into it.
I did some reading about bonding options in https://www.kernel.org/doc/Documentation/networking/bonding.txt. From that I honestly can't tell what the behavior is for 802.3ad (mode 4) with respect to mac address. We might have to find an expert that knows more. Does anyone know what the behavior is for 802.3ad bonding: - by default does it use the same MAC address for all outbound traffic? - when you lose a link on one of your NICs does it continue to use the same MAC for outbound traffic? Also Umesh, just in case, you may want to make sure your switches are set up correctly. The docs mention: ``` Prerequisites: 1. Ethtool support in the base drivers for retrieving the speed and duplex of each slave. 2. A switch that supports IEEE 802.3ad Dynamic link aggregation. Most switches will require some type of configuration to enable 802.3ad mode. ```
In the case that the same MAC isn't used you might want to configure your DHCP server to hand out the same IP to multiple MAC addresses. See "Configuring a Multihomed DHCP Server": https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_a_multihomed_dhcp_server
(In reply to Dusty Mabe from comment #24) > I did some reading about bonding options in > https://www.kernel.org/doc/Documentation/networking/bonding.txt. From that I > honestly can't tell what the behavior is for 802.3ad (mode 4) with respect > to mac address. We might have to find an expert that knows more. > > Does anyone know what the behavior is for 802.3ad bonding: > > - by default does it use the same MAC address for all outbound traffic? Yes. If you cat /proc/net/bonding/bondX with root privs, you'll see additional data that isn't present when cat'd as a normal user, including the lacp "System MAC address:" (output gated by having capable(CAP_NET_ADMIN)). The "System MAC address" refers to the MAC assigned to the LACP link as a whole. > - when you lose a link on one of your NICs does it continue to use the same > MAC for outbound traffic? Yes. > Also Umesh, just in case, you may want to make sure your switches are set up > correctly. The docs mention: > > ``` > > Prerequisites: > > 1. Ethtool support in the base drivers for retrieving > the speed and duplex of each slave. > > 2. A switch that supports IEEE 802.3ad Dynamic link > aggregation. > > Most switches will require some type of configuration > to enable 802.3ad mode. > ``` You 100% must have a LACP-capable switch with 802.3ad mode enabled. But I don't think that's the problem here. I think all that is necessary is to specify the desired MAC for the bond, rather than relying on roulette, where whatever device wins the race to get enslaved first is the MAC the LACP as a whole gets assigned.
Hi Mabe performed a PXE install of RHCOS using the pxe menu with following entry menuentry 'r5aw1' --class fedora --class gnu-linux --class gnu --class os { linuxefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.5/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=bond0:dhcp bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90 initrdefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img } Mac address used in the dhcpd.conf file [root@r5acsah uefi]# cat /etc/dhcp/dhcpd.conf | grep -i r5aw1 -A 2 host r5aw1{ hardware ethernet 98:03:9B:58:A9:22; fixed-address 100.82.42.94; Installation was successful. [root@r5aw1 network-scripts]# cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.5 [root@r5aw1 network-scripts]# ip a s | grep -i bond0 3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 4: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0 [root@r5aw1 network-scripts]# cat ifcfg-bond0 # Generated by dracut initrd NAME="bond0" DEVICE="bond0" ONBOOT=yes NETBOOT=yes UUID="408e745f-7ff0-43dd-96cd-0d5b8c3c0eb9" IPV6INIT=yes BOOTPROTO=dhcp BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4" NAME="bond0" TYPE=Bond DNS1="100.82.42.90" Assigned a static mac address to bond0. Used the same mac address that is used in the dhcpd.conf [root@r5aw1 network-scripts]# nmcli con modify bond0 802-3-ethernet.cloned-mac-address 98:03:9b:58:a9:22 [root@r5aw1 network-scripts]# cat ifcfg-bond0 # Generated by dracut initrd DEVICE=bond0 ONBOOT=yes NETBOOT=yes UUID=408e745f-7ff0-43dd-96cd-0d5b8c3c0eb9 IPV6INIT=yes BOOTPROTO=dhcp BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4" NAME=bond0 TYPE=Bond DNS1=100.82.42.90 BONDING_MASTER=yes HWADDR= MACADDR=98:03:9B:58:A9:22 PROXY_METHOD=none BROWSER_ONLY=no DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no Performed a shutdown of port eno1 in the switch. eno1 mac address is what is referenced in dhcpd.conf R5-S5232F-1(conf-if-eth1/1/2:1)# show configuration ! interface ethernet1/1/2:1 shutdown channel-group 5 mode active no switchport mtu 9216 flowcontrol receive on flowcontrol transmit on lacp port-priority 1 port eno1 is down [root@r5aw1 network-scripts]# ip a s | grep -i bond0 3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 4: eno1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0 rebooted the node IP address is working [core@r5aw1 ~]$ ip a s | grep -E 'eno1|bond0' 3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 4: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0 now, did a no shutdown on the port eno1 at switch side and noticed eno1 is now acting as a slave to bond0 [core@r5aw1 ~]$ ip a s | grep -i bond0 3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 4: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 100.82.42.94/26 brd 100.82.42.127 scope global dynamic noprefixroute bond0 So things seems to be working fine with active-active interfaces and ensuring bond0 is set to the mac address that is being used in dhcpd.conf what I would like to understand is, if this is a supported approach by RedHat and how to set the mac address in the kernel params for RHCOS installations successfully.
(In reply to umesh_sunnapu from comment #27) ... > So things seems to be working fine with active-active interfaces and > ensuring bond0 is set to the mac address that is being used in dhcpd.conf Good, that's what I'd expect. > what I would like to understand is, if this is a supported approach by > RedHat and how to set the mac address in the kernel params for RHCOS > installations successfully. That one I have no idea about, the bonding driver is my realm, I don't really know RHCOS and the installer/bootloader used to get it up and running, but it sure seems like something we ought to have an option for. Worst-case, I suspect you could just populate your dhcpd.conf with entries for each of the MACs in the bond, so regardless of which one wins the init race, you still get the same IP. :)
(In reply to Jarod Wilson from comment #28) > (In reply to umesh_sunnapu from comment #27) > ... > > So things seems to be working fine with active-active interfaces and > > ensuring bond0 is set to the mac address that is being used in dhcpd.conf > > Good, that's what I'd expect. > > > what I would like to understand is, if this is a supported approach by > > RedHat and how to set the mac address in the kernel params for RHCOS > > installations successfully. > > That one I have no idea about, the bonding driver is my realm, I don't > really know RHCOS and the installer/bootloader used to get it up and > running, but it sure seems like something we ought to have an option for. > > Worst-case, I suspect you could just populate your dhcpd.conf with entries > for each of the MACs in the bond, so regardless of which one wins the init > race, you still get the same IP. :) I am not sure how this will work because now you have set multiple mac addresses with same IP. Dont want them to end up assigning the same IP. Something we need to test more. I am guessing there should be a way to set the mac address for the bond interface through kernel params. Hope @Dusty Mabe has an answer for that :)
(In reply to umesh_sunnapu from comment #29) > (In reply to Jarod Wilson from comment #28) > > (In reply to umesh_sunnapu from comment #27) > > ... > > > So things seems to be working fine with active-active interfaces and > > > ensuring bond0 is set to the mac address that is being used in dhcpd.conf > > > > Good, that's what I'd expect. > > > > > what I would like to understand is, if this is a supported approach by > > > RedHat and how to set the mac address in the kernel params for RHCOS > > > installations successfully. > > > > That one I have no idea about, the bonding driver is my realm, I don't > > really know RHCOS and the installer/bootloader used to get it up and > > running, but it sure seems like something we ought to have an option for. RHCOS "is" RHEL, so we should support the same bonding options. Yes it's supported. > > > > Worst-case, I suspect you could just populate your dhcpd.conf with entries > > for each of the MACs in the bond, so regardless of which one wins the init > > race, you still get the same IP. :) > > I am not sure how this will work because now you have set multiple mac > addresses with same IP. Dont want them to end up assigning the same IP. On the DHCP server side you can do this: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_a_multihomed_dhcp_server But you won't need to do that if we can get the mac address set for you so let's concentrate on that approach for now. > Something we need to test more. I am guessing there should be a way to set > the mac address for the bond interface through kernel params. Hope @Dusty > Mabe has an answer for that :) Yep I'll look into if this is possible using `ip=` kargs.
(In reply to Dusty Mabe from comment #30) > (In reply to umesh_sunnapu from comment #29) > > > Something we need to test more. I am guessing there should be a way to set > > the mac address for the bond interface through kernel params. Hope @Dusty > > Mabe has an answer for that :) > > Yep I'll look into if this is possible using `ip=` kargs. You should be able to do this like: ``` ip=bond0:dhcp::98:03:9B:58:A9:22 bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 ``` Please let me know if this works for you and we can close this out.
@Dusty Mabe, It does not look like things worked. Below is the bond network config file and as you can see, the mac address is not setup. [root@r5aw1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0 # Generated by dracut initrd NAME="bond0" DEVICE="bond0" ONBOOT=yes NETBOOT=yes UUID="44194d7b-59b6-42ab-a437-565b8231f8ee" IPV6INIT=yes BOOTPROTO=dhcp BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4" NAME="bond0" TYPE=Bond DNS1="100.82.42.90" I tried shutting down the port at switch level, rebooted the node and I can see that the mac address was pointing to secondary active slave nic port instead of what we tried to assign bond manually. Below is what I have in the grub.cfg file menuentry 'r5aw1' --class fedora --class gnu-linux --class gnu --class os { linuxefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://100.82.42.90:8080/rhcos/4.5/rhcos-4.5.2-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://100.82.42.90:8080/ignition/worker.ign ip=bond0:dhcp::98:03:9B:58:A9:22 bond=bond0:eno1,ens2f1:lacp_rate=1,miimon=100,mode=802.3ad,xmit_hash_policy=layer3+4 nameserver=100.82.42.90 initrdefi rhcos/4.5/rhcos-4.5.2-x86_64-installer-initramfs.x86_64.img }
(In reply to umesh_sunnapu from comment #32) > @Dusty Mabe, It does not look like things worked. Below is the bond network > config file and as you can see, the mac address is not setup. > > [root@r5aw1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0 > # Generated by dracut initrd > NAME="bond0" > DEVICE="bond0" > ONBOOT=yes > NETBOOT=yes > UUID="44194d7b-59b6-42ab-a437-565b8231f8ee" > IPV6INIT=yes > BOOTPROTO=dhcp > BONDING_OPTS="lacp_rate=1 miimon=100 mode=802.3ad xmit_hash_policy=layer3+4" > NAME="bond0" > TYPE=Bond > DNS1="100.82.42.90" Fun. OK, this worked when I tested it, but I tested it against RHCOS 4.6. In 4.6 we switched our initramfs networking (including the thing that parses the arguments) over to NetworkManager. In 4.5 we still use the legacy initscripts stack in the initramfs. So what you're hitting now is a bug in the legacy implementation. I just opened https://bugzilla.redhat.com/show_bug.cgi?id=1866851 for that, but it won't matter much (for RHCOS) because it's already "fixed" in 4.6 with the move to NM. For now you probably either need to provide files via ignition or run `nmcli` once on first boot to add the MAC address. See https://dustymabe.com/2019/11/06/running-a-script-on-bootup-via-ignition/
Thanks Mabe for the info. If that is the case, I will go ahead and put this bug in the guides going forward for OCP 4.5 and mention the need to use static network ip assignment instead of using DHCP. Thats the work around I can think of. And for OCP 4.6, we can go with the approach of setting the bond with a specific mac address so that customers can either go with static or dhcp based ip assignment for bond interface. Thoughts ?
You can use static networking. Or you can just not provide networking information via ip= kernel arguments and provide it via ignition instead. During the initramfs it will default to DHCP so you'll still be able to retrieve the config during the initramfs with the defaults (dhcp by default) and then have the appropriate configs applied via files. The files will cause NetworkManager to bring up the networking with bond and all in the real root. So it should be safe.
Agree Mabe. But creating ignition config files and applying them after the RHCOS worker node is installed, is multiple steps. I can point them to you process but again it should be more like, creating a machine config and ensure that is applied to the machine config pools where the worker node belongs to. And if the worker nodes in the machine config pool has a combination of RHEL and RHCOS worker nodes, then it becomes a bit more tricky to ensure only RHCOS worker nodes get this ignition config file but for RHEL worker nodes, it is a different process.
Yes. Just making sure you know all of the options that are available. Looking forward to this being fixed with the move to NetworkManager in 4.6. I'll go ahead and mark this as fixed in 4.6 and we'll have someone from the team confirm the behavior I describe.
Cool. Please go ahead with your next steps. thanks Mabe for all your support.
Active-Active bonds will by default use the MAC address of the first enslaved NIC for the bond and all interfaces that belong to the bond. If the first enslaved NIC loses link then the bond will continue to use its MAC address, until a reboot. On reboot the bond will come up and use the first enslaved NIC's MAC address, but in this case it won't be the same as the first boot of the machine because that NIC no longer has link. Thus, the MAC address of the bond has changed. In order to keep the MAC address of the bond the same we can explicitly set the MAC address of the bond, which will apply to the bond interface and all enslaved NIC interfaces. This is supported today via `ip=` kernel arguments, but there is a bug [1] that causes the MAC address to not get applied to the configuration when you are doing this with a DHCP configuration (i.e., `ip=bond0:dhcp::aa:bb:cc:dd:ff:11`). The bug does not apply to 4.6 because we are using NetworkManager in the initramfs in 4.6, which does not use the same tools to create the networking configuration. I will mark this as closed in 4.6. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1866851
>Active-Active bonds will by default use the MAC address of the first enslaved NIC for the >bond and all interfaces that belong to the bond. If the first enslaved NIC loses link then >the bond will continue to use its MAC address, until a reboot. On reboot the bond will come >up and use the first enslaved NIC's MAC address, but in this case it won't be the same >as the first boot of the machine because that NIC no longer has link. Thus, the MAC address >of the bond has changed. I have checked it and confirmed this behavior in my tests via libvirt with RHCOS 4.6. - Using the Kernel parameters: ip=bond0:dhcp - On the first boot I have all MACs as following: [core@localhost ~]$ ip a | grep link link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 link/ether 52:54:00:d8:53:8b brd ff:ff:ff:ff:ff:ff link/ether 52:54:00:d8:53:8b brd ff:ff:ff:ff:ff:ff ink/ether 52:54:00:d8:53:8b brd ff:ff:ff:ff:ff:ff - Turned NIC down, all MACs still the same. You can use: virsh domif-setlink coreos-nettest 52:54:00:d8:53:8b down or even: ip link set vnet0 - Reboot the server - As Dusty explained, when the server came up it updated the bond MAC with the first enslaved NIC MAC address available, the second NIC in the bond in this case: ens2: 52:54:00:d8:53:8b ens3: 52:54:00:dc:29:0e bond0: 52:54:00:dc:29:0e >In order to keep the MAC address of the bond the same we can explicitly set the MAC address >of the bond, which will apply to the bond interface and all enslaved NIC interfaces. >This is supported today via `ip=` kernel arguments, but there is a bug [1] that causes the MAC address >to not get applied to the configuration when you are doing this with a DHCP configuration (i.e., `ip=bond0:dhcp::aa:bb:cc:dd:ff:11`). The bug does not apply to 4.6 because we are using NetworkManager >in the initramfs in 4.6, which does not use the same tools to create the networking configuration. - Using the Kernel parameters: ip=bond0:dhcp::52:54:00:00:a6:00 - NetworkManager updated it: NetworkManager/system-connections/bond0.nmconnection:cloned-mac-address=52:54:00:00:A6:00 - On the first boot I have all MACs as following: [core@localhost ~]$ ip a | grep link link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 link/ether 52:54:00:00:a6:00 brd ff:ff:ff:ff:ff:ff link/ether 52:54:00:00:a6:00 brd ff:ff:ff:ff:ff:ff ink/ether 52:54:00:00:a6:00 brd ff:ff:ff:ff:ff:ff - Turned NIC down, all MACs still the same (virsh domif-setlink coreos-nettest 52:54:00:00:a6:00 down). - Reboot the server - The ens2 was updated in order to reflect the first enslaved NIC MAC address. However, ens3 and bond0 still the same. ens2: 52:54:00:ce:ae:92 ens3: 52:54:00:00:a6:00 bond0: 52:54:00:00:a6:00
Thanks Renata. Moving to verified.
>I have checked it and confirmed this behavior in my tests via libvirt with RHCOS 4.6. Just to clarify, the full version I used on the tests is rhcos-46.82.202009150240-0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196