Created attachment 1798331[details]
ovs-configuration.journal
Description of problem:
Baremetal IPI deployment with control plane on bond device fails during bootstrap phase.
bond configuration is passed via machine config at deploy time by the following:
---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
creationTimestamp: null
labels:
machineconfiguration.openshift.io/role: master
name: 11-master-bonding
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWVuczRmMAp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczRmMAptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWU=
path: /etc/NetworkManager/system-connections/ens4f0.nmconnection
filesystem: root
mode: 0600
- contents:
source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWVuczRmMQp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczRmMQptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWU=
path: /etc/NetworkManager/system-connections/ens4f1.nmconnection
filesystem: root
mode: 0600
- contents:
source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWJvbmQwCnR5cGU9Ym9uZAppbnRlcmZhY2UtbmFtZT1ib25kMAphdXRvY29ubmVjdD10cnVlCmNvbm5lY3Rpb24uYXV0b2Nvbm5lY3Qtc2xhdmVzPTEKCltib25kXQptb2RlPTgwMi4zYWQKbWlpbW9uPTEwMAoKW2lwdjRdCm1ldGhvZD1hdXRvCmRoY3AtdGltZW91dD0yMTQ3NDgzNjQ3CgpbaXB2Nl0KbWV0aG9kPWRpc2FibGVkCg==
path: /etc/NetworkManager/system-connections//bond0.nmconnection
filesystem: root
mode: 0600
Looking on the master nodes we can see that the bond0 device includes only ens4f1 interface while ens4f0 interface is configured independently and holding the DHCP ip address(10.46.58.20/24). br-ex bridge includes bond0 device and ends up without an IPv4 address as it is set on the ens4f0.
[root@openshift-master-0 core]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether bc:97:e1:29:9c:80 brd ff:ff:ff:ff:ff:ff
3: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether bc:97:e1:29:9c:81 brd ff:ff:ff:ff:ff:ff
4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 1c:34:da:75:bc:fa brd ff:ff:ff:ff:ff:ff
inet 10.46.58.20/24 brd 10.46.58.255 scope global dynamic noprefixroute ens4f0
valid_lft 29820sec preferred_lft 29820sec
inet6 2620:52:0:2e3a:1e34:daff:fe75:bcfa/64 scope global dynamic noprefixroute
valid_lft 2591926sec preferred_lft 604726sec
inet6 fe80::1e34:daff:fe75:bcfa/64 scope link noprefixroute
valid_lft forever preferred_lft forever
5: ens4f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether 1c:34:da:75:bc:fb brd ff:ff:ff:ff:ff:ff
7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 1c:34:da:75:bc:fb brd ff:ff:ff:ff:ff:ff
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0e:5f:b5:bd:02:67 brd ff:ff:ff:ff:ff:ff
12: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 1c:34:da:75:bc:fa brd ff:ff:ff:ff:ff:ff
inet6 2620:52:0:2e3a:5af4:7e2:a2b0:a8a6/64 scope global dynamic noprefixroute
valid_lft 2591528sec preferred_lft 604328sec
inet6 fe80::ce05:109f:3bec:d287/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@openshift-master-0 core]# nmcli con
NAME UUID TYPE DEVICE
Wired Connection 74efc6ae-ae0b-46a2-b40c-7fc096b261ff ethernet ens4f0
ovs-if-br-ex 1460f265-cbc6-4435-8f96-8dc5d1608640 ovs-interface br-ex
br-ex 091be9cc-88c3-4617-a38f-c1c050a4b9ef ovs-bridge br-ex
ens4f1 9b808a8b-13ba-3749-8b8b-6f6f208666a2 ethernet ens4f1
ovs-if-phys0 fe52c005-e7e7-4b5c-9306-6884f8e79b4c bond bond0
ovs-port-br-ex c22be679-27bc-401a-a7da-fecd0776cd9e ovs-port br-ex
ovs-port-phys0 d1b534b2-afeb-42b4-8a89-6d01b96a789c ovs-port bond0
bond0 52eecf5a-df5e-30ae-9ca1-6297f0239027 bond --
ens4f0 8950883b-a416-360a-a597-cb308946aaa0 ethernet --
[root@openshift-master-0 core]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v4.18.0-305.7.1.el8_4.x86_64
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 1c:34:da:75:bc:fb
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 21
Partner Key: 3
Partner Mac Address: 50:c7:09:34:6a:a0
Slave Interface: ens4f1
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 1c:34:da:75:bc:fb
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 1c:34:da:75:bc:fb
port key: 21
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 127
system mac address: 50:c7:09:34:6a:a0
oper key: 3
port priority: 127
port number: 3
port state: 63
[root@openshift-master-0 core]# cat /etc/NetworkManager/system-connections/ens4f0.nmconnection
[connection]
id=ens4f0
type=ethernet
interface-name=ens4f0
master=bond0
slave-type=bond
autoconnect=true
[root@openshift-master-0 core]# cat /etc/NetworkManager/system-connections/ens4f1.nmconnection
[connection]
id=ens4f1
type=ethernet
interface-name=ens4f1
master=bond0
slave-type=bond
autoconnect=true
[root@openshift-master-0 core]# cat /etc/NetworkManager/system-connections/bond0.nmconnection
[connection]
id=bond0
type=bond
interface-name=bond0
autoconnect=true
connection.autoconnect-slaves=1
[bond]
mode=802.3ad
miimon=100
[ipv4]
method=auto
dhcp-timeout=2147483647
[ipv6]
method=disabled
Version-Release number of selected component (if applicable):
4.8.0-rc.3
How reproducible:
100%
Steps to Reproduce:
1. Deploy baremetal IPI environment with control plane running on a bond device
Actual results:
br-ex bridge on master nodes does not get an IPv4 address via DHCP and bootstrap fails
Expected results:
ens4f0 is part of the bond device according to the NetworkManager system-connections configuration and br-ex bridge gets the DHCP IPV4 address
Additional info:
Attaching NetworkManager ovs-configuration.service journal logs.
Created attachment 1798331 [details] ovs-configuration.journal Description of problem: Baremetal IPI deployment with control plane on bond device fails during bootstrap phase. bond configuration is passed via machine config at deploy time by the following: --- apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: creationTimestamp: null labels: machineconfiguration.openshift.io/role: master name: 11-master-bonding spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWVuczRmMAp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczRmMAptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWU= path: /etc/NetworkManager/system-connections/ens4f0.nmconnection filesystem: root mode: 0600 - contents: source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWVuczRmMQp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczRmMQptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWU= path: /etc/NetworkManager/system-connections/ens4f1.nmconnection filesystem: root mode: 0600 - contents: source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWJvbmQwCnR5cGU9Ym9uZAppbnRlcmZhY2UtbmFtZT1ib25kMAphdXRvY29ubmVjdD10cnVlCmNvbm5lY3Rpb24uYXV0b2Nvbm5lY3Qtc2xhdmVzPTEKCltib25kXQptb2RlPTgwMi4zYWQKbWlpbW9uPTEwMAoKW2lwdjRdCm1ldGhvZD1hdXRvCmRoY3AtdGltZW91dD0yMTQ3NDgzNjQ3CgpbaXB2Nl0KbWV0aG9kPWRpc2FibGVkCg== path: /etc/NetworkManager/system-connections//bond0.nmconnection filesystem: root mode: 0600 Looking on the master nodes we can see that the bond0 device includes only ens4f1 interface while ens4f0 interface is configured independently and holding the DHCP ip address(10.46.58.20/24). br-ex bridge includes bond0 device and ends up without an IPv4 address as it is set on the ens4f0. [root@openshift-master-0 core]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether bc:97:e1:29:9c:80 brd ff:ff:ff:ff:ff:ff 3: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether bc:97:e1:29:9c:81 brd ff:ff:ff:ff:ff:ff 4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 1c:34:da:75:bc:fa brd ff:ff:ff:ff:ff:ff inet 10.46.58.20/24 brd 10.46.58.255 scope global dynamic noprefixroute ens4f0 valid_lft 29820sec preferred_lft 29820sec inet6 2620:52:0:2e3a:1e34:daff:fe75:bcfa/64 scope global dynamic noprefixroute valid_lft 2591926sec preferred_lft 604726sec inet6 fe80::1e34:daff:fe75:bcfa/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: ens4f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether 1c:34:da:75:bc:fb brd ff:ff:ff:ff:ff:ff 7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000 link/ether 1c:34:da:75:bc:fb brd ff:ff:ff:ff:ff:ff 8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0e:5f:b5:bd:02:67 brd ff:ff:ff:ff:ff:ff 12: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 1c:34:da:75:bc:fa brd ff:ff:ff:ff:ff:ff inet6 2620:52:0:2e3a:5af4:7e2:a2b0:a8a6/64 scope global dynamic noprefixroute valid_lft 2591528sec preferred_lft 604328sec inet6 fe80::ce05:109f:3bec:d287/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@openshift-master-0 core]# nmcli con NAME UUID TYPE DEVICE Wired Connection 74efc6ae-ae0b-46a2-b40c-7fc096b261ff ethernet ens4f0 ovs-if-br-ex 1460f265-cbc6-4435-8f96-8dc5d1608640 ovs-interface br-ex br-ex 091be9cc-88c3-4617-a38f-c1c050a4b9ef ovs-bridge br-ex ens4f1 9b808a8b-13ba-3749-8b8b-6f6f208666a2 ethernet ens4f1 ovs-if-phys0 fe52c005-e7e7-4b5c-9306-6884f8e79b4c bond bond0 ovs-port-br-ex c22be679-27bc-401a-a7da-fecd0776cd9e ovs-port br-ex ovs-port-phys0 d1b534b2-afeb-42b4-8a89-6d01b96a789c ovs-port bond0 bond0 52eecf5a-df5e-30ae-9ca1-6297f0239027 bond -- ens4f0 8950883b-a416-360a-a597-cb308946aaa0 ethernet -- [root@openshift-master-0 core]# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v4.18.0-305.7.1.el8_4.x86_64 Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 802.3ad info LACP rate: slow Min links: 0 Aggregator selection policy (ad_select): stable System priority: 65535 System MAC address: 1c:34:da:75:bc:fb Active Aggregator Info: Aggregator ID: 1 Number of ports: 1 Actor Key: 21 Partner Key: 3 Partner Mac Address: 50:c7:09:34:6a:a0 Slave Interface: ens4f1 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 1c:34:da:75:bc:fb Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: none Actor Churned Count: 0 Partner Churned Count: 0 details actor lacp pdu: system priority: 65535 system mac address: 1c:34:da:75:bc:fb port key: 21 port priority: 255 port number: 1 port state: 61 details partner lacp pdu: system priority: 127 system mac address: 50:c7:09:34:6a:a0 oper key: 3 port priority: 127 port number: 3 port state: 63 [root@openshift-master-0 core]# cat /etc/NetworkManager/system-connections/ens4f0.nmconnection [connection] id=ens4f0 type=ethernet interface-name=ens4f0 master=bond0 slave-type=bond autoconnect=true [root@openshift-master-0 core]# cat /etc/NetworkManager/system-connections/ens4f1.nmconnection [connection] id=ens4f1 type=ethernet interface-name=ens4f1 master=bond0 slave-type=bond autoconnect=true [root@openshift-master-0 core]# cat /etc/NetworkManager/system-connections/bond0.nmconnection [connection] id=bond0 type=bond interface-name=bond0 autoconnect=true connection.autoconnect-slaves=1 [bond] mode=802.3ad miimon=100 [ipv4] method=auto dhcp-timeout=2147483647 [ipv6] method=disabled Version-Release number of selected component (if applicable): 4.8.0-rc.3 How reproducible: 100% Steps to Reproduce: 1. Deploy baremetal IPI environment with control plane running on a bond device Actual results: br-ex bridge on master nodes does not get an IPv4 address via DHCP and bootstrap fails Expected results: ens4f0 is part of the bond device according to the NetworkManager system-connections configuration and br-ex bridge gets the DHCP IPV4 address Additional info: Attaching NetworkManager ovs-configuration.service journal logs.