Bug 1884628
| Summary: | ovs-configuration service fails when the external network is configured on a tagged vlan on top of a bond device on a baremetal IPI deployment | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Marius Cornea <mcornea> | ||||||
| Component: | Networking | Assignee: | Tim Rozet <trozet> | ||||||
| Networking sub component: | ovn-kubernetes | QA Contact: | Marius Cornea <mcornea> | ||||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||||
| Severity: | urgent | ||||||||
| Priority: | unspecified | CC: | aconstan, agomezpr, anbhat, asadawar, djuran, ferossi, mleonard, trozet, yjoseph | ||||||
| Version: | 4.6 | ||||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 4.7.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Cause:
Ethernet type interface was always being used when attaching the physical NIC to OVN-Kubernetes managed OVS bridge br-ex in NetworkManager.
Consequence:
Deploying OVN-Kubernetes on nodes that used bond or VLAN interfaces would fail.
Fix:
Correctly detect during deployment if the default gateway interface is a VLAN or bond interface
Result:
VLAN and bond interfaces are now correctly migrated over to the br-ex OVS bridge during OVN-Kubernetes deployments. The ovs-configuration.service systemd service will complete successfully during node deployment.
|
Story Points: | --- | ||||||
| Clone Of: | |||||||||
| : | 1884647 1958363 (view as bug list) | Environment: | |||||||
| Last Closed: | 2021-02-24 15:22:23 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1884647, 1958363 | ||||||||
| Attachments: |
|
||||||||
This looks to be the same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1887545 Only difference is this is a bond vlan subinterface. Will hold off on duping it until Marius confirms the fix for 1887545 also fixes this. After looking at Marius setup (thanks again Marius for setting all this up) this bug is slightly different as we have to add the interface with type vlan and include the right tag and parent interface. I've pushed a fix that will fix it for both bonds and vlans, but will keep this bug separate as the use case is different: https://github.com/openshift/machine-config-operator/pull/2152 I guess this BZ can be closed as a dupe given #comment 3? The bot didn't update this bug and pushed it to the MODIFIED state, it instead did so for 1887545. So I am not sure what this bug tracks anymore. The fix was for both bugs since they were related. They are slightly different use cases though, so we need each bug to validate each test path. @mcornea Can you help us verifying this on IPI BM? I believe UPI is also affected by this issue... I am experiencing issues as well. It fails also for a simple active-standby bond with no vlans. During boot it works but then configure-ovs.sh fails.
Also where can i find the configure-ovs.sh script in github?
Nov 20 23:46:39 master3 systemd[1]: Starting Configures OVS with proper host networking configuration...
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + rpm -qa
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + grep -q openvswitch
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + touch /var/run/ovs-config-executed
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' OVNKubernetes == OVNKubernetes ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + NM_CONN_PATH=/etc/NetworkManager/system-connections
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + iface=
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + counter=0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' 0 -lt 12 ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ ip route show default
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ awk '{if ($4 == "dev") print $5; exit}'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + iface=bond0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + [[ -n bond0 ]]
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + echo 'IPv4 Default gateway interface found: bond0'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: IPv4 Default gateway interface found: bond0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + break
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' bond0 = br-ex ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' -z bond0 ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + iface_mac=52:51:16:b0:00:a8
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + echo 'MAC address found for iface: bond0: 52:51:16:b0:00:a8'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: MAC address found for iface: bond0: 52:51:16:b0:00:a8
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ ip link show bond0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ awk '{print $5; exit}'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + iface_mtu=1500
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + [[ -z 1500 ]]
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + echo 'MTU found for iface: bond0: 1500'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: MTU found for iface: bond0: 1500
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli connection show br-ex
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli c add type ovs-bridge conn.interface br-ex con-name br-ex 802-3-ethernet.mtu 15>
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Connection 'br-ex' (6256497d-4ff9-44e4-af30-fb08c5085969) successfully added.
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ nmcli --fields UUID,DEVICE conn show --active
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ grep bond0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ awk '{print $1}'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + old_conn=effa3926-6fb1-44b2-9cbd-95cf5a064d49
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli connection show ovs-port-phys0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli c add type ovs-port conn.interface bond0 master br-ex con-name ovs-port-phys0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Connection 'ovs-port-phys0' (217c9ca6-d0e5-453d-a5a9-91d6811936da) successfully added.
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli connection show ovs-port-br-ex
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli c add type ovs-port conn.interface br-ex master br-ex con-name ovs-port-br-ex
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Connection 'ovs-port-br-ex' (1e525f9b-8dee-490c-accd-bb5d25d1aec3) successfully added.
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + extra_phys_args=
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ nmcli --get-values connection.type conn show effa3926-6fb1-44b2-9cbd-95cf5a064d49
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' bond == vlan ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: ++ nmcli --get-values connection.type conn show effa3926-6fb1-44b2-9cbd-95cf5a064d49
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' bond == bond ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + iface_type=bond
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli device disconnect bond0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Device 'bond0' successfully disconnected.
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli connection show ovs-if-phys0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli c add type bond conn.interface bond0 master ovs-port-phys0 con-name ovs-if-phys>
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Connection 'ovs-if-phys0' (03c38bd3-bbcc-47de-85c9-a5e8b2db103d) successfully added.
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli conn up ovs-if-phys0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/>
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli connection show ovs-if-br-ex
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli --fields ipv4.method,ipv6.method conn show effa3926-6fb1-44b2-9cbd-95cf5a064d49
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + grep manual
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + nmcli c add type ovs-interface slave-type ovs-port conn.interface br-ex master ovs-po>
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: Connection 'ovs-if-br-ex' (273dbadf-9eb7-442d-ac35-d7fe3acc3160) successfully added.
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + counter=0
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + '[' 0 -lt 5 ']'
Nov 20 23:46:39 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:46:44 master3 configure-ovs.sh[3498]: + nmcli --fields GENERAL.STATE conn show ovs-if-br-ex
Nov 20 23:46:44 master3 configure-ovs.sh[3498]: + grep -i activated
Nov 20 23:46:44 master3 configure-ovs.sh[3498]: + counter=1
Nov 20 23:46:44 master3 configure-ovs.sh[3498]: + '[' 1 -lt 5 ']'
Nov 20 23:46:44 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:46:49 master3 configure-ovs.sh[3498]: + nmcli --fields GENERAL.STATE conn show ovs-if-br-ex
Nov 20 23:46:49 master3 configure-ovs.sh[3498]: + grep -i activated
Nov 20 23:46:49 master3 configure-ovs.sh[3498]: + counter=2
Nov 20 23:46:49 master3 configure-ovs.sh[3498]: + '[' 2 -lt 5 ']'
Nov 20 23:46:49 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:46:54 master3 configure-ovs.sh[3498]: + nmcli --fields GENERAL.STATE conn show ovs-if-br-ex
Nov 20 23:46:54 master3 configure-ovs.sh[3498]: + grep -i activated
Nov 20 23:46:54 master3 configure-ovs.sh[3498]: + counter=3
Nov 20 23:46:54 master3 configure-ovs.sh[3498]: + '[' 3 -lt 5 ']'
Nov 20 23:46:54 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:46:59 master3 configure-ovs.sh[3498]: + nmcli --fields GENERAL.STATE conn show ovs-if-br-ex
Nov 20 23:46:59 master3 configure-ovs.sh[3498]: + grep -i activated
Nov 20 23:46:59 master3 configure-ovs.sh[3498]: + counter=4
Nov 20 23:46:59 master3 configure-ovs.sh[3498]: + '[' 4 -lt 5 ']'
Nov 20 23:46:59 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + nmcli --fields GENERAL.STATE conn show ovs-if-br-ex
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + grep -i activated
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + counter=5
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + '[' 5 -lt 5 ']'
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + echo 'WARN: OVS did not succesfully activate NM connection. Attempting to bring up co>
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: WARN: OVS did not succesfully activate NM connection. Attempting to bring up connections
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + counter=0
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + '[' 0 -lt 5 ']'
Nov 20 23:47:04 master3 configure-ovs.sh[3498]: + nmcli conn up ovs-if-br-ex
Nov 20 23:47:50 master3 configure-ovs.sh[3498]: Error: Connection activation failed: IP configuration could not be reserved (no availab>
Nov 20 23:47:50 master3 configure-ovs.sh[3498]: Hint: use 'journalctl -xe NM_CONNECTION=273dbadf-9eb7-442d-ac35-d7fe3acc3160 + NM_DEVIC>
Nov 20 23:47:50 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:47:55 master3 configure-ovs.sh[3498]: + counter=1
Nov 20 23:47:55 master3 configure-ovs.sh[3498]: + '[' 1 -lt 5 ']'
Nov 20 23:47:55 master3 configure-ovs.sh[3498]: + nmcli conn up ovs-if-br-ex
Nov 20 23:48:40 master3 configure-ovs.sh[3498]: Error: Connection activation failed: IP configuration could not be reserved (no availab>
Nov 20 23:48:40 master3 configure-ovs.sh[3498]: Hint: use 'journalctl -xe NM_CONNECTION=273dbadf-9eb7-442d-ac35-d7fe3acc3160 + NM_DEVIC>
Nov 20 23:48:40 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:48:45 master3 configure-ovs.sh[3498]: + counter=2
Nov 20 23:48:45 master3 configure-ovs.sh[3498]: + '[' 2 -lt 5 ']'
Nov 20 23:48:45 master3 configure-ovs.sh[3498]: + nmcli conn up ovs-if-br-ex
Nov 20 23:49:30 master3 configure-ovs.sh[3498]: Error: Connection activation failed: IP configuration could not be reserved (no availab>
Nov 20 23:49:30 master3 configure-ovs.sh[3498]: Hint: use 'journalctl -xe NM_CONNECTION=273dbadf-9eb7-442d-ac35-d7fe3acc3160 + NM_DEVIC>
Nov 20 23:49:30 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:49:35 master3 configure-ovs.sh[3498]: + counter=3
Nov 20 23:49:35 master3 configure-ovs.sh[3498]: + '[' 3 -lt 5 ']'
Nov 20 23:49:35 master3 configure-ovs.sh[3498]: + nmcli conn up ovs-if-br-ex
Nov 20 23:50:20 master3 configure-ovs.sh[3498]: Error: Connection activation failed: IP configuration could not be reserved (no availab>
Nov 20 23:50:20 master3 configure-ovs.sh[3498]: Hint: use 'journalctl -xe NM_CONNECTION=273dbadf-9eb7-442d-ac35-d7fe3acc3160 + NM_DEVIC>
Nov 20 23:50:20 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:50:25 master3 configure-ovs.sh[3498]: + counter=4
Nov 20 23:50:25 master3 configure-ovs.sh[3498]: + '[' 4 -lt 5 ']'
Nov 20 23:50:25 master3 configure-ovs.sh[3498]: + nmcli conn up ovs-if-br-ex
Nov 20 23:51:10 master3 configure-ovs.sh[3498]: Error: Connection activation failed: IP configuration could not be reserved (no availab>
Nov 20 23:51:10 master3 configure-ovs.sh[3498]: Hint: use 'journalctl -xe NM_CONNECTION=273dbadf-9eb7-442d-ac35-d7fe3acc3160 + NM_DEVIC>
Nov 20 23:51:10 master3 configure-ovs.sh[3498]: + sleep 5
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + counter=5
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + '[' 5 -lt 5 ']'
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + echo 'ERROR: Failed to activate ovs-if-br-ex NM connection'
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: ERROR: Failed to activate ovs-if-br-ex NM connection
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + set +e
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + nmcli conn down ovs-if-br-ex
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: Error: 'ovs-if-br-ex' is not an active connection.
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: Error: no active connection provided.
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + nmcli conn down ovs-if-phys0
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: Connection 'ovs-if-phys0' successfully deactivated (D-Bus active path: /org/freedesktop>
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + nmcli conn up effa3926-6fb1-44b2-9cbd-95cf5a064d49
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/>
Nov 20 23:51:15 master3 configure-ovs.sh[3498]: + exit 1
Nov 20 23:51:15 master3 systemd[1]: ovs-configuration.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 23:51:15 master3 systemd[1]: ovs-configuration.service: Failed with result 'exit-code'.
Nov 20 23:51:15 master3 systemd[1]: Failed to start Configures OVS with proper host networking configuration.
Nov 20 23:51:15 master3 systemd[1]: ovs-configuration.service: Consumed 533ms CPU time
Federico, here: https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml I'm guessing you are probably hitting this bz where we do not consider bond options: https://bugzilla.redhat.com/show_bug.cgi?id=1899350 Created attachment 1735682 [details]
log for the failure of the ovs-configure.sh script for 02814494
log for the failure of the ovs-configure.sh script for 02814494
Created attachment 1735684 [details]
output for *ip -a* to go along w the configuration failure log for 02814494
output for *ip -a* to go along w the configuration failure log for 02814494
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 *** Bug 1958363 has been marked as a duplicate of this bug. *** |
Description of problem: ovs-configuration service fails when the external network is configured on a tagged vlan on top of a bond device on a baremetal IPI deployment: journalctl -l -u ovs-configuration -- Logs begin at Fri 2020-10-02 13:15:24 UTC, end at Fri 2020-10-02 13:43:56 UTC. -- Oct 02 13:20:57 master-0-1 systemd[1]: Starting Configures OVS with proper host networking configuration... Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + '[' OVNKubernetes == OVNKubernetes ']' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + NM_CONN_PATH=/etc/NetworkManager/system-connections Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + iface= Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + counter=0 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + '[' 0 -lt 12 ']' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ ip route show default Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ awk '{if ($4 == "dev") print $5; exit}' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + iface=bond0.404 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + [[ -n bond0.404 ]] Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + echo 'IPv4 Default gateway interface found: bond0.404' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: IPv4 Default gateway interface found: bond0.404 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + break Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + '[' bond0.404 = br-ex ']' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + '[' -z bond0.404 ']' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + iface_mac=52:54:00:fb:a1:55 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + echo 'MAC address found for iface: bond0.404: 52:54:00:fb:a1:55' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: MAC address found for iface: bond0.404: 52:54:00:fb:a1:55 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ ip link show bond0.404 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ awk '{print $5; exit}' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + iface_mtu=1500 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + [[ -z 1500 ]] Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + echo 'MTU found for iface: bond0.404: 1500' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: MTU found for iface: bond0.404: 1500 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli connection show br-ex Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli c add type ovs-bridge conn.interface br-ex con-name br-ex 802-3-ethernet.mtu 1500 802-3-ethernet.cloned-mac-address 52:54:00:fb:a1:55 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: Connection 'br-ex' (fce331ae-8df1-4302-8faa-95b130659313) successfully added. Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ nmcli --fields UUID,DEVICE conn show --active Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ grep bond0.404 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: ++ awk '{print $1}' Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + old_conn=bfd1bafe-f5f5-85b3-93ad-d99629a85abe Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli connection show ovs-port-phys0 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli c add type ovs-port conn.interface bond0.404 master br-ex con-name ovs-port-phys0 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: Connection 'ovs-port-phys0' (774644fc-656f-45bb-853f-cdd446d1667f) successfully added. Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli connection show ovs-port-br-ex Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli c add type ovs-port conn.interface br-ex master br-ex con-name ovs-port-br-ex Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: Connection 'ovs-port-br-ex' (890d19af-57a6-4a84-be78-0aa6c84f50a0) successfully added. Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: + nmcli device disconnect bond0.404 Oct 02 13:20:57 master-0-1 configure-ovs.sh[1591]: Device 'bond0.404' successfully disconnected. Oct 02 13:20:57 localhost.localdomain configure-ovs.sh[1591]: + nmcli connection show ovs-if-phys0 Oct 02 13:20:57 localhost.localdomain configure-ovs.sh[1591]: + nmcli c add type 802-3-ethernet conn.interface bond0.404 master ovs-port-phys0 con-name ovs-if-phys0 connection.autoconnect-priority 100 802-3-ethernet.mtu 1500 Oct 02 13:20:57 localhost.localdomain configure-ovs.sh[1591]: Connection 'ovs-if-phys0' (f79e6a01-64ce-4141-97eb-dc4792cb0773) successfully added. Oct 02 13:20:57 localhost.localdomain configure-ovs.sh[1591]: + nmcli conn up ovs-if-phys0 Oct 02 13:20:57 localhost.localdomain configure-ovs.sh[1591]: Error: Connection activation failed: No suitable device found for this connection (device enp4s0 not available because profile is not compatible with device (mismatching interf> Oct 02 13:20:57 localhost.localdomain systemd[1]: ovs-configuration.service: Main process exited, code=exited, status=4/NOPERMISSION Oct 02 13:20:57 localhost.localdomain systemd[1]: ovs-configuration.service: Failed with result 'exit-code'. Oct 02 13:20:57 localhost.localdomain systemd[1]: Failed to start Configures OVS with proper host networking configuration. Oct 02 13:20:57 localhost.localdomain systemd[1]: ovs-configuration.service: Consumed 313ms CPU time Version-Release number of selected component (if applicable): 4.6.0-0.nightly-2020-10-02-065738: How reproducible: 100% Steps to Reproduce: Setup: openshift nodes have 2 NICs grouped in a bond. Untagged traffic goes to provisioning network and baremetal network traffic goes over a tagged vlan configured on top of the bond. The network interfaces configuration is provided via ignition files. 1. Deploy setup via baremetal IPI flow 2. Workaround BZ#1824331 by following https://docs.google.com/document/d/115lKEulLTmTO4taQ7w1y9y3JjD6wRuDx9qJqSk1rLyE/edit# 3. Provide NIC configs as ifcfg network-scripts via Ignition, this is the master.ign: { "ignition": { "version": "3.1.0", "config": { "merge": [ { "source": "https://192.168.123.5:22623/config/master" } ] }, "security": { "tls": { "certificateAuthorities": [ { "source": "data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lJWGZGc0J0YUt5all3RFFZSktvWklodmNOQVFFTEJRQXdKakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1SQXdEZ1lEVlFRREV3ZHliMjkwTFdOaE1CNFhEVEl3TVRBd01qRXlORGt6TWxvWApEVE13TURrek1ERXlORGt6TWxvd0pqRVNNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUkF3RGdZRFZRUURFd2R5CmIyOTBMV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUE2Z3VvVkttcnNaV2cKck1RSUkzY3BhNmVCaHpkcnpYM0ttMm1wT2VxQ1pOeUpzRE0yMEZBblkvNlBaQmo3QXJXUHBVOExaTU1WWTk5UQpIV1hRTmRFMS9uZTVPL3ZVUVlOTzY4czJVa3d3bGFBUnNobG1IQkNxMkxyNkhIMU1UYzVOMVg2QVlLaTk5YmdUCmM3QWxXUm9leDJhNHcwbjdaZnFsb0RkbDRtNVZ0QkIvb1Ivb0h0VjN0eXRRNzI4ei9QTDVkUlpERXZwbVBEYXkKdTVxeGtaZmFabG5wOGg1R0xvNE9vOEpjMU03cjBKSEZ4Tmt0T2JPWEFuYlJkSzZ6UDMxWXc0bXJTSWVxNjBaQgpLOWJ1UzhZeWZUVS95TXNrenVsYTdtN0FMd0xyOTFVekRQQ3gvaHJpaWFrL0JkSjgyeXJUclp3eWJhNHFTWTF6CmhXWWQ1NXh3bndJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVXg1WkJ3b3c0ZXZrbSt1SGhXcHNrb3VYdU83WXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUh6VFpXWEVudkE0dDV4VnhzOU1TcytHVllrd0IwcjNNbEoyTFpDWXBPNHFHUXFsUnNUelp0WUZHeFFXCmJLbHlHNDJRV1ZuenUrVTF1TGxOYkR0WWN6TUJHQS9lajA2Y2tWMHFRTmVBQ3ZGMjJUdmc5QkdJRVpGZmMySnQKQUxLSDZFV2hyVEZxN0lxKzFxMnN5NHJSYlRYL093Qkl6R1RVQk9oVmtMZHhPUEV3ZUNFWDR1K3FjbnVmZVY5eQp4Y1N2My9aNWV5K2hQZ3BDRTVseWdaY3RURHdrVmdVbm5Vek1XUkZ4ZkEzUVVSL2dqTXFFVXFTeHd2YUwxSHA5CnhveTdQNHlEdWdSRkZRSnVaR0U0NUJzNm1LQmdCelpoTzdiZ1RmSGZ1TDBkRnJpOTZJaWNSdUV2S1hwMk9jTXQKUjNZQ2ZzTjZNVE03b0hWUnlIcTd6NWx4Tlk4PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==" } ] } } }, "storage": { "files": [ { "path": "/etc/sysconfig/network-scripts/ifcfg-enp4s0", "filesystem": "root", "mode": 436, "contents": { "source": "data:text/plain;charset=utf-8;base64,REVWSUNFPWVucDRzMApCT09UUFJPVE89bm9uZQpPTkJPT1Q9eWVzCk1BU1RFUj1ib25kMApTTEFWRT15ZXMK" } }, { "path": "/etc/sysconfig/network-scripts/ifcfg-enp5s0", "filesystem": "root", "mode": 436, "contents": { "source": "data:text/plain;charset=utf-8;base64,REVWSUNFPWVucDVzMApCT09UUFJPVE89bm9uZQpPTkJPT1Q9eWVzCk1BU1RFUj1ib25kMApTTEFWRT15ZXMK" } }, { "path": "/etc/sysconfig/network-scripts/ifcfg-bond0", "filesystem": "root", "mode": 436, "contents": { "source": "data:text/plain;charset=utf-8;base64,Qk9ORElOR19PUFRTPSJkb3duZGVsYXk9MCBsYWNwX3JhdGU9ZmFzdCBtaWltb249MTAwIG1vZGU9ODAyLjNhZCB1cGRlbGF5PTAiClRZUEU9Qm9uZApCT05ESU5HX01BU1RFUj15ZXMKQk9PVFBST1RPPWRoY3AKTkFNRT1ib25kMApERVZJQ0U9Ym9uZDAKT05CT09UPXllcwo=" } }, { "path": "/etc/sysconfig/network-scripts/ifcfg-bond0.404", "filesystem": "root", "mode": 436, "contents": { "source": "data:text/plain;charset=utf-8;base64,REVWSUNFPWJvbmQwLjQwNApCT09UUFJPVE89ZGhjcApPTkJPT1Q9eWVzClZMQU49eWVz" } } ] } } Actual results: When the master nodes boot they lose external connectivity because ovs-configuration service fails. Expected results: Nodes do not lose connectivity and deployment moves forward.