Bug 1927405

Summary: [4.7z] OVS configuration service unable to clone NetworkManager's connections in the overlay FS
Product: OpenShift Container Platform Reporter: Tim Rozet <trozet>
Component: Machine Config OperatorAssignee: Tim Rozet <trozet>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: anbhat, behoward, brad, mkrejci, mnguyen, odepaz, sponnaga, trozet, zzhao
Target Milestone: ---Flags: brad: needinfo-
Target Release: 4.7.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1927366
: 1932108 (view as bug list) Environment:
Last Closed: 2021-03-10 11:24:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1927366    
Bug Blocks: 1932108    

Description Tim Rozet 2021-02-10 16:42:41 UTC 
+++ This bug was initially created as a clone of Bug #1927366 +++

Description of problem:

When configuring a static IP network connection of type IPv6 in the manifest, the OVS configuration service fails to find the cloned interface.

So we have the following nmconnection which is being inserted via manifest/ignition:

# /etc/NetworkManager/conf.d/01-ipv6.conf
[connection]
ipv6.dhcp-iaid=mac
ipv6.dhcp-duid=ll
[keyfile]
path=/etc/NetworkManager/system-connections-merged

But the configure-ovs.sh script cannot find the cloned connection he made a few steps back, because it expects to see it in /etc/NetworkManager/system-connections/ dir:

Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: ++ nmcli --get-values connection.type conn show c564ff37-d7cd-394e-89c9-ecbbc0ab84d3
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + '[' 802-3-ethernet == bond ']'
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + iface_type=802-3-ethernet
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli device disconnect ens3
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: Device 'ens3' successfully disconnected.
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli connection show ovs-if-phys0
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli c add type 802-3-ethernet conn.interface ens3 master ovs-port-phys0 con-name ovs-if-phys0 connection.autoconnect-priority 100 802-3-ethernet.mtu 1500
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: Connection 'ovs-if-phys0' (7a87ffcb-ee06-4ad8-b87a-8b5dec866711) successfully added.
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli conn up ovs-if-phys0
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/6)
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli connection show ovs-if-br-ex
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli --fields ipv4.method,ipv6.method conn show c564ff37-d7cd-394e-89c9-ecbbc0ab84d3
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + grep manual
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: ipv6.method:                            manual
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + echo 'Static IP addressing detected on default gateway connection: c564ff37-d7cd-394e-89c9-ecbbc0ab84d3'
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: Static IP addressing detected on default gateway connection: c564ff37-d7cd-394e-89c9-ecbbc0ab84d3
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + egrep -l '--include=*.nmconnection' c564ff37-d7cd-394e-89c9-ecbbc0ab84d3 /etc/NetworkManager/system-connections/ens3.nmconnection /etc/NetworkManager/system-connections/ens4.nmconnection
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + echo 'WARN: unable to find NM configuration file for conn: c564ff37-d7cd-394e-89c9-ecbbc0ab84d3. Attempting to clone conn'
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: WARN: unable to find NM configuration file for conn: c564ff37-d7cd-394e-89c9-ecbbc0ab84d3. Attempting to clone conn
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + old_conn_file=/etc/NetworkManager/system-connections/c564ff37-d7cd-394e-89c9-ecbbc0ab84d3-clone.nmconnection
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + nmcli conn clone c564ff37-d7cd-394e-89c9-ecbbc0ab84d3 c564ff37-d7cd-394e-89c9-ecbbc0ab84d3-clone
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: ens3 (c564ff37-d7cd-394e-89c9-ecbbc0ab84d3) cloned as c564ff37-d7cd-394e-89c9-ecbbc0ab84d3-clone (8b3ab93e-9213-4611-9efa-1deb90567d5f).
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + cloned=true
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + '[' '!' -f /etc/NetworkManager/system-connections/c564ff37-d7cd-394e-89c9-ecbbc0ab84d3-clone.nmconnection ']'
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + echo 'ERROR: unable to locate cloned conn file: /etc/NetworkManager/system-connections/c564ff37-d7cd-394e-89c9-ecbbc0ab84d3-clone.nmconnection'
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: ERROR: unable to locate cloned conn file: /etc/NetworkManager/system-connections/c564ff37-d7cd-394e-89c9-ecbbc0ab84d3-clone.nmconnection
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 configure-ovs.sh[1362]: + exit 1
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 systemd[1]: ovs-configuration.service: Main process exited, code=exited, status=1/FAILURE
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 systemd[1]: ovs-configuration.service: Failed with result 'exit-code'.
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 systemd[1]: Failed to start Configures OVS with proper host networking configuration.
Feb 07 18:45:08 test-infra-cluster-assisted-installer-master-0 systemd[1]: ovs-configuration.service: Consumed 288ms CPU time

Version-Release number of selected component (if applicable):


How reproducible:
100%


Steps to Reproduce:
1. Create the nmconnection definition above
2. Run the service
3. Read the logs via journalctl

Actual results:
Service fails and the connection is not up.

Expected results:
Connection ens3 should be defined and ready to be used.


Additional info:
Comment 13 errata-xmlrpc 2021-03-10 11:24:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.1 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0678