Bug 2098099 - ovs-configuration.service fails with Error: Failed to modify connection 'ovs-if-br-ex': failed to update connection: error writing to file '/etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection'
Summary: ovs-configuration.service fails with Error: Failed to modify connection 'ovs-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.9
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.9.z
Assignee: Jaime Caamaño Ruiz
QA Contact: Ross Brattain
URL:
Whiteboard: UpdateRecommendationsBlocked
: 2097872 2103877 (view as bug list)
Depends On: 2098097
Blocks: 2094765
TreeView+ depends on / blocked
 
Reported: 2022-06-17 09:03 UTC by Jaime Caamaño Ruiz
Modified: 2022-07-28 09:04 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2098097
Environment:
Last Closed: 2022-06-30 05:31:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 3188 0 None open [release-4.9] Bug 2098099: configure-ovs: clone connection to avoid selinux problems 2022-06-17 15:12:20 UTC
Red Hat Knowledge Base (Article) 6964739 0 None None None 2022-06-24 20:06:55 UTC
Red Hat Product Errata RHBA-2022:5180 0 None None None 2022-06-30 05:31:33 UTC

Comment 1 Lalatendu Mohanty 2022-06-17 19:14:24 UTC
As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2095264#c9 by @jcaamano 

Which 4.y.z to 4.y'.z' updates increase vulnerability? Which types of clusters?

4.9.z to 4.9.38: all ovn-kubernetes clusters
4.9.z to 4.9.39: all ovn-kubernetes with static IP configurations

What is the impact? Is it serious enough to warrant removing update recommendations?

In 4.9.38, ovn-kubernetes with static IP configurations will fail deploying and all other ovn-kubernetes clusters networking will break after first reboot.
In 4.9.39, ovn-kubernetes with static IP configurations will fail deploying.

How involved is remediation?

For deployments with no static IP configuration, access the nodes through the provisioning network or a console and:
- run `systemctl start ovs-configuration`
- run `nmcli -g name c show -active | egrep "(ovs-if-|-slave-ovs-clone)" | xargs -I % nmcli c mod % connect.autoconnect yes`
- reboot

For deployments with static IP configuration, access the nodes through the provisioning network or a console and:
- Set selinux to permissive mode (as documented in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_selinux/changing-selinux-states-and-modes_using-selinux)
- run `systemctl start ovs-configuration`
- run `nmcli -g name c show -active | egrep "(ovs-if-|-slave-ovs-clone)" | xargs -I % nmcli c mod % connect.autoconnect yes`
- reboot
- Set selinux to enforcing mode (as documented in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_selinux/changing-selinux-states-and-modes_using-selinux)

These steps may have to be performed again if the network configuration in the node resets to its original state, although this would be unexpected.

Is this a regression?

Yes.

Comment 2 W. Trevor King 2022-06-21 22:32:03 UTC
I'm bringing FastFix down from bug 2095264 (the series head) to this 4.9.z bug.

Comment 5 Ross Brattain 2022-06-22 20:25:45 UTC
Verified on 4.9.0-0.nightly-2022-06-22-151807 vSphere UPI static-ip.

-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  292 Jun 22 15:56 /etc/NetworkManager/system-connections/br-ex.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  292 Jun 22 15:56 /etc/NetworkManager/systemConnectionsMerged/br-ex.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_var_run_t:s0 527 Jun 22 18:52 /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_var_run_t:s0 324 Jun 22 18:52 /etc/NetworkManager/systemConnectionsMerged/ovs-if-phys0.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  186 Jun 22 15:56 /etc/NetworkManager/systemConnectionsMerged/ovs-port-br-ex.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  187 Jun 22 15:56 /etc/NetworkManager/systemConnectionsMerged/ovs-port-phys0.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  527 Jun 22 18:52 /etc/NetworkManager/system-connections/ovs-if-br-ex.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  324 Jun 22 18:52 /etc/NetworkManager/system-connections/ovs-if-phys0.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  186 Jun 22 15:56 /etc/NetworkManager/system-connections/ovs-port-br-ex.nmconnection
-rw-------. 1 root root system_u:object_r:NetworkManager_etc_rw_t:s0  187 Jun 22 15:56 /etc/NetworkManager/system-connections/ovs-port-phys0.nmconnection


Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + echo 'Static IP addressing detected on default gateway connection: 03da7500-2101-c722-2438-d0d006c28c73'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: Static IP addressing detected on default gateway connection: 03da7500-2101-c722-2438-d0d006c28c73
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + nmcli conn clone 03da7500-2101-c722-2438-d0d006c28c73 ovs-if-br-ex
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: ens192 (03da7500-2101-c722-2438-d0d006c28c73) cloned as ovs-if-br-ex (b736bc6d-6bea-4dae-95d8-4636d48af86d).
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + shopt -s nullglob
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + new_conn_files=(${NM_CONN_PATH}/"${ovs_interface}"*)
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + shopt -u nullglob
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + '[' 1 -ne 1 ']'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + '[' '!' -f /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection ']'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + new_conn_file=/etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^multi-connect=.*$/d' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection


Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^autoconnect=.*$/d' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[connection\]$/a autoconnect=false' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[connection\]$/,/^\[/ s/^type=.*$/type=ovs-interface/' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[connection\]$/a slave-type=ovs-port' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[connection\]$/a master=d71f63ec-2842-4076-9807-16ba587bb611' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + grep interface-name= /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[connection\]$/,/^\[/ s/^interface-name=.*$/interface-name=br-ex/' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + grep cloned-mac-address= /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[ethernet\]$/a cloned-mac-address=00:50:56:ac:64:18' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + grep mtu= /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + sed -i '/^\[ethernet\]$/a mtu=1500' /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + cat
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + nmcli c load /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + echo 'Loaded new ovs-if-br-ex connection file: /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: Loaded new ovs-if-br-ex connection file: /etc/NetworkManager/systemConnectionsMerged/ovs-if-br-ex.nmconnection
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + configure_driver_options ens192
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + intf=ens192
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + '[' '!' -f /sys/class/net/ens192/device/uevent ']'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: ++ cat /sys/class/net/ens192/device/uevent
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: ++ grep DRIVER
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: ++ awk -F = '{print $2}'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + driver=vmxnet3
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + echo 'Driver name is' vmxnet3
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: Driver name is vmxnet3
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + '[' vmxnet3 = vmxnet3 ']'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + ifconfig ens192 allmulti
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + '[' -f /etc/ovnk/extra_bridge ']'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + '[' '!' -f /etc/ovnk/extra_bridge ']'
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + nmcli connection show br-ex1
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + nmcli connection show ovs-if-phys1
Jun 22 15:56:13 compute-0 configure-ovs.sh[1339]: + ovs-vsctl --timeout=30 --if-exists del-br br0

Comment 6 W. Trevor King 2022-06-22 23:11:17 UTC
I'm adding UpdateRecommendationsBlocked to this bug, based on the impact statement in comment 1.  We've made a number of graph-data changes as a result of this issue, including:

* Blocking updates into 4.9.38 and 4.9.39 from both 4.8.z and 4.9.z [1,2].
* Tombstoning 4.9.39 to withold it from supported channels [3].

[1]: https://github.com/openshift/cincinnati-graph-data/pull/2072
[2]: https://github.com/openshift/cincinnati-graph-data/pull/2080
[3]: https://github.com/openshift/cincinnati-graph-data/pull/2093

Comment 7 Prashanth Sundararaman 2022-06-23 13:41:36 UTC
*** Bug 2097872 has been marked as a duplicate of this bug. ***

Comment 9 errata-xmlrpc 2022-06-30 05:31:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.40 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5180

Comment 10 Jaime Caamaño Ruiz 2022-07-08 12:28:11 UTC
*** Bug 2103877 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.