2057160 – configure-ovs selects wrong interface on reboot

Bug 2057160 - configure-ovs selects wrong interface on reboot

Summary: configure-ovs selects wrong interface on reboot

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.11.0
Assignee:	Andreas Karis
QA Contact:	Ross Brattain
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	2056605 (view as bug list)
Depends On:
Blocks:	2081317
TreeView+	depends on / blocked

Reported:	2022-02-22 20:12 UTC by Ben Nemec
Modified:	2022-08-10 10:51 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2081317 (view as bug list)
Environment:
Last Closed:	2022-08-10 10:50:45 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
configure-ovs logs from rebooted node (165.99 KB, text/plain) 2022-02-22 20:12 UTC, Ben Nemec	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift machine-config-operator pull 2971	0	None	Merged	Bug 2057160: configure-ovs.sh: Provide store hint for default route interface	2022-07-26 13:56:51 UTC
Red Hat Product Errata	RHSA-2022:5069	0	None	None	None	2022-08-10 10:51:02 UTC

Description Ben Nemec 2022-02-22 20:12:13 UTC

Created attachment 1862743 [details]
configure-ovs logs from rebooted node

Description of problem: When rebooting a node with multiple network interfaces, configure-ovs selects a different one after the reboot than it did on initial deployment. This appears to be due to the re-creation of the bridge. After the bridge is destroyed and the interface returned to its default state, the metric for the default route gets demoted and a different interface has higher priority:

Feb 22 18:33:12 master-0 configure-ovs.sh[2775]: default via 192.168.221.1 dev enp3s0 proto dhcp metric 101
Feb 22 18:33:12 master-0 configure-ovs.sh[2775]: default via 192.168.222.1 dev enp4s0 proto dhcp metric 102
Feb 22 18:33:12 master-0 configure-ovs.sh[2775]: default via 192.168.111.1 dev enp2s0 proto dhcp metric 103

enp2s0 is the correct primary interface. Note that before destruction of the bridge the ordering is correct:

Feb 22 18:33:11 master-0 configure-ovs.sh[2775]: default via 192.168.111.1 dev br-ex proto dhcp metric 49
Feb 22 18:33:11 master-0 configure-ovs.sh[2775]: default via 192.168.221.1 dev enp3s0 proto dhcp metric 101
Feb 22 18:33:11 master-0 configure-ovs.sh[2775]: default via 192.168.222.1 dev enp4s0 proto dhcp metric 102

Maybe this is some accidental fallout from changing the metric for the br-ex route?


Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-02-18-121223


How reproducible: Happens consistently from what I understand in talking with others who have run into this.


Steps to Reproduce:
1. Deploy cluster with additional nics beyond the primary one
2. Reboot a node
3.

Actual results: Node selects a different interface for br-ex after reboot


Expected results: Node selects the same interface for br-ex after reboot


Additional info: I've attached the full configure-ovs output from an affected node

Comment 1 Andreas Karis 2022-02-23 14:01:22 UTC

So with or without that fix, I suppose it wouldn't make a difference, but to be tested.

Interface enp2s0 is removed from br-ex, and then plugged into the network, now as the last interface. So, the new interface ordering enp3s0 ->  enp4s0 ->  enp2s0 actually makes sense, as this is the new order of activation.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/managing-the-default-gateway-setting_configuring-and-managing-networking
~~~
If you set a default gateway without a metric value, NetworkManager automatically sets the metric value based on the interface type. For that, NetworkManager assigns the default value of this network type to the first connection that is activated, and sets an incremented value to each other connection of the same type in the order they are activated. For example, if two Ethernet connections with a default gateway exist, NetworkManager sets a metric of 100 on the route to the default gateway of the connection that you activate first. For the second connection, NetworkManager sets 101. 
~~~

I just think that we need to come up with a more stable way to guarantee interface ordering. Perhaps by writing the lowered metric to the interface during teardown of the bridge, but that, too, might have its drawbacks.

Comment 6 Andreas Karis 2022-04-04 15:53:18 UTC

*** Bug 2056605 has been marked as a duplicate of this bug. ***

Comment 8 Ross Brattain 2022-04-06 18:17:10 UTC


Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?

Comment 9 Ross Brattain 2022-04-07 11:08:34 UTC

Ah, is looks like the interfaces are on different subnets?  I was testing multiple NICs in the same L2 and DHCP subnet which BZ 2043738 suggests are not supported?

Comment 10 Andreas Karis 2022-04-07 12:51:03 UTC

haha yeah they must be in different subnets indeed :-)

Comment 23 Ben Nemec 2022-04-22 17:20:16 UTC

Tested this in a dev-scripts cluster and it did the right thing:

Before reboot:
default via 192.168.111.1 dev br-ex proto dhcp metric 48 
default via 192.168.221.1 dev enp3s0 proto dhcp metric 101 
default via 192.168.222.1 dev enp4s0 proto dhcp metric 102

After reboot:
default via 192.168.111.1 dev br-ex proto dhcp metric 48 
default via 192.168.221.1 dev enp3s0 proto dhcp metric 101 
default via 192.168.222.1 dev enp4s0 proto dhcp metric 102

The correct ordering was maintained and the correct interface was used by configure-ovs.

Comment 24 Ross Brattain 2022-04-22 18:24:16 UTC

(In reply to Ben Nemec from comment #23)
> Tested this in a dev-scripts cluster and it did the right thing:
> 
> Before reboot:
> default via 192.168.111.1 dev br-ex proto dhcp metric 48 
> default via 192.168.221.1 dev enp3s0 proto dhcp metric 101 
> default via 192.168.222.1 dev enp4s0 proto dhcp metric 102
> 
> After reboot:
> default via 192.168.111.1 dev br-ex proto dhcp metric 48 
> default via 192.168.221.1 dev enp3s0 proto dhcp metric 101 
> default via 192.168.222.1 dev enp4s0 proto dhcp metric 102
> 
> The correct ordering was maintained and the correct interface was used by
> configure-ovs.

configure-ovs ordering is stable in my attempts but etcd seems to be selecting the wrong interface.  Implied in that config is 192.168.111.1 sorting first numerically.

If subnet ordering is different the interfaces will order differently?


cluest-etcd-operator uses GetPreferredInternalIPAddressForNodeName which iterates over node.Status.Addresses

https://github.com/openshift/cluster-etcd-operator/blob/master/pkg/dnshelpers/util.go#L40


	for _, currAddress := range node.Status.Addresses {
		if currAddress.Type == corev1.NodeInternalIP {
			switch ipFamily {
			case "tcp4":

node.Status.Addresses is not sorted?

Is looks like kubelet nodeAddressesFunc provides node.Status.Addresses

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go#L642

	// if cloud is not nil, we expect the cloud resource sync manager to exist
	var nodeAddressesFunc func() ([]v1.NodeAddress, error)
	if kl.cloud != nil {
		nodeAddressesFunc = kl.cloudResourceSyncManager.NodeAddresses
	}

I'm happy to verify that configure-ovs route metric and interface ordering is stable but the cluster is not deploying in the presence of multiple interfaces.

Maybe this is platform specific and only works on OSP.

Comment 25 Ben Nemec 2022-04-22 21:08:57 UTC

This is not a problem in my environment. Only the primary node ip is showing up in the node address list:

status:
  addresses:
  - address: 192.168.111.20
    type: InternalIP
  - address: master-0
    type: Hostname

Which seems right to me since a node shouldn't have multiple internal IPs (except dual stack, I suppose).

I think the problem here might be having multiple interfaces on the same subnet, which can confuse the existing ip selection logic. That's a completely separate issue to the one I was reporting here.

Comment 26 Ross Brattain 2022-04-25 03:41:36 UTC

Could be a difference between IPI and UPI.  I'll try IPI and add interfaces after install.

Comment 28 Ross Brattain 2022-04-26 23:54:59 UTC

Verified on 4.11.0-0.nightly-2022-04-26-085341 on vSphere IPI

Added extra two interfaces via vSphere console after cluster install, rebooted all nodes.

OVN br-ex is stable.

Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]:        valid_lft 86398sec preferred_lft 86398sec
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]:     inet6 fe80::da8a:74b8:eae:7658/64 scope link noprefixroute
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]:        valid_lft forever preferred_lft forever
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + ip route show
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: default via 172.31.248.1 dev br-ex proto dhcp metric 48
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: default via 192.168.70.1 dev ens224 proto dhcp metric 100
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: default via 192.168.71.1 dev ens256 proto dhcp metric 101
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: 172.31.248.0/23 dev br-ex proto kernel scope link src 172.31.249.173 metric 48
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: 192.168.70.0/25 dev ens224 proto kernel scope link src 192.168.70.35 metric 100
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: 192.168.71.0/25 dev ens256 proto kernel scope link src 192.168.71.35 metric 101
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + ip -6 route show
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ::1 dev lo proto kernel metric 256 pref medium
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: fe80::/64 dev br-ex proto kernel metric 48 pref medium
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: fe80::/64 dev ens224 proto kernel metric 100 pref medium
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: fe80::/64 dev ens256 proto kernel metric 101 pref medium
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + exit 0
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 systemd[1]: ovs-configuration.service: Succeeded.
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 systemd[1]: Started Configures OVS with proper host networking configuration.
Apr 26 23:04:10 rbrattai-o411v22m-bqrv7-master-1 systemd[1]: ovs-configuration.service: Consumed 1.814s CPU time

Hint is loaded

Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ local extra_bridge_file=/etc/ovnk/extra_bridge
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ local extra_bridge=
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' -f /etc/ovnk/extra_bridge ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' 0 -lt 12 ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' '' '!=' '' ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ ip route show default
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ grep -v br-ex1
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ awk '{ if ($4 == "dev") { print $5; exit } }'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ iface=ens224
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ [[ -n ens224 ]]
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ break
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' ens224 '!=' br-ex ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' ens224 '!=' br-ex1 ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ get_iface_default_hint /var/lib/ovnk/iface_default_hint
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ '[' -f /var/lib/ovnk/iface_default_hint ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++++ cat /var/lib/ovnk/iface_default_hint
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ local iface_default_hint=ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ '[' ens192 '!=' '' ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ '[' ens192 '!=' br-ex ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ '[' ens192 '!=' br-ex1 ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ '[' -d /sys/class/net/ens192 ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ echo ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ return
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ iface_default_hint=ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' ens192 '!=' '' ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' ens192 '!=' ens224 ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' 0 -le 12 ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: +++ ip route show default dev ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' 'default via 172.31.248.1 proto dhcp metric 102 ' '!=' '' ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ iface=ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ break
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ '[' ens192 '!=' '' ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ write_iface_default_hint /var/lib/ovnk/iface_default_hint ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ local iface=ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ echo ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ echo ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + iface=ens192
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + '[' ens192 '!=' br-ex ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + '[' -f /etc/ovnk/extra_bridge ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: ++ nmcli connection show --active br-ex
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + '[' -z '' ']'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + echo 'Bridge br-ex is not active, restoring previous configuration before proceeding...'
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: Bridge br-ex is not active, restoring previous configuration before proceeding...
Apr 26 23:03:04 rbrattai-o411v22m-bqrv7-master-1 configure-ovs.sh[2145]: + rollback_nm



Only one InternalIP 

oc get nodes --no-headers -o 'custom-columns=I:.status.addresses[*]'
map[address:172.31.249.67 type:ExternalIP],map[address:172.31.249.67 type:InternalIP],map[address:rbrattai-o411v22m-bqrv7-master-0 type:Hostname]
map[address:172.31.249.173 type:ExternalIP],map[address:172.31.249.173 type:InternalIP],map[address:rbrattai-o411v22m-bqrv7-master-1 type:Hostname]
map[address:172.31.249.90 type:ExternalIP],map[address:172.31.249.90 type:InternalIP],map[address:rbrattai-o411v22m-bqrv7-master-2 type:Hostname]
map[address:172.31.249.113 type:ExternalIP],map[address:172.31.249.113 type:InternalIP],map[address:rbrattai-o411v22m-bqrv7-worker-8jdgl type:Hostname]
map[address:172.31.249.92 type:ExternalIP],map[address:172.31.249.92 type:InternalIP],map[address:rbrattai-o411v22m-bqrv7-worker-9vhlh type:Hostname]


etcd seems happy.
+------------------+---------+----------------------------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |               NAME               |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+----------------------------------+-----------------------------+-----------------------------+------------+
| 2216456203e1ee4e | started | rbrattai-o411v22m-bqrv7-master-0 |  https://172.31.249.67:2380 |  https://172.31.249.67:2379 |      false |
| 6070197663463d29 | started | rbrattai-o411v22m-bqrv7-master-2 |  https://172.31.249.90:2380 |  https://172.31.249.90:2379 |      false |
| 67a3d966dcc5457d | started | rbrattai-o411v22m-bqrv7-master-1 | https://172.31.249.173:2380 | https://172.31.249.173:2379 |      false |
+------------------+---------+----------------------------------+-----------------------------+-----------------------------+------------+

Comment 29 Ross Brattain 2022-04-28 05:14:38 UTC

Even thought OVN stably selects an interface on UPI the cluster fails.  OVN is not communicating with kubelet.

OVN is not abiding by https://docs.openshift.com/container-platform/4.10/support/troubleshooting/troubleshooting-network-issues.html#nw-how-nw-iface-selected_troubleshooting-network-issues

OVN's "correct" interface is not matching /etc/systemd/system/kubelet.service.d/20-nodenet.conf

> For installations on bare metal or with virtual machines that have more than one network interface controller (NIC), the NIC that OpenShift Container Platform uses for communication with the Kubernetes API server is determined by the nodeip-configuration.service service unit that is run by systemd when the node boots. The service iterates through the network interfaces on the node and the first network interface that is configured with a subnet than can host the IP address for the API server is selected for OpenShift Container Platform communication.


configure-ovs needs to:

> Create a shell script that determines the IP address to use for OpenShift Container Platform communication. Have the script create a custom unit file such as /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf. Use the custom unit file, 98-nodenet-override.conf, to set the KUBELET_NODE_IP environment variable to the IP address.


Or vice-versa if KUBELET_NODE_IP is defined then that is the "correct" IP and configure-ovs should use that instead of the default route logic.

/etc/systemd/system/kubelet.service.d/20-nodenet.conf
[Service]
Environment="KUBELET_NODE_IP=10.123.74.75" "KUBELET_NODE_IPS=10.123.74.75"

Comment 30 Andreas Karis 2022-04-28 10:01:45 UTC

There's also this here for the IP selection logic for kubelet:
https://github.com/openshift/machine-config-operator/pull/2888

Comment 31 Andreas Karis 2022-04-28 10:50:24 UTC

Ok but then strictly speaking it's still not this bug here to fix this in given that the logic for picking a given interface hasn't changed with this commit and that I suppose that this was already broken before this change.

I opened https://bugzilla.redhat.com/show_bug.cgi?id=2079832

I also drafted up https://github.com/openshift/machine-config-operator/pull/3123 even though I can't test this on an actual cluster. Could we have a look at this together at your cluster and see if that new PR works and if not, is there a way for me to get access to your cluster for testing / tuning / improving that approach?

Thanks!

Comment 32 Ross Brattain 2022-04-28 11:54:19 UTC

On further thought it should be vice-versa.  OVN should defer to KUBELET_NODE_IP.

We actually had a bug or Slack thread on this somewhere before, customers coming from OpenShiftSDN will expect KUBELET_NODE_IP settings to be honored.

If there are multiple default routes configure-ovs should be checking KUBELET_NODE_IP and using that instead of selecting the first ip by output.

Comment 40 Ross Brattain 2022-05-02 14:55:30 UTC

Was able to get multiple default routes on master with IPI OSP and additionalNetworkIDs, but not the same IP sorting, 192.x vs 10.x

name: master
platform:
  openstack:
    additionalNetworkIDs: &1
    - ab3a8289-ca41-4825-a48a-8f8a11faaac7
    - aa7dcdfa-2b6a-4b7a-bba9-3dabc0972bb5
    rootVolume:
      size: 25
      type: tripleo
replicas: 3


Apr 29 18:15:31.563544 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] + ip route show
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] default via 192.168.0.1 dev br-ex proto dhcp metric 48
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] default via 10.0.143.254 dev ens4 proto dhcp metric 100
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] default via 10.0.155.254 dev ens5 proto dhcp metric 101
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] 10.0.140.0/22 dev ens4 proto kernel scope link src 10.0.141.53 metric 100
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] 10.0.152.0/22 dev ens5 proto kernel scope link src 10.0.152.49 metric 101
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] 169.254.169.254 via 192.168.0.10 dev br-ex proto dhcp metric 48
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] 169.254.169.254 via 10.0.140.18 dev ens4 proto dhcp metric 100
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] 169.254.169.254 via 10.0.152.23 dev ens5 proto dhcp metric 101
Apr 29 18:15:31.564918 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] 192.168.0.0/18 dev br-ex proto kernel scope link src 192.168.1.30 metric 48
Apr 29 18:15:31.578694 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] + ip -6 route show
Apr 29 18:15:31.580093 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] ::1 dev lo proto kernel metric 256 pref medium
Apr 29 18:15:31.580093 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] fe80::/64 dev br-ex proto kernel metric 48 pref medium
Apr 29 18:15:31.580093 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] fe80::/64 dev ens4 proto kernel metric 100 pref medium
Apr 29 18:15:31.580093 rbratt-o411o5-nbbpk-master-0 configure-ovs.sh[1451] fe80::/64 dev ens5 proto kernel metric 101 pref medium




Apr 29 19:34:00.935429 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + ip route show
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: default via 10.0.143.254 dev ens4 proto dhcp metric 100
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: default via 10.0.155.254 dev ens5 proto dhcp metric 101
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: default via 192.168.0.1 dev ens3 proto dhcp metric 102
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: 10.0.140.0/22 dev ens4 proto kernel scope link src 10.0.140.38 metric 100
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: 10.0.152.0/22 dev ens5 proto kernel scope link src 10.0.153.220 metric 101
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: 169.254.169.254 via 10.0.140.18 dev ens4 proto dhcp metric 100
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: 169.254.169.254 via 10.0.152.23 dev ens5 proto dhcp metric 101
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: 169.254.169.254 via 192.168.0.10 dev ens3 proto dhcp metric 102
Apr 29 19:34:00.936122 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: 192.168.0.0/18 dev ens3 proto kernel scope link src 192.168.2.95 metric 102
Apr 29 19:34:00.939832 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + ip -6 route show
Apr 29 19:34:00.940366 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ::1 dev lo proto kernel metric 256 pref medium
Apr 29 19:34:00.940366 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: fe80::/64 dev ens4 proto kernel metric 100 pref medium
Apr 29 19:34:00.940366 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: fe80::/64 dev ens5 proto kernel metric 101 pref medium
Apr 29 19:34:00.940366 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: fe80::/64 dev ens3 proto kernel metric 102 pref medium
Apr 29 19:34:00.940366 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
Apr 29 19:34:00.945174 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + touch /run/configure-ovs-boot-done
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ get_default_interface /var/lib/ovnk/iface_default_hint /etc/ovnk/extra_bridge
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local iface=
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local counter=0
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local extra_bridge_file=/etc/ovnk/extra_bridge
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local extra_bridge=
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' -f /etc/ovnk/extra_bridge ']'
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' 0 -lt 12 ']'
Apr 29 19:34:00.945793 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' '' '!=' '' ']'
Apr 29 19:34:00.949930 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ grep -v br-ex1
Apr 29 19:34:00.950433 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ awk '{ if ($4 == "dev") { print $5; exit } }'
Apr 29 19:34:00.950952 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ ip route show default
Apr 29 19:34:00.953641 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ iface=ens4
Apr 29 19:34:00.953641 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ [[ -n ens4 ]]
Apr 29 19:34:00.953641 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ break
Apr 29 19:34:00.953641 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' ens4 '!=' br-ex ']'
Apr 29 19:34:00.953641 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' ens4 '!=' br-ex1 ']'
Apr 29 19:34:00.957302 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ get_iface_default_hint /var/lib/ovnk/iface_default_hint
Apr 29 19:34:00.957302 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 29 19:34:00.957302 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ '[' -f /var/lib/ovnk/iface_default_hint ']'
Apr 29 19:34:00.958967 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++++ cat /var/lib/ovnk/iface_default_hint
Apr 29 19:34:00.960732 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ local iface_default_hint=ens3
Apr 29 19:34:00.962403 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ '[' ens3 '!=' '' ']'
Apr 29 19:34:00.962403 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ '[' ens3 '!=' br-ex ']'
Apr 29 19:34:00.962403 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ '[' ens3 '!=' br-ex1 ']'
Apr 29 19:34:00.962403 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ '[' -d /sys/class/net/ens3 ']'
Apr 29 19:34:00.962403 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ echo ens3
Apr 29 19:34:00.962403 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ return
Apr 29 19:34:00.967113 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ iface_default_hint=ens3
Apr 29 19:34:00.967113 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' ens3 '!=' '' ']'
Apr 29 19:34:00.967113 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' ens3 '!=' ens4 ']'
Apr 29 19:34:00.967113 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' 0 -le 12 ']'
Apr 29 19:34:00.968878 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: +++ ip route show default dev ens3
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' 'default via 192.168.0.1 proto dhcp metric 102 ' '!=' '' ']'
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ iface=ens3
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ break
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ '[' ens3 '!=' '' ']'
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ write_iface_default_hint /var/lib/ovnk/iface_default_hint ens3
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ local iface=ens3
Apr 29 19:34:00.969918 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ echo ens3
Apr 29 19:34:00.976600 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + iface=ens3
Apr 29 19:34:00.976600 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + '[' ens3 '!=' br-ex ']'
Apr 29 19:34:00.976600 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + '[' -f /etc/ovnk/extra_bridge ']'
Apr 29 19:34:00.978147 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: ++ nmcli connection show --active br-ex
Apr 29 19:34:01.027435 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + '[' -z '' ']'
Apr 29 19:34:01.027435 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + echo 'Bridge br-ex is not active, restoring previous configuration before proceeding...'
Apr 29 19:34:01.027435 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: Bridge br-ex is not active, restoring previous configuration before proceeding...
Apr 29 19:34:01.027435 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + rollback_nm

Apr 29 19:34:03.008252 rbratt-o411o5-nbbpk-worker-0-sgkbk configure-ovs.sh[1994]: + nmcli c add type ovs-interface slave-type ovs-port conn.interface br-ex master ovs-port-br-ex con-name ovs-if-br-ex 802-3-ethernet.mtu 1450 802-3-ethernet.cloned-mac-address fa:16:3e:2b:9e:e6 ipv4.route-metric 48 ipv6.route-metric 48 ipv4.may-fail no ipv6.addr-gen-mode stable-privacy


vSphere UPI with KUBELET_NODE_IP set and mixed static and DHCP

# /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf
[Service]
Environment="KUBELET_NODE_IP=172.31.248.76" "KUBELET_NODE_IPS=172.31.248.76"

Apr 29 22:08:24.156680 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + touch /run/configure-ovs-boot-done
Apr 29 22:08:24.158238 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ get_default_interface /var/lib/ovnk/iface_default_hint /etc/ovnk/extra_bridge
Apr 29 22:08:24.158317 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local iface=
Apr 29 22:08:24.158317 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local counter=0
Apr 29 22:08:24.158317 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 29 22:08:24.158317 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local extra_bridge_file=/etc/ovnk/extra_bridge
Apr 29 22:08:24.158481 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local extra_bridge=
Apr 29 22:08:24.158481 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' -f /etc/ovnk/extra_bridge ']'
Apr 29 22:08:24.158481 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' 0 -lt 12 ']'
Apr 29 22:08:24.158481 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' '' '!=' '' ']'
Apr 29 22:08:24.158926 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ grep -v br-ex1
Apr 29 22:08:24.158995 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ ip route show default
Apr 29 22:08:24.159036 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ awk '{ if ($4 == "dev") { print $5; exit } }'
Apr 29 22:08:24.161873 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ iface=ens224
Apr 29 22:08:24.161957 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ [[ -n ens224 ]]
Apr 29 22:08:24.161957 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ break
Apr 29 22:08:24.161957 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' ens224 '!=' br-ex ']'
Apr 29 22:08:24.161957 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' ens224 '!=' br-ex1 ']'
Apr 29 22:08:24.162284 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ get_iface_default_hint /var/lib/ovnk/iface_default_hint
Apr 29 22:08:24.162364 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 29 22:08:24.162364 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ '[' -f /var/lib/ovnk/iface_default_hint ']'
Apr 29 22:08:24.162717 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++++ cat /var/lib/ovnk/iface_default_hint
Apr 29 22:08:24.164096 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ local iface_default_hint=ens192
Apr 29 22:08:24.164166 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ '[' ens192 '!=' '' ']'
Apr 29 22:08:24.164166 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ '[' ens192 '!=' br-ex ']'
Apr 29 22:08:24.164166 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ '[' ens192 '!=' br-ex1 ']'
Apr 29 22:08:24.164166 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ '[' -d /sys/class/net/ens192 ']'
Apr 29 22:08:24.164322 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ echo ens192
Apr 29 22:08:24.164322 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ return
Apr 29 22:08:24.164420 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ iface_default_hint=ens192
Apr 29 22:08:24.164466 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' ens192 '!=' '' ']'
Apr 29 22:08:24.164509 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' ens192 '!=' ens224 ']'
Apr 29 22:08:24.164509 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' 0 -le 12 ']'
Apr 29 22:08:24.164869 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: +++ ip route show default dev ens192
Apr 29 22:08:24.167506 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' 'default via 172.31.248.1 proto static metric 102 ' '!=' '' ']'
Apr 29 22:08:24.167569 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ iface=ens192
Apr 29 22:08:24.167569 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ break
Apr 29 22:08:24.167569 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ '[' ens192 '!=' '' ']'
Apr 29 22:08:24.167569 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ write_iface_default_hint /var/lib/ovnk/iface_default_hint ens192
Apr 29 22:08:24.167795 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local iface_default_hint_file=/var/lib/ovnk/iface_default_hint
Apr 29 22:08:24.167795 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ local iface=ens192
Apr 29 22:08:24.167795 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ echo ens192
Apr 29 22:08:24.169172 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ echo ens192
Apr 29 22:08:24.169664 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + iface=ens192
Apr 29 22:08:24.169664 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + '[' ens192 '!=' br-ex ']'
Apr 29 22:08:24.169664 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + '[' -f /etc/ovnk/extra_bridge ']'
Apr 29 22:08:24.169857 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: ++ nmcli connection show --active br-ex
Apr 29 22:08:24.191567 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + '[' -z '' ']'

Apr 29 22:08:25.269286 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + nmcli c load /etc/NetworkManager/system-connections/ovs-if-br-ex.nmconnection
Apr 29 22:08:25.303595 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + nmcli c mod ovs-if-br-ex ipv4.route-metric 48 ipv6.route-metric 48
Apr 29 22:08:25.336562 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: + echo 'Loaded new ovs-if-br-ex connection file: /etc/NetworkManager/system-connections/ovs-if-br-ex.nmconnection'
Apr 29 22:08:25.336562 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667]: Loaded new ovs-if-br-ex connection file: /etc/NetworkManager/system-connections/ovs-if-br-ex.nmconnection


Apr 29 22:09:26.795976 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] + ip route show
Apr 29 22:09:26.796046 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] default via 172.31.248.1 dev br-ex proto static metric 48
Apr 29 22:09:26.796087 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] default via 192.168.60.1 dev ens224 proto dhcp metric 100
Apr 29 22:09:26.796122 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] default via 192.168.61.1 dev ens256 proto dhcp metric 101
Apr 29 22:09:26.796122 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] 172.31.248.0/23 dev br-ex proto kernel scope link src 172.31.248.76 metric 48
Apr 29 22:09:26.796122 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] 192.168.60.0/25 dev ens224 proto kernel scope link src 192.168.60.33 metric 100
Apr 29 22:09:26.796122 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] 192.168.61.0/25 dev ens256 proto kernel scope link src 192.168.61.65 metric 101
Apr 29 22:09:26.796324 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] + ip -6 route show
Apr 29 22:09:26.797597 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] ::1 dev lo proto kernel metric 256 pref medium
Apr 29 22:09:26.797646 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] fe80::/64 dev ens224 proto kernel metric 100 pref medium
Apr 29 22:09:26.797680 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] fe80::/64 dev ens256 proto kernel metric 101 pref medium
Apr 29 22:09:26.797711 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
Apr 29 22:09:26.797934 rbrattai-o411v51m-h8nkv-compute-0 configure-ovs.sh[1667] + exit 0

Comment 41 Ross Brattain 2022-05-02 15:19:12 UTC

We should probably clarify the implicit requirement for KUBELET_NODE_IP setting in the documentation.  Something like:  

"On OVN cluster with multiple interfaces, before rebooting, the adminstrator should verify that KUBELET_NODE_IP is set to match the OVN br-ex IP address."

OSP IPI seems to set KUBELET_NODE_IP via 20-nodenet.conf

# /etc/systemd/system/kubelet.service.d/20-nodenet.conf
[Service]
Environment="KUBELET_NODE_IP=192.168.2.95" "KUBELET_NODE_IPS=192.168.2.95"

Multiple interfaces on other IPI platforms untested.  UPI needs to manually set KUBELET_NODE_IP.

Comment 43 errata-xmlrpc 2022-08-10 10:50:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069

Note You need to log in before you can comment on or make changes to this bug.