1781223 – Dynamic MAC assignments made before generating MAC prefix

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1781223 - Dynamic MAC assignments made before generating MAC prefix

Summary: Dynamic MAC assignments made before generating MAC prefix

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux Fast Datapath
Classification:	Red Hat
Component:	ovn2.11
Sub Component:
Version:	RHEL 8.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	---
Assignee:	Dumitru Ceara
QA Contact:	Jianlin Shi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1794671
TreeView+	depends on / blocked

Reported:	2019-12-09 14:59 UTC by Russell Bryant
Modified:	2020-03-10 10:08 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1794671 (view as bug list)
Environment:
Last Closed:	2020-03-10 10:07:57 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
backport to ovn2.11 branch, applies cleanly after other backports (4.64 KB, text/plain) 2019-12-09 18:00 UTC, Russell Bryant	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2020:0750	0	None	None	None	2020-03-10 10:08:30 UTC

Description Russell Bryant 2019-12-09 14:59:21 UTC

Description of problem:

ovn-kubernetes makes use of dynamic IP and MAC assignment from OVN.  ovn-northd generates a random prefix for the MAC addresses it uses.  I noticed that on ovn-northd startup, it will assign MAC addresses before it has generated the prefix.  See the log below.  You'll see it generated 36 MAC addresses with a blank prefix before switching to the prefix it generated.

2019-12-09T14:39:08Z|00121|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00122|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00123|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00124|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00125|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00126|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00127|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00128|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00129|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00130|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00131|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00132|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00133|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00134|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:18Z|00135|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:18Z|00136|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:20Z|00137|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:20Z|00138|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::2:b8e1:4cff:fe00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:21Z|00139|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:21Z|00140|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:b8e1:4cff:fe00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:22Z|00141|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:06' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00142|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:6' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00143|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:16' to port 'openshift-apiserver_apiserver-8mqtx'


Version-Release number of selected component (if applicable):

ovn2.11-2.11.1-20.el7fdp.x86_64.rpm

This also has some patches applied to add additional logging, like the messages in this bug report.

How reproducible:

I've noticed the blank prefix MAC addresses every time I've looked into the addresses assigned by ovn-northd.

Comment 1 Russell Bryant 2019-12-09 16:15:47 UTC

Patch posted here: https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365776.html

Comment 2 Russell Bryant 2019-12-09 16:29:16 UTC

In my latest ovn-kubernetes testing, I noticed this issue occurring after an ovn-northd restart.  It appears that if you're running ovn-northd in active-active mode, the same MAC resetting would occur the first time a given instance of ovn-northd becomes the leader.

Comment 3 Russell Bryant 2019-12-09 18:00:35 UTC

Created attachment 1643383 [details]
backport to ovn2.11 branch, applies cleanly after other backports

I've attached a backport that applies cleanly after the other backports applied to ovn2.11.

Comment 4 Russell Bryant 2019-12-11 17:54:23 UTC

Please also backport the following patches to assist in debugging similar issues:

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365705.html

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365664.html

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365313.html

Comment 7 Jianlin Shi 2020-02-03 02:13:12 UTC

as there is no log before the fixed version, so can't reproduce on old version.

Verified on ovn2.11.1-32:

#!/bin/bash                                                                                           
                                                                                                      
systemctl restart openvswitch                                                                         
systemctl restart ovn-northd                                                                          
ovn-nbctl set-connection ptcp:6641                                                                    
ovn-sbctl set-connection ptcp:6642                                                                    
                                                                                                      
ovs-vsctl set open . external-ids:system_id=hv0 external-ids:ovn-remote=tcp:20.0.30.26:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.30.26
                                                                                                      
systemctl restart ovn-controller                                                                      
                                                                                                      
ovn-nbctl ls-add ls1                                                                                  
ovn-nbctl lsp-add ls1 lsp1                                                                            
ovn-nbctl  set Logical-Switch ls1 other_config:subnet=192.168.0.0/16                                  
ovn-nbctl set Logical-switch ls1 other_config:ipv6_prefix=2003::0                                     
ovn-nbctl lsp-set-addresses lsp1 dynamic                                                              
ovn-nbctl list logical_switch_port lsp1 


[root@hp-dl380pg8-12 bz1781223]# rpm -qa | grep -E "ovn|openvswitch"
ovn2.11-central-2.11.1-32.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-2.11.1-32.el7fdp.x86_64
ovn2.11-host-2.11.1-32.el7fdp.x86_64
openvswitch2.11-2.11.0-35.el7fdp.x86_64

[root@hp-dl380pg8-12 bz1781223]# ovn-nbctl list nb_global
_uuid               : ef4b4f8f-4bba-46fb-a60e-ede49f9d7481
connections         : [2cf377e1-489e-49b0-b850-dfe4847b5c91]
external_ids        : {}
hv_cfg              : 0
ipsec               : false
nb_cfg              : 0
options             : {mac_prefix="ca:8e:0c", svc_monitor_mac="82:f8:6f:cf:3c:97"}
sb_cfg              : 0
ssl                 : []

log from ovn-northd:

[root@hp-dl380pg8-12 bz1781223]# cat /var/log/openvswitch/ovn-northd.log 
2020-02-03T02:07:29.758Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-northd.log
2020-02-03T02:07:29.760Z|00002|reconnect|INFO|unix:/run/openvswitch/ovnnb_db.sock: connecting...
2020-02-03T02:07:29.760Z|00003|reconnect|INFO|unix:/run/openvswitch/ovnsb_db.sock: connecting...
2020-02-03T02:07:29.760Z|00004|reconnect|INFO|unix:/run/openvswitch/ovnnb_db.sock: connected
2020-02-03T02:07:29.760Z|00005|reconnect|INFO|unix:/run/openvswitch/ovnsb_db.sock: connected
2020-02-03T02:07:29.760Z|00006|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.
2020-02-03T02:07:29.853Z|00007|ovn_northd|INFO|Assigned dynamic IPv4 address '192.168.0.2' to port 'lsp1'
2020-02-03T02:07:29.853Z|00008|ovn_northd|INFO|Assigned dynamic MAC address 'ca:8e:0c:a8:00:03' to port 'lsp1'

<=== mac is generated from mac prefix

2020-02-03T02:07:29.853Z|00009|ovn_northd|INFO|Assigned dynamic IPv6 address '2003::c88e:cff:fea8:3' to port 'lsp1'

Comment 9 errata-xmlrpc 2020-03-10 10:07:57 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0750

Note You need to log in before you can comment on or make changes to this bug.