1794671 – Dynamic MAC assignments made before generating MAC prefix

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1794671 - Dynamic MAC assignments made before generating MAC prefix

Summary: Dynamic MAC assignments made before generating MAC prefix

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux Fast Datapath
Classification:	Red Hat
Component:	ovn2.12
Sub Component:
Version:	RHEL 8.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	---
Assignee:	Dumitru Ceara
QA Contact:	Jianlin Shi
Docs Contact:
URL:
Whiteboard:
Depends On:	1781223
Blocks:
TreeView+	depends on / blocked

Reported:	2020-01-24 10:02 UTC by Dumitru Ceara
Modified:	2020-03-10 10:08 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1781223
Environment:
Last Closed:	2020-03-10 10:08:18 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2020:0752	0	None	None	None	2020-03-10 10:08:33 UTC

Description Dumitru Ceara 2020-01-24 10:02:50 UTC

+++ This bug was initially created as a clone of Bug #1781223 +++

Description of problem:

ovn-kubernetes makes use of dynamic IP and MAC assignment from OVN.  ovn-northd generates a random prefix for the MAC addresses it uses.  I noticed that on ovn-northd startup, it will assign MAC addresses before it has generated the prefix.  See the log below.  You'll see it generated 36 MAC addresses with a blank prefix before switching to the prefix it generated.

2019-12-09T14:39:08Z|00121|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00122|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00123|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00124|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00125|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00126|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00127|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00128|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00129|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00130|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00131|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00132|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00133|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00134|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:18Z|00135|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:18Z|00136|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:20Z|00137|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:20Z|00138|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::2:b8e1:4cff:fe00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:21Z|00139|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:21Z|00140|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:b8e1:4cff:fe00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:22Z|00141|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:06' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00142|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:6' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00143|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:16' to port 'openshift-apiserver_apiserver-8mqtx'


Version-Release number of selected component (if applicable):

ovn2.11-2.11.1-20.el7fdp.x86_64.rpm

This also has some patches applied to add additional logging, like the messages in this bug report.

How reproducible:

I've noticed the blank prefix MAC addresses every time I've looked into the addresses assigned by ovn-northd.

--- Additional comment from Russell Bryant on 2019-12-09 16:15:47 UTC ---

Patch posted here: https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365776.html

--- Additional comment from Russell Bryant on 2019-12-09 16:29:16 UTC ---

In my latest ovn-kubernetes testing, I noticed this issue occurring after an ovn-northd restart.  It appears that if you're running ovn-northd in active-active mode, the same MAC resetting would occur the first time a given instance of ovn-northd becomes the leader.

--- Additional comment from Russell Bryant on 2019-12-09 18:00:35 UTC ---

I've attached a backport that applies cleanly after the other backports applied to ovn2.11.

--- Additional comment from Russell Bryant on 2019-12-11 17:54:23 UTC ---

Please also backport the following patches to assist in debugging similar issues:

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365705.html

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365664.html

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365313.html

Comment 3 Jianlin Shi 2020-02-03 02:31:57 UTC

as there is no log before the fixed version, so can't reproduce on old version.

Verified on 2.12.0-26:

[root@dell-per740-12 bz1781223]# rpm -qa | grep -E "ovn|openvswitch"                                  
ovn2.12-host-2.12.0-26.el7fdp.x86_64                                                                  
openvswitch2.12-2.12.0-21.el7fdp.x86_64                                                               
ovn2.12-central-2.12.0-26.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch                                                 
ovn2.12-2.12.0-26.el7fdp.x86_64

#!/bin/bash

systemctl restart openvswitch
systemctl restart ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642

ovs-vsctl set open . external-ids:system_id=hv1 external-ids:ovn-remote=tcp:20.0.30.25:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.30.25

systemctl restart ovn-controller

ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 lsp1
ovn-nbctl  set Logical-Switch ls1 other_config:subnet=192.168.0.0/16
ovn-nbctl set Logical-switch ls1 other_config:ipv6_prefix=2003::0                                     
ovn-nbctl lsp-set-addresses lsp1 dynamic                                                              
ovn-nbctl list logical_switch_port lsp1

[root@dell-per740-12 bz1781223]# ovn-nbctl list nb_global                                             
_uuid               : 80d7eda8-2f45-4be4-b138-d6a9bc9891a2                                            
connections         : [dd39351f-6b81-4ab3-b074-6c0550219fa7]                                          
external_ids        : {}                                                                              
hv_cfg              : 0                                                                               
ipsec               : false                                                                           
nb_cfg              : 0                                                                               
options             : {mac_prefix="ea:8c:8d", svc_monitor_mac="96:7c:e8:d3:ea:a7"}                    
sb_cfg              : 0                                                                               
ssl                 : []

[root@dell-per740-12 bz1781223]# cat /var/log/ovn/ovn-northd.log                                      
2020-02-03T02:28:59.186Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-northd.log
2020-02-03T02:28:59.196Z|00002|reconnect|INFO|unix:/run/ovn/ovnnb_db.sock: connecting...
2020-02-03T02:28:59.196Z|00003|reconnect|INFO|unix:/run/ovn/ovnsb_db.sock: connecting...
2020-02-03T02:28:59.196Z|00004|reconnect|INFO|unix:/run/ovn/ovnnb_db.sock: connected
2020-02-03T02:28:59.196Z|00005|reconnect|INFO|unix:/run/ovn/ovnsb_db.sock: connected
2020-02-03T02:28:59.197Z|00006|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.
2020-02-03T02:28:59.415Z|00007|ovn_northd|INFO|Assigned dynamic IPv4 address '192.168.0.2' to port 'lsp1'
2020-02-03T02:28:59.415Z|00008|ovn_northd|INFO|Assigned dynamic MAC address 'ea:8c:8d:a8:00:03' to port 'lsp1'

<==== mac is generated from mac prefix

2020-02-03T02:28:59.415Z|00009|ovn_northd|INFO|Assigned dynamic IPv6 address '2003::e88c:8dff:fea8:3' to port 'lsp1'

Comment 5 errata-xmlrpc 2020-03-10 10:08:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0752

Note You need to log in before you can comment on or make changes to this bug.