Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1794671

Summary: Dynamic MAC assignments made before generating MAC prefix
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Dumitru Ceara <dceara>
Component: ovn2.12Assignee: Dumitru Ceara <dceara>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: RHEL 8.0CC: ctrautma, dcbw, jishi, mmichels, ralongi, rbryant, tredaelli, william.caban, zzhao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1781223 Environment:
Last Closed: 2020-03-10 10:08:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1781223    
Bug Blocks:    

Description Dumitru Ceara 2020-01-24 10:02:50 UTC
+++ This bug was initially created as a clone of Bug #1781223 +++

Description of problem:

ovn-kubernetes makes use of dynamic IP and MAC assignment from OVN.  ovn-northd generates a random prefix for the MAC addresses it uses.  I noticed that on ovn-northd startup, it will assign MAC addresses before it has generated the prefix.  See the log below.  You'll see it generated 36 MAC addresses with a blank prefix before switching to the prefix it generated.

2019-12-09T14:39:08Z|00121|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00122|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00123|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00124|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00125|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00126|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00127|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00128|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00129|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00130|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00131|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00132|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00133|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00134|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:18Z|00135|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:18Z|00136|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:20Z|00137|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:20Z|00138|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::2:b8e1:4cff:fe00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:21Z|00139|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:21Z|00140|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:b8e1:4cff:fe00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:22Z|00141|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:06' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00142|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:6' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00143|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:16' to port 'openshift-apiserver_apiserver-8mqtx'


Version-Release number of selected component (if applicable):

ovn2.11-2.11.1-20.el7fdp.x86_64.rpm

This also has some patches applied to add additional logging, like the messages in this bug report.

How reproducible:

I've noticed the blank prefix MAC addresses every time I've looked into the addresses assigned by ovn-northd.

--- Additional comment from Russell Bryant on 2019-12-09 16:15:47 UTC ---

Patch posted here: https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365776.html

--- Additional comment from Russell Bryant on 2019-12-09 16:29:16 UTC ---

In my latest ovn-kubernetes testing, I noticed this issue occurring after an ovn-northd restart.  It appears that if you're running ovn-northd in active-active mode, the same MAC resetting would occur the first time a given instance of ovn-northd becomes the leader.

--- Additional comment from Russell Bryant on 2019-12-09 18:00:35 UTC ---

I've attached a backport that applies cleanly after the other backports applied to ovn2.11.

--- Additional comment from Russell Bryant on 2019-12-11 17:54:23 UTC ---

Please also backport the following patches to assist in debugging similar issues:

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365705.html

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365664.html

 - https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365313.html

Comment 3 Jianlin Shi 2020-02-03 02:31:57 UTC
as there is no log before the fixed version, so can't reproduce on old version.

Verified on 2.12.0-26:

[root@dell-per740-12 bz1781223]# rpm -qa | grep -E "ovn|openvswitch"                                  
ovn2.12-host-2.12.0-26.el7fdp.x86_64                                                                  
openvswitch2.12-2.12.0-21.el7fdp.x86_64                                                               
ovn2.12-central-2.12.0-26.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch                                                 
ovn2.12-2.12.0-26.el7fdp.x86_64

#!/bin/bash

systemctl restart openvswitch
systemctl restart ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642

ovs-vsctl set open . external-ids:system_id=hv1 external-ids:ovn-remote=tcp:20.0.30.25:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.30.25

systemctl restart ovn-controller

ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 lsp1
ovn-nbctl  set Logical-Switch ls1 other_config:subnet=192.168.0.0/16
ovn-nbctl set Logical-switch ls1 other_config:ipv6_prefix=2003::0                                     
ovn-nbctl lsp-set-addresses lsp1 dynamic                                                              
ovn-nbctl list logical_switch_port lsp1

[root@dell-per740-12 bz1781223]# ovn-nbctl list nb_global                                             
_uuid               : 80d7eda8-2f45-4be4-b138-d6a9bc9891a2                                            
connections         : [dd39351f-6b81-4ab3-b074-6c0550219fa7]                                          
external_ids        : {}                                                                              
hv_cfg              : 0                                                                               
ipsec               : false                                                                           
nb_cfg              : 0                                                                               
options             : {mac_prefix="ea:8c:8d", svc_monitor_mac="96:7c:e8:d3:ea:a7"}                    
sb_cfg              : 0                                                                               
ssl                 : []

[root@dell-per740-12 bz1781223]# cat /var/log/ovn/ovn-northd.log                                      
2020-02-03T02:28:59.186Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-northd.log
2020-02-03T02:28:59.196Z|00002|reconnect|INFO|unix:/run/ovn/ovnnb_db.sock: connecting...
2020-02-03T02:28:59.196Z|00003|reconnect|INFO|unix:/run/ovn/ovnsb_db.sock: connecting...
2020-02-03T02:28:59.196Z|00004|reconnect|INFO|unix:/run/ovn/ovnnb_db.sock: connected
2020-02-03T02:28:59.196Z|00005|reconnect|INFO|unix:/run/ovn/ovnsb_db.sock: connected
2020-02-03T02:28:59.197Z|00006|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.
2020-02-03T02:28:59.415Z|00007|ovn_northd|INFO|Assigned dynamic IPv4 address '192.168.0.2' to port 'lsp1'
2020-02-03T02:28:59.415Z|00008|ovn_northd|INFO|Assigned dynamic MAC address 'ea:8c:8d:a8:00:03' to port 'lsp1'

<==== mac is generated from mac prefix

2020-02-03T02:28:59.415Z|00009|ovn_northd|INFO|Assigned dynamic IPv6 address '2003::e88c:8dff:fea8:3' to port 'lsp1'

Comment 5 errata-xmlrpc 2020-03-10 10:08:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0752