Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1781223

Summary: Dynamic MAC assignments made before generating MAC prefix
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Russell Bryant <rbryant>
Component: ovn2.11Assignee: Dumitru Ceara <dceara>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: RHEL 8.0CC: ctrautma, dceara, mmichels, tredaelli, william.caban, zzhao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1794671 (view as bug list) Environment:
Last Closed: 2020-03-10 10:07:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1794671    
Attachments:
Description Flags
backport to ovn2.11 branch, applies cleanly after other backports none

Description Russell Bryant 2019-12-09 14:59:21 UTC
Description of problem:

ovn-kubernetes makes use of dynamic IP and MAC assignment from OVN.  ovn-northd generates a random prefix for the MAC addresses it uses.  I noticed that on ovn-northd startup, it will assign MAC addresses before it has generated the prefix.  See the log below.  You'll see it generated 36 MAC addresses with a blank prefix before switching to the prefix it generated.

2019-12-09T14:39:08Z|00121|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00122|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:31' to port 'openshift-machine-config-operator_machine-config-controller-6cdf899474-rgxcm'
2019-12-09T14:39:08Z|00123|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00124|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:32' to port 'openshift-controller-manager_controller-manager-kcgp8'
2019-12-09T14:39:08Z|00125|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00126|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:33' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00127|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00128|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:34' to port 'openshift-authentication-operator_authentication-operator-647f9697ff-w6zzg'
2019-12-09T14:39:08Z|00129|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00130|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:35' to port 'openshift-service-catalog-controller-manager-operator_openshift-service-catalog-controller-manager-operator-57cfpzzpf'
2019-12-09T14:39:08Z|00131|ovn_northd|INFO|Assigned dynamic MAC address '00:00:00:00:00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:39:08Z|00132|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:200:ff:fe00:36' to port 'openshift-kube-apiserver_installer-2-ip-10-0-153-153.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00133|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:17Z|00134|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:a' to port 'openshift-kube-scheduler_revision-pruner-5-ip-10-0-141-106.us-east-2.compute.internal'
2019-12-09T14:40:18Z|00135|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:0c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:18Z|00136|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:c' to port 'openshift-controller-manager_controller-manager-h59fk'
2019-12-09T14:40:20Z|00137|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:20Z|00138|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::2:b8e1:4cff:fe00:13' to port 'openshift-kube-scheduler_installer-5-ip-10-0-161-127.us-east-2.compute.internal'
2019-12-09T14:40:21Z|00139|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:21Z|00140|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::3:b8e1:4cff:fe00:15' to port 'openshift-cluster-storage-operator_cluster-storage-operator-58bd68d66c-m8lg6'
2019-12-09T14:40:22Z|00141|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:06' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00142|ovn_northd|INFO|Assigned dynamic IPv6 address 'fd01::1:b8e1:4cff:fe00:6' to port 'openshift-apiserver_apiserver-grcsc'
2019-12-09T14:40:22Z|00143|ovn_northd|INFO|Assigned dynamic MAC address 'ba:e1:4c:00:00:16' to port 'openshift-apiserver_apiserver-8mqtx'


Version-Release number of selected component (if applicable):

ovn2.11-2.11.1-20.el7fdp.x86_64.rpm

This also has some patches applied to add additional logging, like the messages in this bug report.

How reproducible:

I've noticed the blank prefix MAC addresses every time I've looked into the addresses assigned by ovn-northd.

Comment 1 Russell Bryant 2019-12-09 16:15:47 UTC
Patch posted here: https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365776.html

Comment 2 Russell Bryant 2019-12-09 16:29:16 UTC
In my latest ovn-kubernetes testing, I noticed this issue occurring after an ovn-northd restart.  It appears that if you're running ovn-northd in active-active mode, the same MAC resetting would occur the first time a given instance of ovn-northd becomes the leader.

Comment 3 Russell Bryant 2019-12-09 18:00:35 UTC
Created attachment 1643383 [details]
backport to ovn2.11 branch, applies cleanly after other backports

I've attached a backport that applies cleanly after the other backports applied to ovn2.11.

Comment 7 Jianlin Shi 2020-02-03 02:13:12 UTC
as there is no log before the fixed version, so can't reproduce on old version.

Verified on ovn2.11.1-32:

#!/bin/bash                                                                                           
                                                                                                      
systemctl restart openvswitch                                                                         
systemctl restart ovn-northd                                                                          
ovn-nbctl set-connection ptcp:6641                                                                    
ovn-sbctl set-connection ptcp:6642                                                                    
                                                                                                      
ovs-vsctl set open . external-ids:system_id=hv0 external-ids:ovn-remote=tcp:20.0.30.26:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.30.26
                                                                                                      
systemctl restart ovn-controller                                                                      
                                                                                                      
ovn-nbctl ls-add ls1                                                                                  
ovn-nbctl lsp-add ls1 lsp1                                                                            
ovn-nbctl  set Logical-Switch ls1 other_config:subnet=192.168.0.0/16                                  
ovn-nbctl set Logical-switch ls1 other_config:ipv6_prefix=2003::0                                     
ovn-nbctl lsp-set-addresses lsp1 dynamic                                                              
ovn-nbctl list logical_switch_port lsp1 


[root@hp-dl380pg8-12 bz1781223]# rpm -qa | grep -E "ovn|openvswitch"
ovn2.11-central-2.11.1-32.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-2.11.1-32.el7fdp.x86_64
ovn2.11-host-2.11.1-32.el7fdp.x86_64
openvswitch2.11-2.11.0-35.el7fdp.x86_64

[root@hp-dl380pg8-12 bz1781223]# ovn-nbctl list nb_global
_uuid               : ef4b4f8f-4bba-46fb-a60e-ede49f9d7481
connections         : [2cf377e1-489e-49b0-b850-dfe4847b5c91]
external_ids        : {}
hv_cfg              : 0
ipsec               : false
nb_cfg              : 0
options             : {mac_prefix="ca:8e:0c", svc_monitor_mac="82:f8:6f:cf:3c:97"}
sb_cfg              : 0
ssl                 : []

log from ovn-northd:

[root@hp-dl380pg8-12 bz1781223]# cat /var/log/openvswitch/ovn-northd.log 
2020-02-03T02:07:29.758Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-northd.log
2020-02-03T02:07:29.760Z|00002|reconnect|INFO|unix:/run/openvswitch/ovnnb_db.sock: connecting...
2020-02-03T02:07:29.760Z|00003|reconnect|INFO|unix:/run/openvswitch/ovnsb_db.sock: connecting...
2020-02-03T02:07:29.760Z|00004|reconnect|INFO|unix:/run/openvswitch/ovnnb_db.sock: connected
2020-02-03T02:07:29.760Z|00005|reconnect|INFO|unix:/run/openvswitch/ovnsb_db.sock: connected
2020-02-03T02:07:29.760Z|00006|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.
2020-02-03T02:07:29.853Z|00007|ovn_northd|INFO|Assigned dynamic IPv4 address '192.168.0.2' to port 'lsp1'
2020-02-03T02:07:29.853Z|00008|ovn_northd|INFO|Assigned dynamic MAC address 'ca:8e:0c:a8:00:03' to port 'lsp1'

<=== mac is generated from mac prefix

2020-02-03T02:07:29.853Z|00009|ovn_northd|INFO|Assigned dynamic IPv6 address '2003::c88e:cff:fea8:3' to port 'lsp1'

Comment 9 errata-xmlrpc 2020-03-10 10:07:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0750