Bug 1474638

Summary: [SR-IOV] - Vdsm doesn't report VFs with zero MAC addresses 00:00:00:00:00:00 - bnx2x driver
Product: [oVirt] vdsm Reporter: Michael Burman <mburman>
Component: CoreAssignee: Edward Haas <edwardh>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: high    
Version: 4.19.23CC: bugs, danken, ipilcher, myakove, pbrilla, roxenham
Target Milestone: ovirt-4.1.5Flags: rule-engine: ovirt-4.1+
rule-engine: exception+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-23 08:03:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1476155    

Description Michael Burman 2017-07-25 06:01:47 UTC 
Description of problem:
[SR-IOV] - Vdsm doesn't report VFs with zero MAC addresses 00:00:00:00:00:00 -  bnx2x driver.
And such VFs are not visible in the rhv-m UI. 

-Not all the devices with zero mac should be reported by vdsm, but we should start reporting VFs with zero MAC addresses for the bnx2x drivers, so they can be used in the rhv-m UI.

Version-Release number of selected component (if applicable):
vdsm-4.19.23

How reproducible:
100%
Comment 1 Rhys Oxenham 2017-07-25 06:51:51 UTC 
We have bnx2 based machines with this very problem...

[root@node08 ~]# ethtool -i eno2
driver: bnx2x
version: 1.712.30-0
firmware-version: FFV08.07.25 bc 7.13.54
expansion-rom-version:
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[root@node08 ~]# cat /sys/class/net/eno2/device/sriov_numvfs
16

[root@node08 ~]# cat /proc/cmdline
BOOT_IMAGE=/rhvh-4.1-0.20170706.0+1/vmlinuz-3.10.0-514.26.1.el7.x86_64 root=/dev/rhvh_node08/rhvh-4.1-0.20170706.0+1 ro crashkernel=auto rd.lvm.lv=rhvh_node08/swap rd.lvm.lv=rhvh_node08/rhvh-4.1-0.20170706.0+1 biosdevname=0 intel_iommu=on LANG=en_US.UTF-8 img.bootid=rhvh-4.1-0.20170706.0+1

(Note intel_iommu=on)

We had to manually set the VF's MAC addresses to something other than 00:00:00:00:00:00 for them to show up in the RHV-M UI, e.g.

$ counter=1; for i in $(ip a | grep enp1s | awk '{print $2;}' | tr -d ":"); do ip link set $i addr aa:bb:cc:dd:ee:$counter; ((counter++)); done

This gives us...

5: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether f8:ca:b8:91:1c:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC aa:bb:cc:dd:ee:01, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 1 MAC aa:bb:cc:dd:ee:02, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 2 MAC aa:bb:cc:dd:ee:03, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 3 MAC aa:bb:cc:dd:ee:04, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 4 MAC aa:bb:cc:dd:ee:05, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 5 MAC aa:bb:cc:dd:ee:06, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 6 MAC aa:bb:cc:dd:ee:07, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 7 MAC aa:bb:cc:dd:ee:08, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 8 MAC aa:bb:cc:dd:ee:09, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 9 MAC aa:bb:cc:dd:ee:10, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 10 MAC aa:bb:cc:dd:ee:11, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 11 MAC aa:bb:cc:dd:ee:12, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 12 MAC aa:bb:cc:dd:ee:13, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 13 MAC aa:bb:cc:dd:ee:14, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 14 MAC 00:1a:4a:16:01:59, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 15 MAC aa:bb:cc:dd:ee:16, vlan 1, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto

Note that VF 14 was in-use by a RHV VM in the above output.

At this point, the VF's show up in the RHV-M UI, and we can allocate these to RHV networks based on vNIC profile being set to passthrough.
Comment 2 Rhys Oxenham 2017-07-25 06:54:59 UTC 
Problem witnessed with the following setup...

[root@node08 ~]# nodectl info
layers:
  rhvh-4.1-0.20170706.0:
    rhvh-4.1-0.20170706.0+1
bootloader:
  default: rhvh-4.1-0.20170706.0+1
  entries:
    rhvh-4.1-0.20170706.0+1:
      index: 0
      title: rhvh-4.1-0.20170706.0
      kernel: /boot/rhvh-4.1-0.20170706.0+1/vmlinuz-3.10.0-514.26.1.el7.x86_64
      args: "ro crashkernel=auto rd.lvm.lv=rhvh_node08/swap rd.lvm.lv=rhvh_node08/rhvh-4.1-0.20170706.0+1 biosdevname=0 intel_iommu=on LANG=en_US.UTF-8 img.bootid=rhvh-4.1-0.20170706.0+1"
      initrd: /boot/rhvh-4.1-0.20170706.0+1/initramfs-3.10.0-514.26.1.el7.x86_64.img
      root: /dev/rhvh_node08/rhvh-4.1-0.20170706.0+1
current_layer: rhvh-4.1-0.20170706.0+1

[root@node08 ~]# rpm -qa | egrep '(ovirt|rhv|vdsm|libvirt)'
ovirt-host-deploy-1.6.6-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
vdsm-xmlrpc-4.19.20-1.el7ev.noarch
libvirt-daemon-2.0.0-10.el7_3.9.x86_64
libvirt-daemon-driver-storage-2.0.0-10.el7_3.9.x86_64
libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.9.x86_64
libvirt-daemon-driver-interface-2.0.0-10.el7_3.9.x86_64
vdsm-hook-vmfex-dev-4.19.20-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.4-1.el7ev.noarch
vdsm-client-4.19.20-1.el7ev.noarch
ovirt-setup-lib-1.1.3-1.el7ev.noarch
vdsm-hook-openstacknet-4.19.20-1.el7ev.noarch
python-ovirt-engine-sdk4-4.1.5-1.el7ev.x86_64
vdsm-yajsonrpc-4.19.20-1.el7ev.noarch
ovirt-node-ng-nodectl-4.1.3-0.20170608.1.el7.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-python-4.19.20-1.el7ev.noarch
vdsm-cli-4.19.20-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.9.x86_64
libvirt-python-2.0.0-2.el7.x86_64
libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.9.x86_64
libvirt-daemon-driver-qemu-2.0.0-10.el7_3.9.x86_64
libvirt-daemon-driver-secret-2.0.0-10.el7_3.9.x86_64
libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.9.x86_64
vdsm-hook-vhostmd-4.19.20-1.el7ev.noarch
libvirt-daemon-kvm-2.0.0-10.el7_3.9.x86_64
vdsm-4.19.20-1.el7ev.x86_64
vdsm-gluster-4.19.20-1.el7ev.noarch
vdsm-hook-fcoe-4.19.20-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.3.3-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
vdsm-jsonrpc-4.19.20-1.el7ev.noarch
libvirt-daemon-driver-network-2.0.0-10.el7_3.9.x86_64
vdsm-api-4.19.20-1.el7ev.noarch
libvirt-lock-sanlock-2.0.0-10.el7_3.9.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
vdsm-hook-ethtool-options-4.19.20-1.el7ev.noarch
cockpit-ovirt-dashboard-0.10.7-0.0.20.el7ev.noarch
Comment 4 Yaniv Kaul 2017-08-06 08:17:39 UTC 
Is this on track to 4.1.5?
Comment 5 rhev-integ 2017-08-10 17:53:17 UTC 
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Open patch attached]

For more info please contact: infra
Comment 6 Dan Kenigsberg 2017-08-10 21:23:38 UTC 
We somehow missed merging the patch to the stable branch, even though it was ready before Wednesday's build.

I've merged the tiny patch it now, and would be available in next the build.
Comment 7 Michael Burman 2017-08-16 10:46:55 UTC 
Verified on - vdsm-4.19.28-1.el7ev.x86_64 using - Dell PowerEdge FC430

bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 
(2014/02/10)

01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
                Product Name: QLogic 57810 10 Gigabit Ethernet
        Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
Comment 10 Pavol Brilla 2017-10-26 07:57:59 UTC 
Cleaning flag, it was not attached to erratum, now is too late
Comment 11 Dan Kenigsberg 2017-10-26 10:40:15 UTC 
That's unfortunate. For the record, this bug is solved in code included in https://errata.devel.redhat.com/advisory/29585