This patch is also required for proper operation on a host with ixgbe:

commit 997134fb8b17eef6eba439303b382b239996208b
Author: Laine Stump <laine>
Date:   Mon Apr 17 10:50:48 2017 -0400

    util: allow ignoring SIOCSIFHWADDR when errno is EPERM

1) you can setup VFs on a PF without reloading the PF driver, just by issuing the following commands:


  echo 0 >/sys/class/net/${PFnetdev}/device/sriov_numvfs
  echo ${numvfs} >/sys/class/net/${PFnetdev}/device/sriov_numvfs

(replacing ${PFnetdev} with the network device name of the PF, and ${numvfs} with the number of VFs you want to be active.) If the current value of sriov_numvfs is 0, you don't need the first command. Note that any time you do this, all existing VFs will be destroyed, and then new VFs created in their place, so you can't use it to change the number of VFs while any existing VF is being used.

2) Experimenting a bit with ixgbe, I've found the following pieces of info:

* CREATING VFs by reloading the PF driver with max_vfs=n

As you've found, when I removed and reloaded the PF driver (ixgbe), although enp5s0f0 vf 0 had admin MAC 00:00:00:00:00:00, I was refused permission to change that VF's (enp5s16) own MAC:

  # ip link set enp5s16 addr 56:57:58:59:60:61
  RTNETLINK answers: Operation not permitted

I even tried setting "trust" to ON (this appears to be a feature of only the ixgbe driver):

  # ip link set enp5s0f0 vf 1 trust on

and it still failed (so apparently the setting of "trust" has nothing to do with granting the VF driver the ability to set its MAC address).

So even though the admin MAC was 0, meaning that it had apparently never been set (since it's not possible to set an admin MAC to 0, this can only be the initial value), something in either the VF or PF driver is preventing us from setting the VF's own MAC. This seems like a bug in the ixgbe driver (or else some "feature" that is inconsistent with other SRIOV drivers).

Since we are unable to set the VF's MAC, we then fall back to setting the admin MAC + rebinding the VF to its host net driver (which will reinitialize the VF's MAC from the admin MAC)


* CREATING THE VF DRIVER BY CHANGING /sys/class/net/$PFnetdev/devices/sriov_numvfs

On a freshly booted machine with 0 vfs, if I run:

  echo ${numvfs} >/sys/class/net/${PFnetdev}/device/sriov_numvfs

then 2 things are different:

   1) the admin MAC that shows up in "ip link show" output is *NOT* 0,
      but is instead the same random MAC that is shown for the VF itself.

   2) I *can* set the VF's own MAC directly:

      # ip link set enp5s16 addr 56:57:58:59:60:61
      #

When I conclude from this:


1) Don't set the number of VFs by reloading the ixgbe driver, set it by echoing the desired number of VFs to the PF's "sriov_numvfs".

2) The ixgbe drivers behave inconsistently depending on how their VF count has been set. Apparently the "administratively set" flag for all VFs is set ON for all VFs when the PF driver is loaded, and is only set OFF if the VF count is set via sysfs. In my opinion this inconsistency is a bug and should be reported against the ixgbe driver.

You problems with the secondary MACs on Mellanox are related to the fact that Mellanox has multiple ports (netdevs) on a single PCI address, whereas the libvirt code is written with the assumption that each PCI address corresponds to a single netdev and vice versa - we convert back and forth between netdev name and PCI address, and this most likely ends with us misaligning the PF <--> VF relationships.

There is a BZ for this (the description sounds different, but it is the same problem): Bug 1267191

Is this something that worked before? (it seems unlikely that it worked before, since we were always setting *only* the admin MAC, and never setting the VF MAC).

From Comment 36:
> The(1)and(2) are both as expected,right?

Yes, those are both expected. The Cisco PF driver doesn't support reporting or setting an "admin MAC", that's why you don't see any of the "vf ....." lines below the info for a PF in the output of "ip link show". And of course when you are doing vfio device assignment, the VF netdev no longer exists on the host, so you won't be able to see the VF's MAC address there either.

We have test the ixgbe, igb, mlx, cisco, Emulex five type NICs with the five scenarios as below:
1.Do the VF pci assignment with vfio
2.Do the macvtap passthrough with VF which has been used in VF pci assignment with vfio
3.Do the macvtap passthrough with VF which never has been used in VF pci assignment with vfio
4.Do the VF pci assignment with vfio which has been used in macvtap passthrough with VF
5.Detach VF from host and Reattach VF to host

And we do the regression testing with the test run as below：
https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/testrun?id=regression%20test%20for%20bug%201442040-ixgbe

The remaining issues are as below, they will be tracked in the bugs as below
1.On the ixgbe nic, the admin mac will be set as 02:00:00:00:00:00 after macvtap passthrough when the initialized admin mac is all zero, which is caused by the issue of ixgbe driver:
https://bugzilla.redhat.com/show_bug.cgi?id=1447549

2.On the mlx NIC, there are two issues as below.
   (1)The admin MAC address of VF does not be saved and will be changed everytime
   (2) the second mac of vf can not be saved and restored as expected.
https://bugzilla.redhat.com/show_bug.cgi?id=1415609#c30


According to the test as above， I change the status to verified

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1304