RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1415609 - libvirt fails to remove guest mac from VFs of certain drivers
Summary: libvirt fails to remove guest mac from VFs of certain drivers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 7.4
Assignee: Laine Stump
QA Contact: Jingjing Shao
URL:
Whiteboard:
: 1374384 (view as bug list)
Depends On: 1341248 1401837
Blocks: 1442040
TreeView+ depends on / blocked
 
Reported: 2017-01-23 09:00 UTC by Dan Kenigsberg
Modified: 2021-09-24 05:39 UTC (History)
13 users (show)

Fixed In Version: libvirt-3.2.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1442040 (view as bug list)
Environment:
Last Closed: 2017-08-01 17:21:45 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1341248 0 unspecified CLOSED igb driver forbids resetting VF MAC address back to 00:00:00:00:00:00, which was its original value 2021-09-24 05:53:45 UTC
Red Hat Bugzilla 1372944 0 unspecified CLOSED libvirt doesn't set the mac on VF's netdev name 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1401837 0 medium CLOSED MAC address of VF is not reset by libvirt since ixgbe driver does not accept 00:00:00:00:00 2021-09-24 05:53:43 UTC
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Internal Links: 1341248 1372944 1401837

Description Dan Kenigsberg 2017-01-23 09:00:19 UTC 
Description of problem:
When passing a VF to a guest, libvirt sets its MAC according to the domain xml. When the VF is later released back to the host, libvirt attempts to restore its original MAC, but some drivers do not allow to set it back to all-zero mac.

libvirt then keeps the guest MAC on the VF, which can cause connectivity problems if that MAC is later assigned to another VF and the VM is restarted.

libvirt could use a random MAC instead, or 00:00:00:01

See the driver bugs:
Bug 1401837
Bug 1341248
Comment 1 Jingjing Shao 2017-02-09 08:57:31 UTC 
Hi laine,

I have two questions about this issue, May I have you take some time to check them and give some feedback about them? Thank you in advance.

1, According to the driver bug : https://bugzilla.redhat.com/show_bug.cgi?id=1341248 

I try to reproduce this issue with igb drvier.

But I find that the VF MAC addresses are common address not  00:00:00:00:00:00.
such as :
# ip l
8: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC ee:6e:7d:c5:c8:c5, spoof checking on, link-state auto
    vf 1 MAC 3e:0f:24:b2:16:4f, spoof checking on, link-state auto
    vf 2 MAC 4e:53:17:d5:81:f5, spoof checking on, link-state auto
...
12: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:19 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 46:93:7e:f0:f6:e8, spoof checking on, link-state auto
    vf 1 MAC 06:f7:1a:ed:1c:8b, spoof checking on, link-state auto
    vf 2 MAC 52:54:00:6d:90:02, spoof checking on, link-state auto

I also check the ixgbe driver, it is in the same situation.

So I can not reproduce this issue, so I want to know what situation is that the mac will be 00:00:00:00:00:00.


2, I find another issue that the two vfs can share the same mac address on the one host.  Is it a bug? the steps are as below.

(1) # virsh dumpxml vm1 | grep interface -A 5
<interface type='hostdev' managed='yes'>
      <mac address='52:54:00:6d:90:02'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
      <vlan>
        <tag id='24'/>
      </vlan>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

 # virsh dumpxml vm2 | grep interface -A 5
 <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:6d:90:02'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
      <vlan>
        <tag id='24'/>
      </vlan>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


(2)#  virsh start vm1
Domain vm1 started

# virsh start vm2
Domain vm2 started


(3) check the mac address
 # ip l
8: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 52:54:00:6d:90:02, vlan 24, spoof checking on, link-state auto
    vf 1 MAC 3e:0f:24:b2:16:4f, spoof checking on, link-state auto
    vf 2 MAC 4e:53:17:d5:81:f5, spoof checking on, link-state auto
...
12: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:19 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 52:54:00:6d:90:02, vlan 24, spoof checking on, link-state auto
    vf 1 MAC 06:f7:1a:ed:1c:8b, spoof checking on, link-state auto
    vf 2 MAC e6:f8:ac:88:96:c2, spoof checking on, link-state auto
Comment 2 Laine Stump 2017-02-10 16:48:44 UTC 
Is your host running RHEL6 or RHEL7?

On RHEL6 the igb driver behaves as you report - it initializes each VF with a random MAC address at boot time. But on RHEL7, it initializes each VF MAC to 00:00:00:00:00:00. (I don't have a machine with hardware that uses ixgbe, but last time I looked the VF MACs were all initialized to 0 at least on RHEL7)

As for item (2) (two guests sharing the same MAC address), there are valid scenarios where two machines could have the same MAC address for their interfaces, so libvirt can't forbid it.
Comment 3 Laine Stump 2017-02-12 19:23:12 UTC 
I found the Mellanox bug related to this: Bug 1302166. It has already been fixed.

It's my opinion that the problem should be fixed at the source, i.e. the drivers. Any driver that initializes its "administrative " MAC addresses for VFs to all 0 should also allow setting them to all 0. If we add a workaround in the middle of the stack, it's going to be there forever, creating maintenance headaches

Here is the current score for SRIOV-capable drivers that I know about:

enic - allows 0 (never broken)
mlx  - allows 0 (fixed by patch in Bug 1302166)
igb  - doesn't allow 0 (Bug 1341248)
ixgbe - doesn't allow 0 (Bug 1401837)

Corinna already had a patch to allow 0 MAC on igb for Bug 1341248, but the discussion got mixed up with figuring out a way to turn off the "administratively set" bit (which is a separate, but equally troubling, problem), I didn't have time to look into what she was suggesting, and the BZ languished. Especially with the precedent of enic and mlx allowing a setting of 0, it shouldn't be too difficult to get the kernel people to fix igb and ixgbe, eliminating the need for an ugly hack workaround in libvirt or vdsm/ovirt.
Comment 4 Jingjing Shao 2017-02-13 06:04:08 UTC 
(In reply to Laine Stump from comment #2)
> Is your host running RHEL6 or RHEL7?
> 
> On RHEL6 the igb driver behaves as you report - it initializes each VF with
> a random MAC address at boot time. But on RHEL7, it initializes each VF MAC
> to 00:00:00:00:00:00. (I don't have a machine with hardware that uses ixgbe,
> but last time I looked the VF MACs were all initialized to 0 at least on
> RHEL7)

My host is RHEL7, but I find the mac addresses are all random MAC address not 00:00:00:00:00:00. Is there something wrong ?

# uname -r
3.10.0-558.el7.x86_64
# rpm -q libvirt
libvirt-3.0.0-1.el7.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64

#ip l | grep  ens1f0 -A4
8: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC ee:6e:7d:c5:c8:c5, spoof checking on, link-state auto
    vf 1 MAC 3e:0f:24:b2:16:4f, spoof checking on, link-state auto
    vf 2 MAC 4e:53:17:d5:81:f5, spoof checking on, link-state auto

# virsh dumpxml vm1 | grep interface -A9
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:6d:90:02'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
      <vlan>
        <tag id='24'/>
      </vlan>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


# virsh start vm1
Domain vm1 started

# ip l | grep  ens1f0 -A4                
8: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 52:54:00:6d:90:02, vlan 24, spoof checking on, link-state auto
    vf 1 MAC 3e:0f:24:b2:16:4f, spoof checking on, link-state auto
    vf 2 MAC 4e:53:17:d5:81:f5, spoof checking on, link-state auto

# virsh destroy vm1
Domain vm1 destroyed

# ip l | grep  ens1f0 -A4
8: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC ee:6e:7d:c5:c8:c5, spoof checking on, link-state auto
    vf 1 MAC 3e:0f:24:b2:16:4f, spoof checking on, link-state auto
    vf 2 MAC 4e:53:17:d5:81:f5, spoof checking on, link-state auto



> As for item (2) (two guests sharing the same MAC address), there are valid
> scenarios where two machines could have the same MAC address for their
> interfaces, so libvirt can't forbid it.

OK, thank you for your reply.
Comment 5 Martin Tessun 2017-02-13 09:24:38 UTC 
Laine,

still if this might get broken in other drivers as well, could we not move forward with this as follows:

1. Try to reset the MAC of the VF to its previous value once it is released by libvirt
2. If that does fail (due to the above mentioned bugs), could we just try to set it to a "specific value" like 00:00:00:00:00:01

Another approach would be to just set the released MAC addresses by default to a predefined value (like step (2) above).

Also an option would be, in case the previous MAC address has been all zeroes, we should change this saved value already to a "valid" MAC address.

The idea behind this is to have a "more stable API" that does behave correctly, even if some network drivers don't behave correctly. As there probably will always be a bug in some VF network driver, we should just avoid them, if possible.

Thanks!
Martin
Comment 6 Corinna Vinschen 2017-02-13 10:01:23 UTC 
We can "fix" the existing drivers so they all allow resetting the
administrative MAC to 00:00:00:00:00:00, so far so good.

The problem I have with this is that this is, in fact, a policy thingy.

If libvirt/vfio/whatever relies on this behaviour it will be
potentially broken with new drivers, unless you enforce this policy
from the start.  And it will potentially break with out-of-tree
drivers.

So I wonder.  Shouldn't a long-term solution be resilient against this
ability to set the admin MAC back to all zero, rather than trying to
enforce it on the driver level?


Corinna
Comment 7 Corinna Vinschen 2017-02-13 10:02:44 UTC 
Oops.  Looks like I'm basically just repeating what Martin wrote in comment 5.


Sorry,
Corinna
Comment 8 Laine Stump 2017-02-27 15:33:44 UTC 
Jingjing - I can't explain that. On my RHEL7 system, the admin MAC addresses are all initialized to 00:00:00:00:00:00 by the igb PF driver. Maybe look at the output of uname -a to see what kernel you're running?

Martin - due to a combination of libvirt's code doing "the wrong thing" wrt to the MAC address for macvtap passthrough mode (ie I don't know how macvtap passthrough can possibly work for *anyone* except 802.11QbX users in its current state), and my apparent inability to explain/convince that it should be possible to restore both the VF MAC *and* the admin MAC for the VF that's stored in the PF back to their original states without needing to reload the PF driver, I'm currently working on a complete overhaul of how libvirt manages the MAC addresses of SRIOV VFs both for VFIO assignment and macvtap passthrough mode.

Once that's done, it won't matter if setting the admin MAC address to 00:00:00:00:00:00 fails (since the important thing is what the VF's active MAC address is set to). However, I will still be proposing a patch that attempts setting the MAC to 02:00:00:00:00:00 if the desired value was 00:00:00:00:00:00 and it failed, mainly for asthetic reasons. This should be in upstream shortly after the 3.1 release, and will be easily backportable to the RHEL version.
Comment 9 Laine Stump 2017-03-10 20:37:58 UTC 
These patches should make everybody happy:

 https://www.redhat.com/archives/libvir-list/2017-March/msg00455.html

(Just posted, waiting on upstream review)
Comment 11 Laine Stump 2017-03-21 15:52:31 UTC 
There are 2 MAC addresses associated with each VF:

1) the VF's own MAC address which it is currently using. This will be a part of the net_* device name for the VF shown in the output of virsh nodedev-list, and also will show up in the output of "ip link show $vfname". In your example, there is a device in the nodedev-list that is named:

   net_enp134s16_9a_8f_97_85_0f_61

If you run "ip link show enp134s16" you will see that the MAC address listed is "9a:8f:97:85:0f:61" (or it *would have been* if you'd done that at the time you got the output shown above - once you reboot the host, the VF will have a different random MAC address).

2) the "admin MAC", which is simply stored in a table in the *PF* driver, and shows up on the lines starting with "vf" when you run "ip link show $pfname". This MAC address is what the VF will have as its initial MAC driver *the next time* a VF net driver is initialized for that VF (e.g., if you bind it to vfio-pci then assign it to a guest - the guest will load its driver for the device, which will cause the MAC address to be initialized from the "admin MAC" that's stored in the PF; likewise when the VF is bound back to the host's VF net driver - the initial MAC of the VF will be whatever had been set as the "admin MAC" for that VF in the PF.


I'm not seeing any "error in the PF info". To what are you referring?
Comment 12 Jingjing Shao 2017-03-22 04:50:03 UTC 
(In reply to Laine Stump from comment #11)
> There are 2 MAC addresses associated with each VF:

Thanks for your detailed explanation. The "00:00:00:00:00:00" is the "admin mac" address the patches mentioned.


> 
> I'm not seeing any "error in the PF info". To what are you referring?

I mean this part

86:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Physical Slot: 2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
....

***  pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable  ***
....

**Is there some errors caused the mac address "00:00:00:00:00:00"?***



PS: In the igb system, I can not find this error

# ip l 
...
68: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:39:8b:18 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 9a:68:3a:09:4e:cb, spoof checking on, link-state auto
    vf 1 MAC da:23:1c:9e:cc:52, spoof checking on, link-state auto


# lspci -vvv | grep 03:00.0 -A60
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Physical Slot: 1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 30
	Region 0: Memory at f4800000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at f4000000 (32-bit, non-prefetchable) [size=4M]
	Region 2: I/O ports at d000 [size=32]
	Region 3: Memory at f4840000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-39-8b-18
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 2, Function Dependency Link: 00
		VF offset: 128, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000f4848000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000f4868000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: igb
	Kernel modules: igb
Comment 13 Laine Stump 2017-03-22 17:45:35 UTC 
> ***  pcilib: sysfs_read_vpd: read failed: Input/output error
>		Not readable  ***
> ....
>
> **Is there some errors caused the mac address "00:00:00:00:00:00"?***


I've never seen that error and don't know where it is coming from (possibly the people who know about the driver for that card might have an idea?), but you can be fairly certain that nothing in the output of "lspci" would be affected in any way by the setting of a device's admin MAC (or its own MAC, or anything else net-related).
Comment 14 Laine Stump 2017-03-27 02:09:44 UTC 
*** Bug 1374384 has been marked as a duplicate of this bug. ***
Comment 15 Laine Stump 2017-03-27 04:29:08 UTC 
Half of the required patches have been pushed, the other half are waiting on ACK of a single patch. Here are the ones that have been pushed already:


commit 176229dd05a6c96dec2b7b1c2fb0143d7331b10e
Author: Laine Stump <laine>
Date:   Fri Feb 17 14:28:55 2017 -0500

    util: permit querying a VF MAC address or VLAN tag by itself

commit 0a583c26f766d973c08d05ddf0b8fb297eb10f4f
Author: Laine Stump <laine>
Date:   Sun Feb 19 22:06:33 2017 -0500

    util: remove unused args from virNetDevSetVfConfig()

commit 19c5db749cdc3ddf53c2b23d5803a0f17cc5077a
Author: Laine Stump <laine>
Date:   Sun Feb 26 13:27:48 2017 -0500

    util: use cleanup label consistently in virHostdevNetConfigReplace()

commit 606a01339563b5371f21a9b2c10ef818c0557fb1
Author: Laine Stump <laine>
Date:   Sun Mar 5 17:32:15 2017 -0500

    util: eliminate useless local variable

commit 30b07a425d7b5b3ac7ca68d3574692ac6415ccbe
Author: Laine Stump <laine>
Date:   Wed Mar 8 15:27:05 2017 -0500

    util: make virMacAddrParse more versatile

commit d6ee56d723761d408eeef32a96e3c118e0e742c2
Author: Laine Stump <laine>
Date:   Fri Mar 3 11:54:59 2017 -0500

    util: change virPCIGetNetName() to not return error if device has no net name

commit 9a238c16b382f3366bf474a648ff0e4b777b0af8
Author: Laine Stump <laine>
Date:   Tue Mar 7 14:23:01 2017 -0500

    util: make virPCIGetDeviceAddressFromSysfsLink() public

commit 251d179bf2c10c7d7fb594c68f1aac56a9b0d932
Author: Laine Stump <laine>
Date:   Tue Mar 7 12:58:15 2017 -0500

    util: new function virPCIDeviceRebind()

commit f4ef3a71f8abc83771952604d309b39ca3ea64cf
Author: Laine Stump <laine>
Date:   Thu Mar 9 14:04:16 2017 -0500

    util: new internal function to permit silent failure of virNetDevSetMAC()

commit 554253ad04463e0442030f4630f0a4a02613a595
Author: Laine Stump <laine>
Date:   Thu Mar 2 14:55:01 2017 -0500

    util: new function virNetDevPFGetVF()

Those yet to push are here:

  https://www.redhat.com/archives/libvir-list/2017-March/msg01125.html
Comment 16 Laine Stump 2017-03-27 19:18:14 UTC 
The rest of the patches are pushed upstream. The entire series will be present in libvirt-3.2.0:

commit 26694daf0997290e4a02fd10b1c2e24678ca8653
Author: Laine Stump <laine>
Date:   Mon Feb 20 16:14:53 2017 -0500

    util: new functions virNetDev(Save|Read|Set)NetConfig()

commit b91a33638476cf57d910b6056a8fc11921edd029
Author: Laine Stump <laine>
Date:   Sun Mar 5 17:43:39 2017 -0500

    util: use new virNetDev*NetConfig() functions for macvtap setup/teardown

commit 9c004d55d02422b8eaa6db857372bdd4af3e26ed
Author: Laine Stump <laine>
Date:   Sun Mar 5 20:28:08 2017 -0500

    util: use new virNetDev*NetConfig() functions for hostdev setup/teardown

commit b684734bef7d475a5d3f20acf6d46c4a34e77d63
Author: Laine Stump <laine>
Date:   Tue Feb 28 13:47:12 2017 -0500

    util: replace virHostdevNetConfigReplace with ...(Save|Set)NetConfig()

commit cceada574e7b057c82927f7939c7491488e67463
Author: Laine Stump <laine>
Date:   Tue Feb 28 16:44:26 2017 -0500

    util: save hostdev network device config before unbinding from host driver

commit d6ef331f112ab51af7d14c59cd0b460c2e9609a8
Author: Laine Stump <laine>
Date:   Mon Mar 6 09:36:40 2017 -0500

    util: after hostdev assignment, restore VF MAC address via setting admin MAC

commit bc4168f3e199a9167da544dfaa5847573a5f5d6b
Author: Laine Stump <laine>
Date:   Mon Mar 6 10:01:02 2017 -0500

    util: remove unused functions from virnetdev.c

commit d5f4abefc231a71e21e33e0d2eb4da8f22552c70
Author: Laine Stump <laine>
Date:   Mon Mar 6 13:59:29 2017 -0500

    util: if setting admin MAC to 00:00:00:00:00:00 fails, try 02:00:00:00:00:00

commit 86556e167a634599ac9975bd5dad98e3ef9f7017
Author: Laine Stump <laine>
Date:   Tue Mar 7 14:24:37 2017 -0500

    util: try *really* hard to set the MAC address of an SRIOV VF

commit 6ec36b06999cad155b75ed07ef656905b8a45180
Author: Laine Stump <laine>
Date:   Sat Mar 11 10:29:29 2017 -0500

    util: log all setting of MAC addresses and vlan tags
    

commit 43da69158249318a7127ddca6a1813f03b18259c
Author: Laine Stump <laine>
Date:   Mon Mar 13 19:07:02 2017 -0400

    util: rename virHostdevNetConfigRestore() to virHostdevRestoreNetConfig()
Comment 18 Jingjing Shao 2017-04-13 03:47:37 UTC 
Hi laine,

I go through the patches and  try to verified this bug and do the test as below.
But I have some questions about this bug, may I get your help to check them ?

scenario 1， After pci assignment with vfio, the mac address keep the random address, shouldn't it be the admin mac address?
scenario 2， Use the VF which has been used in vfio attachment to guest with macvtap. The VF mac address get another random address,shouldn't it be the origin VF mac address? 
scenario 3， Use the VF which has not  been used in vfio attachment to guest with macvtap, the mac address is 00:00:00:00:00:00 and get errors. Is there something wrong ?
scenario 4， Detach the VF from host and reattach it back to host. It also get a random mac address, Is it expected ? 


# rpm -q libvirt
libvirt-3.2.0-2.virtcov.el7.x86_64


Get the admin MAC “00:00:00:00:00:00”
# ip l
..
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
...
p7p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a5 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off


Get the vf mac address
# virsh nodedev-list --tree

 +- pci_0000_00_02_2
  |   |
  |   +- pci_0000_05_00_0
  |   |   |
  |   |   +- net_p7p1_90_e2_ba_11_d7_a4
  |   |     
  |   +- pci_0000_05_00_1
  |   |   |
  |   |   +- net_p7p2_90_e2_ba_11_d7_a5
  |   |     
  |   +- pci_0000_05_10_0
  |   |   |
  |   |   +- net_p7p3_52_7c_b7_d2_e1_48
  |   |     
  |   +- pci_0000_05_10_1
  |   |   |
  |   |   +- net_eth2_ea_84_22_df_0f_49
  |   |     
  |   +- pci_0000_05_10_2
  |   |   |
  |   |   +- net_eth0_1a_90_97_4e_1a_85
  |   |     
  |   +- pci_0000_05_10_3
  |       |
  |       +- net_eth3_ce_03_75_9a_20_22


scenario 1， Pci assignment with vfio

（1）Attach a VF to guest

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     jishao                         running


# cat hostdev.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
 <source>
  <address domain='0x0' bus='0x05' slot='0x10' function='0x0'/>
 </source>
</hostdev>


# virsh attach-device jishao hostdev.xml 
Device attached successfully

# virsh dumpxml jishao | grep  hostdev -A8
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>


（2）The VF attach to vfio and get the random address
# ip l
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
==> vf 0 MAC 1a:f9:06:58:ab:da, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off


login the guest 

# ip l
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 1a:f9:06:58:ab:da brd ff:ff:ff:ff:ff:ff


(3) Destroy the guest, the vf reattach to host network driver and check the mac address,
    *****The mac address still the random address not the admin mac address*****

# virsh destroy jishao
Domain jishao destroyed

# ip l
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 1a:f9:06:58:ab:da, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off

# virsh nodedev-list --tree 
...
 +- pci_0000_00_02_2
  |   |
  |   +- pci_0000_05_00_0
  |   |   |	
  |   |   +- net_p7p1_90_e2_ba_11_d7_a4
  |   |     
  |   +- pci_0000_05_00_1
  |   |   |
  |   |   +- net_p7p2_90_e2_ba_11_d7_a5
  |   |     
  |   +- pci_0000_05_10_0
  |   |   |
  |   |   +- net_p7p1_0_1a_f9_06_58_ab_da
  |   |     
  |   +- pci_0000_05_10_1
...


scenario 2， Use the VF which has been used in vfio attachment to guest with macvtap
***The VF mac address get another random address, but on the host still is the first random address***

# cat macvtap.xml 
 <network>
<name>direct-pool</name>
<uuid>a3a19e70-c92b-8f6a-5bc1-85422f29b955</uuid>
<forward dev='p7p1_0' mode='passthrough'>
<interface dev='p7p1_0'/>
</forward>
</network>

# virsh net-define macvtap.xml 
Network direct-pool defined from macvtap.xml

# virsh net-start direct-pool
Network direct-pool started

# cat macv_attach.xml
<interface type='network'>
<source network='direct-pool'/>
</interface>

# virsh start jishao
Domain jishao started

# virsh dumpxml jishao | grep interface -A8
 <interface type='direct'>
      <mac address='52:54:00:72:ac:52'/>
      <source network='direct-pool' dev='p7p1_0' mode='passthrough'/>
      <target dev='macvtap0'/>
      <model type='rtl8139'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
 </interface>

# ip l
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
==> vf 0 MAC 1a:f9:06:58:ab:da, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off

# virsh nodedev-list --tree
 |   |     
  |   +- pci_0000_05_10_0
  |   |   |
  |   |   +- net_p7p1_0_1a_f9_06_58_ab_da
  |   |     


# virsh destroy jishao
Domain jishao destroyed




scenario 3， Use the VF which has not  been used in vfio attachment to guest with macvtap
***The VF mac address get the errors info ***

# ip l
349: p7p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a5 brd ff:ff:ff:ff:ff:ff
==> vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off


# virsh nodedev-list --tree 
  |   +- pci_0000_05_10_0
  |   |   |
  |   |   +- net_p7p1_0_1a_f9_06_58_ab_da
  |   |     
  |   +- pci_0000_05_10_1
  |   |   |
  |   |   +- net_eth2_ea_84_22_df_0f_49

# cat macvtap.xml 
 <network>
<name>direct-pool</name>
<uuid>a3a19e70-c92b-8f6a-5bc1-85422f29b955</uuid>
<forward dev='eth2' mode='passthrough'>
<interface dev='eth2'/>
</forward>
</network>

# virsh net-define macvtap.xml 
Network direct-pool defined from macvtap.xml

# virsh net-start direct-pool
Network direct-pool started

# virsh attach-device jishao macv_attach.xml 
error: Failed to attach device from macv_attach.xml
error: Cannot set interface MAC to 52:54:00:15:5c:66 on 'eth2': Operation not permitted


scenario 4，Detach the VF from host and reattach it back to host. 
***It also get a random mac address***

# ip l
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 1a:f9:06:58:ab:da, spoof checking on, link-state auto, trust off, query_rss off
==> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off


  |   +- pci_0000_05_10_2
  |   |   |
  |   |   +- net_p7p1_1_1e_87_56_22_f2_89
  |   |     

# virsh nodedev-reattach pci_0000_05_10_2
Device pci_0000_05_10_2 re-attached

# ip l
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 1a:f9:06:58:ab:da, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off


# virsh nodedev-reattach pci_0000_05_10_2
Device pci_0000_05_10_2 re-attached


# ip l
346: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 1a:f9:06:58:ab:da, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 1e:87:56:22:f2:89, spoof checking on, link-state auto, trust off, query_rss off
Comment 23 Laine Stump 2017-04-24 01:42:49 UTC 
This patch is also required for proper operation on a host with ixgbe:

commit 997134fb8b17eef6eba439303b382b239996208b
Author: Laine Stump <laine>
Date:   Mon Apr 17 10:50:48 2017 -0400

    util: allow ignoring SIOCSIFHWADDR when errno is EPERM
Comment 25 Laine Stump 2017-04-24 14:27:24 UTC 
Before anything else - what have you done to your machines to cause their network devices to have names like p7p3 and eth2 instead of (e.g.) enp3s5f2?

(In reply to Jingjing Shao from comment #18)
> Hi laine,
> 
> I go through the patches and  try to verified this bug and do the test as
> below.
> But I have some questions about this bug, may I get your help to check them ?
> 
> scenario 1， After pci assignment with vfio, the mac address keep the random
> address, shouldn't it be the admin mac address?

In "scenario 1" you are using "<hostdev>" rather than "<interface type='hostdev'>". libvirt doesn't do *anything* with any MAC address when you use plain <hostdev> (it doesn't even know that it's a network device), so what you see is expected. If you want to manage the MAC address of the VF, use <interface type='hostdev'>.


> scenario 2， Use the VF which has been used in vfio attachment to guest with
> macvtap. The VF mac address get another random address,shouldn't it be the
> origin VF mac address? 

This is the result of assigning the VF to the guest using <hostdev>, followed by the guest (?) setting the strange MAC address. Somehow that MAC was put into the admin MAC for the VF, as can be seen in the "ip l" output in step 3 of scenario 1.

> scenario 3， Use the VF which has not  been used in vfio attachment to guest
> with macvtap, the mac address is 00:00:00:00:00:00 and get errors. Is there
> something wrong ?

Arte you using a device that uses the ixgbe driver? retest with the patch in Comment 23.

> scenario 4， Detach the VF from host and reattach it back to host. It also
> get a random mac address, Is it expected ?

If the device never has its admin MAC set, then the admin MAC will be 00:00:00:00:00:00; if you detach/reattach it, the VF will be given a random MAC address. The nodedev detach/reattach commands deal with PCI devices, and know nothing of MAC addresses. The saving/restoring of MAC addresses happens only when you attach/detach a device that is <interface type='hostdev'> or <interface type='macvtap'>

TL;DR - get the new build when it is available, and test using <interface>, *not* using <hostdev> or with the nodedev-detach/reattach commands.
Comment 30 Laine Stump 2017-05-10 05:51:19 UTC 
Looking on a system with a mellanox (mlx4 driver) dual port SRIOV card, I can see why the mac addresses aren't restored properly when a PF has multiple ports / netdevs. libvirt's config will contain the netdev name of the VF, and libvirt derives the PF from the VF by following links in sysfs. For example, if you have a VF named p7p1_1, you would look in

    /sys/class/net/p7p1_1/device/physfn/net

where you would find a link to the PF. The problem is that, the "device" oart if the path is actually a link to a PCI device which is shared by another VF netdev, p7p2_1, and the "physfn" part of the path is also a link to a PCI device that is shared by two PF netdevs:

    # ls /sys/class/net/p7p1_1/0000:05:00.2/0000:05:00.0/net
    p7p1    p7p2

As it turns out, this *coincidentally* works properly for p7p1_1, but when you look at /sys/class/net/p7p1_2/device/physfn/net, you will end up with

    # ls /sys/class/net/p7p2_1/0000:05:00.2/0000:05:00.0/net
    p7p1    p7p2

so if you take the first netdev there, you end up with the *wrong* answer.

However, by examining the contents of /sys/class/net/* on this system, I found a file called "phys_port_id" that is set to a hex number on each VF and each PF. If we look for the PF netdev that has a phys_port_id matching that of the VF, then we will get the correct device for PF:

    # cat /sys/class/net/p7p1_1/phys_port_id
    e41d2d0300482611
    # cat /sys/class/net/p7p2_1/phys_port_id
    e41d2d0300482612
    # cat /sys/class/net/p7p1/phys_port_id
    e41d2d0300482611
    # cat /sys/class/net/p7p2/phys_port_id
    e41d2d0300482612

I can modify libvirt to check for matching phys_port_id (when it's available - for all other SRIOV net drivers the file exists but can't be read because the driver doesn't support it), and I think this may help Bug 1267191 as well.
Comment 32 yafu 2017-05-11 04:33:23 UTC 
Hi, Laine,

I found when attach vfio passthrough (<interface ...) with managed='no' , after destroy the guest, the admin mac will be set as 02:00:00:00:00:00. And as I know RHV use the vfio passthrough with managed='no'. Unfortunately,RHV also support hostdev (<hostdev...) passthrough. So it will cause there are VFs with same mac in guests and lead connectivity problems in RHV. 

It's a long story to explain using sriov in RHV. There are 3 points:
1.For the latset RHV version, it set all admin mac to 02:00:00:00:01 while creating vfs via RHV to fix the issue with SR-IOV + libvirt + MAC addresses that starts as 00:00:00:00:00:00 and libvirt couldn't get it back to zeros when VMs shutdown (which is fixed in libvirt in this bug).

2.All the VFs with the same admin mac 02:00:00:00:01 will also cause VM will get the same MAC address and will lead to connectivity issues on the guests. There is an existing bug in RHV to fix this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1447549

3.If RHV do nothing while creating VFs, libvirt will set the admin mac to 02:00:00:00:00:00 after vfio passthrough. Again, the connectivity problems happens.

For now, the issue will not affect rhv since rhv set all the vf admin mac to 02:00:00:00:00：01. So, I plan to file a libvirt bug in rhel7.5 to track this issue.

Steps to reproduce:
1.The initialized admin mac is all zero:
#ip l
...
219: enp5s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off


2.Define a vfio passthrough xml with managed='no':
#cat interface.xml
<interface type='hostdev' managed='no'>
<mac address='66:18:0b:c2:85:b1'/>
<source>
<address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
</source>
<model type='virtio'/>
</interface>

3.Attach the interface to guest:
#virsh attach-device rhel7.3-min interface.xml

4.Destroy the guest:
#virsh destroy rhel7.3-min

5.Check the admin mac:
...
219: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 90:e2:ba:11:d7:a4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

6.Check the libvirtd log:
2017-05-11 03:53:25.233+0000: 29490: debug : virNetDevSetVfConfig:1562 : RTM_SETLINK enp5s0f0 vf 0 MAC=66:18:0b:c2:85:b1 vlanid=0 - Success
2017-05-11 03:53:45.638+0000: 29487: debug : virNetDevSetVfConfig:1562 : RTM_SETLINK enp5s0f0 vf 0 MAC=00:00:00:00:00:00 vlanid=0 - Fail
2017-05-11 03:53:45.756+0000: 29487: debug : virNetDevSetVfConfig:1562 : RTM_SETLINK enp5s0f0 vf 0 MAC=02:00:00:00:00:00 vlanid=0 - Success
Comment 33 Laine Stump 2017-05-24 14:52:21 UTC 
I'm not sure what is the question is in Comment 32, but I will say that as long as devices are assigned to the guest using <interface type='hostdev'> (and *not* <hostdev>) the VF will always have the MAC address from the config while it is being used by the guest, and there will be no connectivity problems. It's not really practical to actively prevent using <hostdev> to assign VFs (since they are just another PCI device, after all), but anybody who does that deserves what they get.

Or were you asking something else?
Comment 34 Jingjing Shao 2017-06-09 03:08:16 UTC 
Hi laine,

I saw your comment (https://bugzilla.redhat.com/show_bug.cgi?id=1267191#c35)
The issue on mellanox NIC will be moved to rhel7.5. 

So According to the comment 31, I file a new bug about mlx NIC to track it.
https://bugzilla.redhat.com/show_bug.cgi?id=1460082
Comment 35 Jingjing Shao 2017-06-09 03:11:56 UTC 
With libvirt-3.2.0-9.el7.x86_64, libvirt QE  have test the ixgbe, igb, mlx, cisco, Emulex five type NICs with the five scenarios as below:

1.Do the VF pci assignment with vfio
2.Do the macvtap passthrough with VF which has been used in VF pci assignment with vfio
3.Do the macvtap passthrough with VF which never has been used in VF pci assignment with vfio
4.Do the VF pci assignment with vfio which has been used in macvtap passthrough with VF
5.Detach VF from host and Reattach VF to host


The remaining issues are as below, they will be tracked in the bugs as below
1.On the ixgbe nic, the admin mac will be set as 02:00:00:00:00:00 after macvtap passthrough when the initialized admin mac is all zero, which is caused by the issue of ixgbe driver:
https://bugzilla.redhat.com/show_bug.cgi?id=1447549

2.On the mlx NIC, there are two issues as below.
   (1)The admin MAC address of VF does not be saved and will be changed everytime
   (2) the second mac of vf can not be saved and restored as expected.
https://bugzilla.redhat.com/show_bug.cgi?id=1460082


According to the test as above， I change the status to verified
Comment 36 errata-xmlrpc 2017-08-01 17:21:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 37 errata-xmlrpc 2017-08-02 00:01:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.