Description of problem: VF's MAC address should be generated once enabled. When enabling VFs on a SR-IOV host(igb) via the engine, the MAC addresses of the VFs are the same. Before our fix they all were zeros, but after out fix they start as: 2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 44:1e:a1:71:a9:c8 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:00:00:00:00:01, spoof checking on, link-state auto vf 1 MAC 02:00:00:00:00:01, spoof checking on, link-state auto As long as those VFs used as sr-iov vNICs on the guest, everything is working great, because each vNIC of the guest, will get new MAC, which is the vNIC's MAC address assigned from the cluster's pool range. But if those VFs used as a passthrough host-devices, the MAC address will the same. And if we have multiple guests, each has a different VF as a host-device, it means that VM will get the same MAC address when starts and will lead to connectivity issues on the guests. So, we must generate the MAC addresses of the VFs once enabled, to something like that maybe: 2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 44:1e:a1:71:a9:c8 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:00:00:00:00:01, spoof checking on, link-state auto vf 1 MAC 02:00:00:00:00:02, spoof checking on, link-state auto vf 2 MAC 02:00:00:00:00:03, spoof checking on, link-state auto Version-Release number of selected component (if applicable): 4.1.2-0.1.el7 vdsm-4.19.11-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Enable 3 VFs on a SR-IOV host(igb) via the setup networks dialog in the engine 2. Go to the host and run 'ip link show' 3. Create 3 VMs, add 1 VF as a host-device to each host(not vNIC) 4. Run the VMs and check which MAC address the VMs got Actual results: 2. 2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 44:1e:a1:71:a9:c8 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:00:00:00:00:01, spoof checking on, link-state auto vf 1 MAC 02:00:00:00:00:01, spoof checking on, link-state auto vf 2 MAC 02:00:00:00:00:01, spoof checking on, link-state auto 4. Each VM got the same MAC address: 02:00:00:00:00:01 Expected results: The MAC address of each VF must be different once VFs enabled on the host and we should take care of it. Each guest should get a different MAC address if used as a passthrough host-device.
You shouldn't be assigning the VFs to guests using <hostdev>. Use <interface type='hostdev'> instead and libvirt will setup their MAC address (and optionally a vlan tag) before they are assigned.
BTW, even if you reverted the change to RHEV that initializes all the admin MACs to 02:00:00:00:00:01, using <hostdev> for a VF netdev is *still* a bad idea, because the guest will get a new & different MAC address *every time it is restarted*, which creates havoc in the guest's network config (each new MAC address causes the guest OS to think that new hardware has been installed, and that new hardware will require new config). It really is never a good idea to use <hostdev> for VF netdevs.
Yan Fu, would you explain why are you passing VFs as generic host devices? RHV's SRIOV feature is intended to manage VFs; when you opt for passing host devices you are basically left on your own. We do not block it, but do not recommend it, either. What happened before our fix, when an all-zero mac was set on a VF that is passed to a guest? Did something in the stack supplied a random mac for it?
(In reply to Dan Kenigsberg from comment #3) > Yan Fu, would you explain why are you passing VFs as generic host devices? > > RHV's SRIOV feature is intended to manage VFs; when you opt for passing host > devices you are basically left on your own. We do not block it, but do not > recommend it, either. > > What happened before our fix, when an all-zero mac was set on a VF that is > passed to a guest? Did something in the stack supplied a random mac for it? I just tested the sriov using in RHV and worried about maybe customer will use in this way. And in my opinion, RHV/libvirt do not recommend it but not block it, but it always have issues when using it, it's better to add description in the document to avoid using of vf pci passthourgh. All zero mac will generate a random mac in the guest since all zero is not a valid mac.
Dan - just fyi, I recently gave up trying to get all the various kernel drivers to properly support re-setting admin MAC to 00:00:00:00:00:00 (even the ones that do that probably don't also reset the "administratively set" flag for that VF anyway, which would lead to other problems). Upstream already has, and RHEL7.4 & and RHEL7.3 libvirt will very soon have, much more intelligent saving/restoring of VF MAC addresses that should make it unnecessary for you to set 02:00:00:00:00:01 for all the VF admin MAC addresses - You can see details in Bug 1415609 and Bug 1442040.
Thanks, Laine. Are you safe from the problem reported here? Do you use clear the VF mac by using a random one? Regardless, we'll drop our code and require libvirt-2.0.0-10.el7_3.8 that solves bug 1442040.
(In reply to Dan Kenigsberg from comment #6) > Thanks, Laine. Are you safe from the problem reported here? Do you use clear > the VF mac by using a random one? Not exactly. But we don't *pre-emptively* set the admin MAC as you do. Here's what we do: 1) for vfio assignment: a) save the current VF MAC & admin MAC b) unbind from host net driver c) set admin MAC to the MAC the guest wants to use d) .... e) set admin MAC to the original *VF* MAC f) rebind VF to host net driver (result - VF MAC will be the original (probably random) MAC, admin MAC will be set to that same (probably random) MAC 2) for macvtap passthrough a) save the current VF MAC and admin MAC b) attempt to set VF MAC to the MAC the guest wants to use c) if (b) fails, set the *admin* MAC to the MAC the guest wants, then rebind the VF net driver --> this sets the VF MAC to the one desired by guest. d) ... e) attempt to set VF MAC to the saved VF MAC f) if (e) fails, set *admin* MAC to the saved VF MAC, then rebind VF to the host net driver. (result - VF MAC will always be set to what it was originally, admin MAC may or may not be changed (but is unimportant) libvirt does now have code to retry setting the admin MAC with 02:00:00:00:00:00 if we ever try to set 00:00:00:00:00:00 and it fails, but in practice this ends up only happening when there is some other problem (e.g. if the kernel erroneously blocks setting VF MAC at a time when admin MAC is 00:00:00:00:00:00 (meaning it has never been set) - this can happen with ixgbe, see kernel Bug 1447549). If a customer *always* uses <hostdev> for their VFs, then libvirt will never set either the VF or admin MAC address, and the guest will get a new and different MAC address each time they start, and they will deserve all the guest-side headache that comes with this (since each new MAC address will be detected by the guest OS as an entirely new interface, which will need new config). If they mix between using <interface type='hostdev'> and <hostdev> on the same VF, then the guest using <hostdev> will end up with [some other previously-used non-random MAC], and there really is nothing we can do about that.
I hate our hack, but removing it requires thorough verification. Let us keep it for now.