Bug 1427780

Test packages:
libvirt-3.2.0-2.el7.ppc64le
qemu-kvm-rhev-2.8.0-6.el7.ppc64le
kernel-3.10.0-652.el7.ppc64le

Intel-X710 card is used.

######Test case 1: RHEL7-86879 - [SRIOV] Check the mac address of VFs  -- PASS

1. Enable VF and check VF's mac address and found mac is zero 
# echo 0 > /sys/bus/pci/devices/0003\:0f\:00.0/sriov_numvfs 
# ip l 
7: enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000 
link/ether 68:05:ca:37:df:21 brd ff:ff:ff:ff:ff:ff 
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 

2. Add <interface> section with one VF, then mac address is assigned randomly and start VM 
# lspci 
0003:0f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01) 
0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01) 

# virsh dumpxml vm1 | grep interface -A7 
<interface type='hostdev' managed='yes'> 
<mac address='52:54:00:b5:3b:24'/> 
<driver name='vfio'/> 
<source> 
<address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/> 
</source> 
<alias name='hostdev0'/> 
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> 
</interface> 

enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000 
link/ether 68:05:ca:37:df:21 brd ff:ff:ff:ff:ff:ff 
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 
vf 1 MAC 52:54:00:b5:3b:24, spoof checking on, link-state auto, trust on 

3. Check lspci in guest and VF can be found 
00:0a.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01) 

4. Destroy guest and found VF has another mac address 
# ip l 
enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000 
link/ether 68:05:ca:37:df:21 brd ff:ff:ff:ff:ff:ff 
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 
vf 1 MAC ca:69:c3:49:7a:32, spoof checking on, link-state auto, trust on 

######Test case 2: RHEL7-18498 [SR-IOV] Create up to MAX VFs --- PASS

1. Check max VF supported of the card
# lspci -s 0003:0f:00.1 -vv |grep 'Initial VFs' 
Initial VFs: 32, Total VFs: 32, Number of VFs: 0, Function Dependency Link: 00 
2. Enable 32 VFs
# echo 32 > /sys/bus/pci/devices/0003\:0f\:00.1/sriov_numvfs 
3. list VFs
0003:0f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
...
# lspci |grep X710 | wc -l
36        => 4 PFs + 32 VFs
# lspci -s 0003:0f:00.1 -vv |grep 'Initial VFs' 
		Initial VFs: 32, Total VFs: 32, Number of VFs: 32, Function Dependency Link: 01

4. # virsh nodedev-list --tree 
can list correct VFs.

#### Test case 3: RHEL7-18493 - [SR-IOV] Assign several VFs to the same guest.	 --- PASS

1. Use <hostdev> to assign 2 VFs to the guest
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
2. Start the guest and check guest
# lspci
00:07.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
00:08.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
3. As the X710 card does not connect to switch, no dhcp ip can be retrieved. I configure it with static IP.

### Test case 4: RHEL7-18511 - [SR-IOV] Suspend and resume guest with assigned VF  -- PASS	
1. Assign VF to guest using <interface>
2. Start VM and suspend the VM and then resume the VM.
3. The lspci in guest can list the VF out

## Test case 5 : RHEL7-18508 [SR-IOV] Reboot guest with assigned VF ---PASS
1. Assign VF to guest using
2. Start VM and reboot VM.
3. The lspci in guest can list the VF out

### Test case 6: RHEL7-18505 [SR-IOV] Hotplug/unplug network device with managed='yes'  --- PASS

As bug 1272300, detach-device will cause host crash. So I did not test detach.
1. Start VM
2. Hotplug a VF as interface with mac address specified
3. Check lspci in guest and can find the VF
4. Destroy vm and check the driver of the VF is back to the host kernel driver.
# virsh nodedev-dumpxml pci_0003_0f_06_1
<device>
<name>pci_0003_0f_06_1</name>
<path>/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/0003:02:11.0/0003:0f:06.1</path>
<parent>pci_0003_02_11_0</parent>
<driver>
<name>***i40evf***</name>
</driver>

(In reply to Dan Zheng from comment #9)
> ### Test case 6: RHEL7-18505 [SR-IOV] Hotplug/unplug network device with
> managed='yes'  --- PASS
> 
> As bug 1272300, detach-device will cause host crash. So I did not test
> detach.

Detaching should work just fine since, unlike PFs, VFs each
have their own IOMMU group.

Test packages:
qemu-kvm-rhev-2.9.0-2.el7.ppc64le
kernel-3.10.0-657.el7.ppc64le
libvirt-3.2.0-3.el7.ppc64le

### Test case 5: RHEL7-18526 - [SR-IOV] Hot-plug VFs from sriov vfs pool to guest and hot-unplug VF  --- hotunplug will cause host to crash        
1. Prepare one network with hostdev forward mode
<network connections='2'>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
    <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
  </forward>
</network>

2. Prepare one xml like the following one:
# cat vfpool.xml
<interface type='network'>
<source network='hostnet'/>
</interface>

3. Hot-plug the vf to the guest.
# virsh attach-device vm1 vfpool.xml
Device attached successfully
# virsh attach-device vm1 vfpool.xml
Device attached successfully
# virsh attach-device vm1 vfpool.xml
error: Failed to attach device from vfpool.xml
error: internal error: network 'hostnet' requires exclusive access to interfaces, but none are available  <=== This is as expected

4. Dumpxml:
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:e7:5f:78'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
      </source>
      <model type='virtio'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:63:0b:cc'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
      </source>
      <model type='virtio'/>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

5. Detach the last interface
#  virsh detach-device vm1 vf-detach.xml  
Device detached successfully

6. The last interface is removed in dumpxml and lspci within guest.

7. Try four VFs (2 of them are from another PF port) and attach the vf pool to guest and detach them ok.

8. Repeat attach and detach one VF of the pool, the host will crash and restart.
0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
pci_0003_0f_06_1 and pci_0003_0f_06_0 belong to different iommuGroup
    <iommuGroup number='7'>
      <address domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
    </iommuGroup>
    <iommuGroup number='6'>
      <address domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
    </iommuGroup>

But the host still restart after executing detach.
vf-detach.xml has same content as those in dumpxml of this interface

# virsh detach-device vm1 vf-detach.xml
Write failed: Broken pipe
[root@work-machine xxx]#

kernel-3.10.0-666.el7.ppc64le
qemu-kvm-rhev-2.9.0-2.el7.ppc64le
libvirt-3.2.0-4.el7.ppc64le

### Test case 8:  [SR-IOV] Attach interfaces from macvtap network with passthrough mode ---PASS
1. Configure 2 VFs and define a macvtap passthrough network.
# ip l
...
7: enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000
    link/ether 68:05:ca:37f:21 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
...
12: enP3p15s6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 52:56:29:6b:86:2f brd ff:ff:ff:ff:ff:ff
13: enP3p15s6f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 72:cd:51:38:64:ce brd ff:ff:ff:ff:ff:ff

# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 macvtap-passthrough  active     no            yes

# virsh net-dumpxml macvtap-passthrough
<network>
  <name>macvtap-passthrough</name>
  <uuid>a53c4f33-58ea-4b2e-80b4-f9d103b7b0a3</uuid>
  <forward dev='enP3p15s6' mode='passthrough'>
    <interface dev='enP3p15s6'/>
    <interface dev='enP3p15s6f1'/>
  </forward>
</network>

2. Add the following xml to the guests( vm1 and vm2)
<interface type='network'>
<source network='macvtap-passthrough'/>
<model type='virtio'/>
</interface>

3. Start vm1, vm1 can be logon successfully.
# virsh dumpxml vm1|grep interface -A7
    <interface type='direct'>
      <mac address='52:54:00:6c:b03'/>
      <source network='macvtap-passthrough' dev='enP3p15s6' mode='passthrough'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

4. Destroy vm1 and start it again.

5. Check network 
# virsh net-dumpxml macvtap-passthrough
<network connections='1'>
<name>macvtap-passthrough</name>
<uuid>a53c4f33-58ea-4b2e-80b4-f9d103b7b0a3</uuid>
<forward dev='enP3p15s6' mode='passthrough'>
<interface dev='enP3p15s6' connections='1'/>
<interface dev='enP3p15s6f1'/>
</forward>
</network>

6. Start vm2. 

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     vm1                            running
 4     vm2                            running

7. Check network. 2 connections are displayed.
# virsh net-dumpxml macvtap-passthrough
<network connections='2'>
  <name>macvtap-passthrough</name>
  <uuid>a53c4f33-58ea-4b2e-80b4-f9d103b7b0a3</uuid>
  <forward dev='enP3p15s6' mode='passthrough'>
    <interface dev='enP3p15s6' connections='1'/>
    <interface dev='enP3p15s6f1' connections='1'/>
  </forward>
</network>

8. Check dumpxml of vm2
# virsh dumpxml vm2|grep interface -A7
    <interface type='direct'>
      <mac address='52:54:00:a1:d4:de'/>
      <source network='macvtap-passthrough' dev='enP3p15s6f1' mode='passthrough'/>
      <target dev='macvtap1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

9. Logon vm1 and vm2 ok. lspci can list the nic device.

(In reply to Dan Zheng from comment #11)
[...]
> 7. Try four VFs (2 of them are from another PF port) and attach the vf pool
> to guest and detach them ok.
> 
> 8. Repeat attach and detach one VF of the pool, the host will crash and
> restart.
> 0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual
> Function (rev 01)
> 0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual
> Function (rev 01)
> pci_0003_0f_06_1 and pci_0003_0f_06_0 belong to different iommuGroup
>     <iommuGroup number='7'>
>       <address domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
>     </iommuGroup>
>     <iommuGroup number='6'>
>       <address domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
>     </iommuGroup>
> 
> But the host still restart after executing detach.
> vf-detach.xml has same content as those in dumpxml of this interface
> 
> # virsh detach-device vm1 vf-detach.xml
> Write failed: Broken pipe
> [root@work-machine xxx]#

I've been able to make the host reach an unusable state
without even resorting to using VFs backed by separate PFs,
just with the simple case described in steps 1-6.

Even though we're not talking about a full host crash, it
was still enough to force me to reboot the machine in order
to be able to move forward with testing.

It seems to me that the i40evf driver is just not very
reliable, at least on ppc64le: see Bug 1445460 for some
more evidence of that.

The host crash problem is tracked by BZ 1445460 now. And no other new bugs are found. So it is marked verified now.