RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1427780 - RHEV for Power: VFIO passthrough of SR-IOV virtual functions
Summary: RHEV for Power: VFIO passthrough of SR-IOV virtual functions
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: ppc64le
OS: Linux
high
high
Target Milestone: rc
: 7.4
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1314131 1315131 1421940
Blocks: 1299988 RHV4.1PPC 1395265 1401400 1445460
TreeView+ depends on / blocked
 
Reported: 2017-03-01 08:18 UTC by Dan Zheng
Modified: 2017-08-02 07:44 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1314131
Environment:
Last Closed: 2017-08-02 07:44:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 152103 0 None None None 2019-02-21 07:04:37 UTC

Comment 9 Dan Zheng 2017-04-27 09:40:42 UTC 
Test packages:
libvirt-3.2.0-2.el7.ppc64le
qemu-kvm-rhev-2.8.0-6.el7.ppc64le
kernel-3.10.0-652.el7.ppc64le

Intel-X710 card is used.

######Test case 1: RHEL7-86879 - [SRIOV] Check the mac address of VFs  -- PASS

1. Enable VF and check VF's mac address and found mac is zero 
# echo 0 > /sys/bus/pci/devices/0003\:0f\:00.0/sriov_numvfs 
# ip l 
7: enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000 
link/ether 68:05:ca:37:df:21 brd ff:ff:ff:ff:ff:ff 
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 

2. Add <interface> section with one VF, then mac address is assigned randomly and start VM 
# lspci 
0003:0f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 
0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01) 
0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01) 

# virsh dumpxml vm1 | grep interface -A7 
<interface type='hostdev' managed='yes'> 
<mac address='52:54:00:b5:3b:24'/> 
<driver name='vfio'/> 
<source> 
<address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/> 
</source> 
<alias name='hostdev0'/> 
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> 
</interface> 

enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000 
link/ether 68:05:ca:37:df:21 brd ff:ff:ff:ff:ff:ff 
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 
vf 1 MAC 52:54:00:b5:3b:24, spoof checking on, link-state auto, trust on 

3. Check lspci in guest and VF can be found 
00:0a.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01) 

4. Destroy guest and found VF has another mac address 
# ip l 
enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000 
link/ether 68:05:ca:37:df:21 brd ff:ff:ff:ff:ff:ff 
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on 
vf 1 MAC ca:69:c3:49:7a:32, spoof checking on, link-state auto, trust on 

######Test case 2: RHEL7-18498 [SR-IOV] Create up to MAX VFs --- PASS

1. Check max VF supported of the card
# lspci -s 0003:0f:00.1 -vv |grep 'Initial VFs' 
Initial VFs: 32, Total VFs: 32, Number of VFs: 0, Function Dependency Link: 00 
2. Enable 32 VFs
# echo 32 > /sys/bus/pci/devices/0003\:0f\:00.1/sriov_numvfs 
3. list VFs
0003:0f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
...
# lspci |grep X710 | wc -l
36        => 4 PFs + 32 VFs
# lspci -s 0003:0f:00.1 -vv |grep 'Initial VFs' 
		Initial VFs: 32, Total VFs: 32, Number of VFs: 32, Function Dependency Link: 01

4. # virsh nodedev-list --tree 
can list correct VFs.

#### Test case 3: RHEL7-18493 - [SR-IOV] Assign several VFs to the same guest.	 --- PASS

1. Use <hostdev> to assign 2 VFs to the guest
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
2. Start the guest and check guest
# lspci
00:07.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
00:08.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
3. As the X710 card does not connect to switch, no dhcp ip can be retrieved. I configure it with static IP.

### Test case 4: RHEL7-18511 - [SR-IOV] Suspend and resume guest with assigned VF  -- PASS	
1. Assign VF to guest using <interface>
2. Start VM and suspend the VM and then resume the VM.
3. The lspci in guest can list the VF out

## Test case 5 : RHEL7-18508 [SR-IOV] Reboot guest with assigned VF ---PASS
1. Assign VF to guest using
2. Start VM and reboot VM.
3. The lspci in guest can list the VF out

### Test case 6: RHEL7-18505 [SR-IOV] Hotplug/unplug network device with managed='yes'  --- PASS

As bug 1272300, detach-device will cause host crash. So I did not test detach.
1. Start VM
2. Hotplug a VF as interface with mac address specified
3. Check lspci in guest and can find the VF
4. Destroy vm and check the driver of the VF is back to the host kernel driver.
# virsh nodedev-dumpxml pci_0003_0f_06_1
<device>
<name>pci_0003_0f_06_1</name>
<path>/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/0003:02:11.0/0003:0f:06.1</path>
<parent>pci_0003_02_11_0</parent>
<driver>
<name>***i40evf***</name>
</driver>
Comment 10 Andrea Bolognani 2017-04-27 11:57:57 UTC 
(In reply to Dan Zheng from comment #9)
> ### Test case 6: RHEL7-18505 [SR-IOV] Hotplug/unplug network device with
> managed='yes'  --- PASS
> 
> As bug 1272300, detach-device will cause host crash. So I did not test
> detach.

Detaching should work just fine since, unlike PFs, VFs each
have their own IOMMU group.
Comment 11 Dan Zheng 2017-05-03 09:15:06 UTC 
Test packages:
qemu-kvm-rhev-2.9.0-2.el7.ppc64le
kernel-3.10.0-657.el7.ppc64le
libvirt-3.2.0-3.el7.ppc64le

### Test case 5: RHEL7-18526 - [SR-IOV] Hot-plug VFs from sriov vfs pool to guest and hot-unplug VF  --- hotunplug will cause host to crash        
1. Prepare one network with hostdev forward mode
<network connections='2'>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
    <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
  </forward>
</network>

2. Prepare one xml like the following one:
# cat vfpool.xml
<interface type='network'>
<source network='hostnet'/>
</interface>

3. Hot-plug the vf to the guest.
# virsh attach-device vm1 vfpool.xml
Device attached successfully
# virsh attach-device vm1 vfpool.xml
Device attached successfully
# virsh attach-device vm1 vfpool.xml
error: Failed to attach device from vfpool.xml
error: internal error: network 'hostnet' requires exclusive access to interfaces, but none are available  <=== This is as expected

4. Dumpxml:
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:e7:5f:78'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
      </source>
      <model type='virtio'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:63:0b:cc'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
      </source>
      <model type='virtio'/>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

5. Detach the last interface
#  virsh detach-device vm1 vf-detach.xml  
Device detached successfully

6. The last interface is removed in dumpxml and lspci within guest.

7. Try four VFs (2 of them are from another PF port) and attach the vf pool to guest and detach them ok.

8. Repeat attach and detach one VF of the pool, the host will crash and restart.
0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 01)
pci_0003_0f_06_1 and pci_0003_0f_06_0 belong to different iommuGroup
    <iommuGroup number='7'>
      <address domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
    </iommuGroup>
    <iommuGroup number='6'>
      <address domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
    </iommuGroup>

But the host still restart after executing detach.
vf-detach.xml has same content as those in dumpxml of this interface

# virsh detach-device vm1 vf-detach.xml
Write failed: Broken pipe
[root@work-machine xxx]#
Comment 12 Dan Zheng 2017-05-12 09:07:21 UTC 
kernel-3.10.0-666.el7.ppc64le
qemu-kvm-rhev-2.9.0-2.el7.ppc64le
libvirt-3.2.0-4.el7.ppc64le

### Test case 8:  [SR-IOV] Attach interfaces from macvtap network with passthrough mode ---PASS
1. Configure 2 VFs and define a macvtap passthrough network.
# ip l
...
7: enP3p15s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6805ca37df21 state UP mode DEFAULT qlen 1000
    link/ether 68:05:ca:37f:21 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
...
12: enP3p15s6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 52:56:29:6b:86:2f brd ff:ff:ff:ff:ff:ff
13: enP3p15s6f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 72:cd:51:38:64:ce brd ff:ff:ff:ff:ff:ff

# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 macvtap-passthrough  active     no            yes

# virsh net-dumpxml macvtap-passthrough
<network>
  <name>macvtap-passthrough</name>
  <uuid>a53c4f33-58ea-4b2e-80b4-f9d103b7b0a3</uuid>
  <forward dev='enP3p15s6' mode='passthrough'>
    <interface dev='enP3p15s6'/>
    <interface dev='enP3p15s6f1'/>
  </forward>
</network>

2. Add the following xml to the guests( vm1 and vm2)
<interface type='network'>
<source network='macvtap-passthrough'/>
<model type='virtio'/>
</interface>

3. Start vm1, vm1 can be logon successfully.
# virsh dumpxml vm1|grep interface -A7
    <interface type='direct'>
      <mac address='52:54:00:6c:b03'/>
      <source network='macvtap-passthrough' dev='enP3p15s6' mode='passthrough'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

4. Destroy vm1 and start it again.

5. Check network 
# virsh net-dumpxml macvtap-passthrough
<network connections='1'>
<name>macvtap-passthrough</name>
<uuid>a53c4f33-58ea-4b2e-80b4-f9d103b7b0a3</uuid>
<forward dev='enP3p15s6' mode='passthrough'>
<interface dev='enP3p15s6' connections='1'/>
<interface dev='enP3p15s6f1'/>
</forward>
</network>

6. Start vm2. 

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     vm1                            running
 4     vm2                            running

7. Check network. 2 connections are displayed.
# virsh net-dumpxml macvtap-passthrough
<network connections='2'>
  <name>macvtap-passthrough</name>
  <uuid>a53c4f33-58ea-4b2e-80b4-f9d103b7b0a3</uuid>
  <forward dev='enP3p15s6' mode='passthrough'>
    <interface dev='enP3p15s6' connections='1'/>
    <interface dev='enP3p15s6f1' connections='1'/>
  </forward>
</network>

8. Check dumpxml of vm2
# virsh dumpxml vm2|grep interface -A7
    <interface type='direct'>
      <mac address='52:54:00:a1:d4:de'/>
      <source network='macvtap-passthrough' dev='enP3p15s6f1' mode='passthrough'/>
      <target dev='macvtap1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

9. Logon vm1 and vm2 ok. lspci can list the nic device.
Comment 13 Andrea Bolognani 2017-05-18 13:23:15 UTC 
(In reply to Dan Zheng from comment #11)
[...]
> 7. Try four VFs (2 of them are from another PF port) and attach the vf pool
> to guest and detach them ok.
> 
> 8. Repeat attach and detach one VF of the pool, the host will crash and
> restart.
> 0003:0f:06.0 Ethernet controller: Intel Corporation XL710/X710 Virtual
> Function (rev 01)
> 0003:0f:06.1 Ethernet controller: Intel Corporation XL710/X710 Virtual
> Function (rev 01)
> pci_0003_0f_06_1 and pci_0003_0f_06_0 belong to different iommuGroup
>     <iommuGroup number='7'>
>       <address domain='0x0003' bus='0x0f' slot='0x06' function='0x1'/>
>     </iommuGroup>
>     <iommuGroup number='6'>
>       <address domain='0x0003' bus='0x0f' slot='0x06' function='0x0'/>
>     </iommuGroup>
> 
> But the host still restart after executing detach.
> vf-detach.xml has same content as those in dumpxml of this interface
> 
> # virsh detach-device vm1 vf-detach.xml
> Write failed: Broken pipe
> [root@work-machine xxx]#

I've been able to make the host reach an unusable state
without even resorting to using VFs backed by separate PFs,
just with the simple case described in steps 1-6.

Even though we're not talking about a full host crash, it
was still enough to force me to reboot the machine in order
to be able to move forward with testing.

It seems to me that the i40evf driver is just not very
reliable, at least on ppc64le: see Bug 1445460 for some
more evidence of that.
Comment 14 Dan Zheng 2017-06-08 06:36:59 UTC 
The host crash problem is tracked by BZ 1445460 now. And no other new bugs are found. So it is marked verified now.
Note You need to log in before you can comment on or make changes to this bug.