Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1540964

Summary: Booting guest with q35 multifuction, vIOMMU and device assignment, then dpdk's testpmd will show "VFIO group is not viable!"
Product: Red Hat Enterprise Linux 7 Reporter: Pei Zhang <pezhang>
Component: qemu-kvm-rhevAssignee: Amnon Ilan <ailan>
Status: CLOSED WONTFIX QA Contact: Pei Zhang <pezhang>
Severity: medium Docs Contact:
Priority: high    
Version: 7.5CC: ailan, alex.williamson, chayang, hhuang, juzhang, marcel, maxime.coquelin, michen, peterx, siliu, virt-maint
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-19 18:44:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1558351    

Description Pei Zhang 2018-02-01 12:16:52 UTC 
Description of problem:
Boot guest with q35, vIOMMU and device assignment, then start testpmd in guest, fail, the terminal shows info like "0000:01:00.0 VFIO group is not viable".

Version-Release number of selected component (if applicable):
3.10.0-841.el7.x86_64
kernel-3.10.0-837.el7.x86_64
qemu-kvm-rhev-2.10.0-18.el7.x86_64
dpdk-17.11-7.el7.x86_64


How reproducible:
100%


Steps to Reproduce:
1. Boot VM with q35 multifunction, vIOMMU and 2 PFs, refer to [1]

2. Start testpmd in guest , refer to[2]. It shows "0000:01:00.0 VFIO group is not viable" like below:

EAL: Detected 6 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:1528 net_ixgbe
EAL:   0000:01:00.0 VFIO group is not viable!
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:1528 net_ixgbe
EAL:   0000:02:00.0 VFIO group is not viable!
EAL: Requested device 0000:02:00.0 cannot be used
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=179456, size=2176, socket=0
Done


Actual results:
In guest, the assigned network devices can not be recognized by dpdk's testpmd.

Expected results:
The testpmd should boot up successfully.

Additional info:
1. Boot vm without multifuction like below, everything works well.

...
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-device pcie-root-port,id=root.4,chassis=4 \
-device vfio-pci,host=0000:81:00.0,bus=root.1 \
-device vfio-pci,host=0000:81:00.1,bus=root.2 \
...

2. I'm not sure if this is same issue with below bug. Feel free to close it if they are same issue. Thanks.

Bug 1526948 - Do PVP testing with vIOMMU and Q35 pcie multifunction, guest dpdk's testpmd boot up with errors


Reference:
[1]/usr/libexec/qemu-kvm -name rhel7.5 -M q35,kernel-irqchip=split \
-cpu host -m 8G \
-device intel-iommu,intremap=true,caching-mode=true \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 6,sockets=1,cores=6,threads=1 \
-device pcie-root-port,id=root.1,chassis=1,multifunction=on,addr=0x2.0 \
-device pcie-root-port,id=root.2,chassis=2,addr=0x2.1 \
-device pcie-root-port,id=root.3,chassis=3,addr=0x2.2 \
-device pcie-root-port,id=root.4,chassis=4,addr=0x2.3 \
-device vfio-pci,host=0000:81:00.0,bus=root.1 \
-device vfio-pci,host=0000:81:00.1,bus=root.2 \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,bus=root.3,mac=88:66:da:5f:dd:01 \
-drive file=/home/images_nfv-virt-rt-kvm/rhel7.5_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.4 \
-vnc :2 \
-monitor stdio \

[2]
/usr/bin/testpmd \
-l 1,2,3,4,5 -n 4 \
-d /usr/lib64/librte_pmd_ixgbe.so \
-w 0000:01:00.0 -w 0000:02:00.0 \
-- \
--nb-cores=2 \
--disable-hw-vlan \
-i --disable-rss \
--rxq=1 --txq=1
Comment 2 Peter Xu 2018-02-02 06:49:43 UTC 
The devices are not viable because after specifying multi-function for the pcie root ports, I see that all the pcie root port (including their downstream devices) are put into the same IOMMU group 2:

[root@bootp-73-75-70 ~]# lspci -tv
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Device 1234:1111
           +-02.0-[01]----00.0  Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
           +-02.1-[02]----00.0  Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
           +-02.2-[03]----00.0  Red Hat, Inc. Virtio network device
           +-02.3-[04]----00.0  Red Hat, Inc. Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:00.0 to group 0
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:01.0 to group 1
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:02.0 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:02.1 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:02.2 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:02.3 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:1f.0 to group 3
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:1f.2 to group 3
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:00:1f.3 to group 3
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:01:00.0 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:02:00.0 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:03:00.0 to group 2
Feb 02 14:40:14 localhost.localdomain kernel: iommu: Adding device 0000:04:00.0 to group 2

Since there are devices that are still drived by kernel drivers in IOMMU group 2, so these two ixgbe devices are not viable to be assigned to DPDK, which is what the error is talking about.

If not using multi-function pcie root ports, each pcie-root-port (along with its downstream device) will be setup with its own IOMMU group.

Marcel, do you know why this happened?

(CCing Alex too)

Peter
Comment 3 Alex Williamson 2018-02-02 15:15:01 UTC 
(In reply to Peter Xu from comment #2)
> Marcel, do you know why this happened?
> 
> (CCing Alex too)

There's no ACS on the emulated root ports, this not only means that all devices downstream of the root port will be grouped together, but also that any root ports (or other devices) which are part of a multifunction slot are also grouped together, as are all downstream devices from those ports.  The root ports can be split into separate slots to avoid this.
Comment 4 Marcel Apfelbaum 2018-02-02 16:33:00 UTC 
(In reply to Alex Williamson from comment #3)
> (In reply to Peter Xu from comment #2)
> > Marcel, do you know why this happened?
> > 
> > (CCing Alex too)
> 
> There's no ACS on the emulated root ports, this not only means that all
> devices downstream of the root port will be grouped together, but also that
> any root ports (or other devices) which are part of a multifunction slot are
> also grouped together, as are all downstream devices from those ports.  The
> root ports can be split into separate slots to avoid this.

I proposed a GSOC project to implement ACS emulation for generic
PCIe Root Ports. We can use your scenario as a reference point.

Thanks,
Marcel
Comment 5 Peter Xu 2018-02-06 07:32:49 UTC 
Thanks Alex, Marcel.

Let's avoid using multi-function when we need separate IOMMU groups, or we need to wait until the project that Marcel mentioned on ACS emulation.

According to above, I'm closing this one.

Peter
Comment 6 Maxime Coquelin 2018-02-06 09:02:43 UTC 
*** Bug 1526948 has been marked as a duplicate of this bug. ***
Comment 8 Amnon Ilan 2018-02-06 10:55:01 UTC 
The current status: if a user needs to have assigned device to Guest userspace (which requires vIOMMU) then he cannot use multi-function root-ports - a slot on bus0, for which we want to put the assigned device, will have only a single root port (i.e. cannot use "multi-function=on" in the QEMU command line for it).