Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1038424

Summary: fail to detect the vfio-pci assign with PF device in guest (Broadcom BCM57810 card)
Product: Red Hat Enterprise Linux 7 Reporter: Sibiao Luo <sluo>
Component: qemu-kvmAssignee: Bandan Das <bdas>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, alex.williamson, chayang, hhuang, juzhang, michen, qzhang, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-24 17:22:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
guest_dmesg_log. none

Description Sibiao Luo 2013-12-05 05:57:02 UTC
Description of problem:
boot guest with a PF vfio-pci assgined, excute 'info pci' in HMP monitor can output this device but fail to detect the vfio-pci assign PF/VF device in guest.

Version-Release number of selected component (if applicable):
host info:
3.10.0-57.el7.x86_64
qemu-kvm-1.5.3-20.el7.x86_64
seabios-1.7.2.2-4.el7.x86_64
guest info:
3.10.0-57.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Load vfio-pci, vfio, vfio_iommu_type1 modules.
# lsmod | grep vfio
vfio_pci               36474  0 
vfio_iommu_type1       17636  0 
vfio                   20777  2 vfio_iommu_type1,vfio_pci

2.Check what other devices are in the same group as PF, unbind all of them and bind to vfio-pci.
# lspci | grep -i BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
# readlink /sys/bus/pci/devices/0000:08:00.0/iommu_group
../../../../kernel/iommu_groups/14
# readlink /sys/bus/pci/devices/0000:08:00.1/iommu_group
../../../../kernel/iommu_groups/14
# lspci -n -s 0000:08:00.0 | awk '{ print $3 }'
14e4:168e
# echo "14e4 168e" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:00.0 > /sys/bus/pci/devices/0000\:08\:00.0/driver/unbind 
# echo 0000:08:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
# lspci -n -s 0000:08:00.1 | awk '{ print $3 }'
14e4:168e
# echo "14e4 168e" >> /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:00.1 >> /sys/bus/pci/devices/0000\:08\:00.1/driver/unbind 
# echo 0000:08:00.1 >> /sys/bus/pci/drivers/vfio-pci/bind

3.Do vfio-pci with PF to guest.
e.g:...-device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0

4.info pci/qtree in HMP monitor and lspci/dmesg in guest.
(qemu) info pci
...
  Bus  0, device   5, function 0:
    Class 65535: PCI device ffff:ffff
      IRQ 0.
      BAR0: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe].
      BAR2: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe].
      BAR4: 64 bit prefetchable memory at 0xffffffffffffffff [0x0000fffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "sluo_guest_nic_pf0"
(qemu) info qtree
...
      dev: vfio-pci, id "sluo_guest_nic_pf0"
        host = 0000:08:00.0
        x-intx-mmap-timeout-ms = 1100
        x-vga = off
        bootindex = -1
        addr = 05.0
        romfile = <null>
        rombar = 1
        multifunction = off
        command_serr_enable = on
        class Class ffff, addr 00:05.0, pci id ffff:ffff (sub ffff:ffff)
        bar 0: mem at 0xffffffffffffffff [0x7ffffe]
        bar 2: mem at 0xffffffffffffffff [0x7ffffe]
        bar 4: mem at 0xffffffffffffffff [0xfffe]
        bar 6: mem at 0xffffffffffffffff [0x1fffe]
...
guest] # lspci
guest] # dmesg | grep 00:05.0

Actual results:
after step 4, fail to detect the vfio-pci assign PF/VF device in guest, I will paste the guest dmesg log later.
# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Communication controller: Red Hat, Inc Virtio console
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
# dmesg | grep 00:05.0
         <---------output nothing

Expected results:
it should can do vfio-pci assgin the PF/VF to guest successfully.

Additional info:
# /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20131127.1_Server_x86_64.qcow2,if=none,id=drive-disk,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,vectors=0,bus=pci.0,addr=0x4,scsi=off,drive=drive-disk,id=system-disk,bootindex=1 -net none -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio -device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0

Comment 1 Sibiao Luo 2013-12-05 06:00:27 UTC
Created attachment 833015 [details]
guest_dmesg_log.

Comment 2 Sibiao Luo 2013-12-05 06:44:06 UTC
vfio-pci assgin VF did not hit this issue, so this issue only specified the PF.

-Generate the VFs.
# lspci | grep -i BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
# modprobe vfio; modprobe vfio_pci; modprobe vfio_iommu_type1

# echo 4 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs

# lspci | grep BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:01.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function

-Unbind the VF of BCM57810 from bnx2x and bind it to vfio-pci.
# lspci -n -s 0000:08:01.0 | awk '{ print $3 }'
14e4:16af
# echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:01.0 > /sys/bus/pci/devices/0000\:08\:01.0/driver/unbind
# echo 0000:08:01.0 > /sys/bus/pci/drivers/vfio-pci/bind

-Boot guest the vfio-pci assgin the VF to guest.
# /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection...-device vfio-pci,host=08:01.0,id=vf0
(qemu) info pci
...
  Bus  0, device   7, function 0:
    Ethernet controller: PCI device 14e4:16af
      BAR0: 64 bit prefetchable memory at 0xfe000000 [0xfe007fff].
      BAR4: 64 bit prefetchable memory at 0xfe008000 [0xfe009fff].
      id "vf0"
(qemu) info qtree
...
  dev: i440FX-pcihost, id ""
    irq 0
    bus: pci.0
      type PCI
      dev: vfio-pci, id "vf0"
        host = 0000:08:01.0
        x-intx-mmap-timeout-ms = 1100
        x-vga = off
        bootindex = -1
        addr = 07.0
        romfile = <null>
        rombar = 1
        multifunction = off
        command_serr_enable = on
        class Ethernet controller, addr 00:07.0, pci id 14e4:16af (sub 14e4:3106)
        bar 0: mem at 0xfe000000 [0xfe007fff]
        bar 4: mem at 0xfe008000 [0xfe009fff]
...

-Check the vfio-pci assigned work well in guest.
guest ]# lspci | grep BCM57810
00:07.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
guest ]# dmesg | grep 00:07.0
[    0.231287] pci 0000:00:07.0: [14e4:16af] type 00 class 0x020000
[    0.233003] pci 0000:00:07.0: reg 0x10: [mem 0xfe000000-0xfe007fff 64bit pref]
[    0.240003] pci 0000:00:07.0: reg 0x20: [mem 0xfe008000-0xfe009fff 64bit pref]
[    7.780319] bnx2x 0000:00:07.0: msix capability found
[    8.213705] bnx2x 0000:00:07.0: irq 43 for MSI/MSI-X
[    8.213843] bnx2x 0000:00:07.0: irq 44 for MSI/MSI-X

# ifconfig -a
ens7: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfe000000-fe007fff  
# ethtool -i ens7
driver: bnx2x
version: 1.78.17-0
firmware-version: bc 7.8.22
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Best Regards,
sluo

Comment 3 Bandan Das 2014-02-24 17:22:35 UTC
This is very likely a side effect of a inoperable card due to the rom behavior as mentioned in bug 1037956. A fix for it is being reviewed upstream.

*** This bug has been marked as a duplicate of bug 1037956 ***