Bug 1038424 - fail to detect the vfio-pci assign with PF device in guest (Broadcom BCM57810 card)
Summary: fail to detect the vfio-pci assign with PF device in guest (Broadcom BCM57810...
Keywords:
Status: CLOSED DUPLICATE of bug 1037956
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Bandan Das
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-05 05:57 UTC by Sibiao Luo
Modified: 2014-02-24 17:22 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-24 17:22:35 UTC
Target Upstream Version:


Attachments (Terms of Use)
guest_dmesg_log. (32.96 KB, text/plain)
2013-12-05 06:00 UTC, Sibiao Luo
no flags Details

Description Sibiao Luo 2013-12-05 05:57:02 UTC
Description of problem:
boot guest with a PF vfio-pci assgined, excute 'info pci' in HMP monitor can output this device but fail to detect the vfio-pci assign PF/VF device in guest.

Version-Release number of selected component (if applicable):
host info:
3.10.0-57.el7.x86_64
qemu-kvm-1.5.3-20.el7.x86_64
seabios-1.7.2.2-4.el7.x86_64
guest info:
3.10.0-57.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Load vfio-pci, vfio, vfio_iommu_type1 modules.
# lsmod | grep vfio
vfio_pci               36474  0 
vfio_iommu_type1       17636  0 
vfio                   20777  2 vfio_iommu_type1,vfio_pci

2.Check what other devices are in the same group as PF, unbind all of them and bind to vfio-pci.
# lspci | grep -i BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
# readlink /sys/bus/pci/devices/0000:08:00.0/iommu_group
../../../../kernel/iommu_groups/14
# readlink /sys/bus/pci/devices/0000:08:00.1/iommu_group
../../../../kernel/iommu_groups/14
# lspci -n -s 0000:08:00.0 | awk '{ print $3 }'
14e4:168e
# echo "14e4 168e" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:00.0 > /sys/bus/pci/devices/0000\:08\:00.0/driver/unbind 
# echo 0000:08:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
# lspci -n -s 0000:08:00.1 | awk '{ print $3 }'
14e4:168e
# echo "14e4 168e" >> /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:00.1 >> /sys/bus/pci/devices/0000\:08\:00.1/driver/unbind 
# echo 0000:08:00.1 >> /sys/bus/pci/drivers/vfio-pci/bind

3.Do vfio-pci with PF to guest.
e.g:...-device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0

4.info pci/qtree in HMP monitor and lspci/dmesg in guest.
(qemu) info pci
...
  Bus  0, device   5, function 0:
    Class 65535: PCI device ffff:ffff
      IRQ 0.
      BAR0: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe].
      BAR2: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe].
      BAR4: 64 bit prefetchable memory at 0xffffffffffffffff [0x0000fffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "sluo_guest_nic_pf0"
(qemu) info qtree
...
      dev: vfio-pci, id "sluo_guest_nic_pf0"
        host = 0000:08:00.0
        x-intx-mmap-timeout-ms = 1100
        x-vga = off
        bootindex = -1
        addr = 05.0
        romfile = <null>
        rombar = 1
        multifunction = off
        command_serr_enable = on
        class Class ffff, addr 00:05.0, pci id ffff:ffff (sub ffff:ffff)
        bar 0: mem at 0xffffffffffffffff [0x7ffffe]
        bar 2: mem at 0xffffffffffffffff [0x7ffffe]
        bar 4: mem at 0xffffffffffffffff [0xfffe]
        bar 6: mem at 0xffffffffffffffff [0x1fffe]
...
guest] # lspci
guest] # dmesg | grep 00:05.0

Actual results:
after step 4, fail to detect the vfio-pci assign PF/VF device in guest, I will paste the guest dmesg log later.
# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Communication controller: Red Hat, Inc Virtio console
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
# dmesg | grep 00:05.0
         <---------output nothing

Expected results:
it should can do vfio-pci assgin the PF/VF to guest successfully.

Additional info:
# /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20131127.1_Server_x86_64.qcow2,if=none,id=drive-disk,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,vectors=0,bus=pci.0,addr=0x4,scsi=off,drive=drive-disk,id=system-disk,bootindex=1 -net none -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio -device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0

Comment 1 Sibiao Luo 2013-12-05 06:00:27 UTC
Created attachment 833015 [details]
guest_dmesg_log.

Comment 2 Sibiao Luo 2013-12-05 06:44:06 UTC
vfio-pci assgin VF did not hit this issue, so this issue only specified the PF.

-Generate the VFs.
# lspci | grep -i BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
# modprobe vfio; modprobe vfio_pci; modprobe vfio_iommu_type1

# echo 4 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs

# lspci | grep BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:01.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function

-Unbind the VF of BCM57810 from bnx2x and bind it to vfio-pci.
# lspci -n -s 0000:08:01.0 | awk '{ print $3 }'
14e4:16af
# echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:01.0 > /sys/bus/pci/devices/0000\:08\:01.0/driver/unbind
# echo 0000:08:01.0 > /sys/bus/pci/drivers/vfio-pci/bind

-Boot guest the vfio-pci assgin the VF to guest.
# /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection...-device vfio-pci,host=08:01.0,id=vf0
(qemu) info pci
...
  Bus  0, device   7, function 0:
    Ethernet controller: PCI device 14e4:16af
      BAR0: 64 bit prefetchable memory at 0xfe000000 [0xfe007fff].
      BAR4: 64 bit prefetchable memory at 0xfe008000 [0xfe009fff].
      id "vf0"
(qemu) info qtree
...
  dev: i440FX-pcihost, id ""
    irq 0
    bus: pci.0
      type PCI
      dev: vfio-pci, id "vf0"
        host = 0000:08:01.0
        x-intx-mmap-timeout-ms = 1100
        x-vga = off
        bootindex = -1
        addr = 07.0
        romfile = <null>
        rombar = 1
        multifunction = off
        command_serr_enable = on
        class Ethernet controller, addr 00:07.0, pci id 14e4:16af (sub 14e4:3106)
        bar 0: mem at 0xfe000000 [0xfe007fff]
        bar 4: mem at 0xfe008000 [0xfe009fff]
...

-Check the vfio-pci assigned work well in guest.
guest ]# lspci | grep BCM57810
00:07.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
guest ]# dmesg | grep 00:07.0
[    0.231287] pci 0000:00:07.0: [14e4:16af] type 00 class 0x020000
[    0.233003] pci 0000:00:07.0: reg 0x10: [mem 0xfe000000-0xfe007fff 64bit pref]
[    0.240003] pci 0000:00:07.0: reg 0x20: [mem 0xfe008000-0xfe009fff 64bit pref]
[    7.780319] bnx2x 0000:00:07.0: msix capability found
[    8.213705] bnx2x 0000:00:07.0: irq 43 for MSI/MSI-X
[    8.213843] bnx2x 0000:00:07.0: irq 44 for MSI/MSI-X

# ifconfig -a
ens7: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfe000000-fe007fff  
# ethtool -i ens7
driver: bnx2x
version: 1.78.17-0
firmware-version: bc 7.8.22
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Best Regards,
sluo

Comment 3 Bandan Das 2014-02-24 17:22:35 UTC
This is very likely a side effect of a inoperable card due to the rom behavior as mentioned in bug 1037956. A fix for it is being reviewed upstream.

*** This bug has been marked as a duplicate of bug 1037956 ***


Note You need to log in before you can comment on or make changes to this bug.