| Summary: | fail to detect the vfio-pci assign with PF device in guest (Broadcom BCM57810 card) | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Sibiao Luo <sluo> | ||||
| Component: | qemu-kvm | Assignee: | Bandan Das <bdas> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 7.0 | CC: | acathrow, alex.williamson, chayang, hhuang, juzhang, michen, qzhang, virt-maint, xfu | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-02-24 17:22:35 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
Created attachment 833015 [details]
guest_dmesg_log.
vfio-pci assgin VF did not hit this issue, so this issue only specified the PF.
-Generate the VFs.
# lspci | grep -i BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
# modprobe vfio; modprobe vfio_pci; modprobe vfio_iommu_type1
# echo 4 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs
# lspci | grep BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:01.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
08:01.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
-Unbind the VF of BCM57810 from bnx2x and bind it to vfio-pci.
# lspci -n -s 0000:08:01.0 | awk '{ print $3 }'
14e4:16af
# echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:01.0 > /sys/bus/pci/devices/0000\:08\:01.0/driver/unbind
# echo 0000:08:01.0 > /sys/bus/pci/drivers/vfio-pci/bind
-Boot guest the vfio-pci assgin the VF to guest.
# /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection...-device vfio-pci,host=08:01.0,id=vf0
(qemu) info pci
...
Bus 0, device 7, function 0:
Ethernet controller: PCI device 14e4:16af
BAR0: 64 bit prefetchable memory at 0xfe000000 [0xfe007fff].
BAR4: 64 bit prefetchable memory at 0xfe008000 [0xfe009fff].
id "vf0"
(qemu) info qtree
...
dev: i440FX-pcihost, id ""
irq 0
bus: pci.0
type PCI
dev: vfio-pci, id "vf0"
host = 0000:08:01.0
x-intx-mmap-timeout-ms = 1100
x-vga = off
bootindex = -1
addr = 07.0
romfile = <null>
rombar = 1
multifunction = off
command_serr_enable = on
class Ethernet controller, addr 00:07.0, pci id 14e4:16af (sub 14e4:3106)
bar 0: mem at 0xfe000000 [0xfe007fff]
bar 4: mem at 0xfe008000 [0xfe009fff]
...
-Check the vfio-pci assigned work well in guest.
guest ]# lspci | grep BCM57810
00:07.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
guest ]# dmesg | grep 00:07.0
[ 0.231287] pci 0000:00:07.0: [14e4:16af] type 00 class 0x020000
[ 0.233003] pci 0000:00:07.0: reg 0x10: [mem 0xfe000000-0xfe007fff 64bit pref]
[ 0.240003] pci 0000:00:07.0: reg 0x20: [mem 0xfe008000-0xfe009fff 64bit pref]
[ 7.780319] bnx2x 0000:00:07.0: msix capability found
[ 8.213705] bnx2x 0000:00:07.0: irq 43 for MSI/MSI-X
[ 8.213843] bnx2x 0000:00:07.0: irq 44 for MSI/MSI-X
# ifconfig -a
ens7: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfe000000-fe007fff
# ethtool -i ens7
driver: bnx2x
version: 1.78.17-0
firmware-version: bc 7.8.22
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Best Regards,
sluo
This is very likely a side effect of a inoperable card due to the rom behavior as mentioned in bug 1037956. A fix for it is being reviewed upstream. *** This bug has been marked as a duplicate of bug 1037956 *** |
Description of problem: boot guest with a PF vfio-pci assgined, excute 'info pci' in HMP monitor can output this device but fail to detect the vfio-pci assign PF/VF device in guest. Version-Release number of selected component (if applicable): host info: 3.10.0-57.el7.x86_64 qemu-kvm-1.5.3-20.el7.x86_64 seabios-1.7.2.2-4.el7.x86_64 guest info: 3.10.0-57.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Load vfio-pci, vfio, vfio_iommu_type1 modules. # lsmod | grep vfio vfio_pci 36474 0 vfio_iommu_type1 17636 0 vfio 20777 2 vfio_iommu_type1,vfio_pci 2.Check what other devices are in the same group as PF, unbind all of them and bind to vfio-pci. # lspci | grep -i BCM57810 08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) 08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) # readlink /sys/bus/pci/devices/0000:08:00.0/iommu_group ../../../../kernel/iommu_groups/14 # readlink /sys/bus/pci/devices/0000:08:00.1/iommu_group ../../../../kernel/iommu_groups/14 # lspci -n -s 0000:08:00.0 | awk '{ print $3 }' 14e4:168e # echo "14e4 168e" > /sys/bus/pci/drivers/vfio-pci/new_id # echo 0000:08:00.0 > /sys/bus/pci/devices/0000\:08\:00.0/driver/unbind # echo 0000:08:00.0 > /sys/bus/pci/drivers/vfio-pci/bind # lspci -n -s 0000:08:00.1 | awk '{ print $3 }' 14e4:168e # echo "14e4 168e" >> /sys/bus/pci/drivers/vfio-pci/new_id # echo 0000:08:00.1 >> /sys/bus/pci/devices/0000\:08\:00.1/driver/unbind # echo 0000:08:00.1 >> /sys/bus/pci/drivers/vfio-pci/bind 3.Do vfio-pci with PF to guest. e.g:...-device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0 4.info pci/qtree in HMP monitor and lspci/dmesg in guest. (qemu) info pci ... Bus 0, device 5, function 0: Class 65535: PCI device ffff:ffff IRQ 0. BAR0: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe]. BAR2: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe]. BAR4: 64 bit prefetchable memory at 0xffffffffffffffff [0x0000fffe]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe]. id "sluo_guest_nic_pf0" (qemu) info qtree ... dev: vfio-pci, id "sluo_guest_nic_pf0" host = 0000:08:00.0 x-intx-mmap-timeout-ms = 1100 x-vga = off bootindex = -1 addr = 05.0 romfile = <null> rombar = 1 multifunction = off command_serr_enable = on class Class ffff, addr 00:05.0, pci id ffff:ffff (sub ffff:ffff) bar 0: mem at 0xffffffffffffffff [0x7ffffe] bar 2: mem at 0xffffffffffffffff [0x7ffffe] bar 4: mem at 0xffffffffffffffff [0xfffe] bar 6: mem at 0xffffffffffffffff [0x1fffe] ... guest] # lspci guest] # dmesg | grep 00:05.0 Actual results: after step 4, fail to detect the vfio-pci assign PF/VF device in guest, I will paste the guest dmesg log later. # lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Communication controller: Red Hat, Inc Virtio console 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device # dmesg | grep 00:05.0 <---------output nothing Expected results: it should can do vfio-pci assgin the PF/VF to guest successfully. Additional info: # /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20131127.1_Server_x86_64.qcow2,if=none,id=drive-disk,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,vectors=0,bus=pci.0,addr=0x4,scsi=off,drive=drive-disk,id=system-disk,bootindex=1 -net none -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio -device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0