Hide Forgot
Description of problem: boot guest with a PF vfio-pci assgined, excute 'info pci' in HMP monitor can output this device but fail to detect the vfio-pci assign PF/VF device in guest. Version-Release number of selected component (if applicable): host info: 3.10.0-57.el7.x86_64 qemu-kvm-1.5.3-20.el7.x86_64 seabios-1.7.2.2-4.el7.x86_64 guest info: 3.10.0-57.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Load vfio-pci, vfio, vfio_iommu_type1 modules. # lsmod | grep vfio vfio_pci 36474 0 vfio_iommu_type1 17636 0 vfio 20777 2 vfio_iommu_type1,vfio_pci 2.Check what other devices are in the same group as PF, unbind all of them and bind to vfio-pci. # lspci | grep -i BCM57810 08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) 08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) # readlink /sys/bus/pci/devices/0000:08:00.0/iommu_group ../../../../kernel/iommu_groups/14 # readlink /sys/bus/pci/devices/0000:08:00.1/iommu_group ../../../../kernel/iommu_groups/14 # lspci -n -s 0000:08:00.0 | awk '{ print $3 }' 14e4:168e # echo "14e4 168e" > /sys/bus/pci/drivers/vfio-pci/new_id # echo 0000:08:00.0 > /sys/bus/pci/devices/0000\:08\:00.0/driver/unbind # echo 0000:08:00.0 > /sys/bus/pci/drivers/vfio-pci/bind # lspci -n -s 0000:08:00.1 | awk '{ print $3 }' 14e4:168e # echo "14e4 168e" >> /sys/bus/pci/drivers/vfio-pci/new_id # echo 0000:08:00.1 >> /sys/bus/pci/devices/0000\:08\:00.1/driver/unbind # echo 0000:08:00.1 >> /sys/bus/pci/drivers/vfio-pci/bind 3.Do vfio-pci with PF to guest. e.g:...-device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0 4.info pci/qtree in HMP monitor and lspci/dmesg in guest. (qemu) info pci ... Bus 0, device 5, function 0: Class 65535: PCI device ffff:ffff IRQ 0. BAR0: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe]. BAR2: 64 bit prefetchable memory at 0xffffffffffffffff [0x007ffffe]. BAR4: 64 bit prefetchable memory at 0xffffffffffffffff [0x0000fffe]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe]. id "sluo_guest_nic_pf0" (qemu) info qtree ... dev: vfio-pci, id "sluo_guest_nic_pf0" host = 0000:08:00.0 x-intx-mmap-timeout-ms = 1100 x-vga = off bootindex = -1 addr = 05.0 romfile = <null> rombar = 1 multifunction = off command_serr_enable = on class Class ffff, addr 00:05.0, pci id ffff:ffff (sub ffff:ffff) bar 0: mem at 0xffffffffffffffff [0x7ffffe] bar 2: mem at 0xffffffffffffffff [0x7ffffe] bar 4: mem at 0xffffffffffffffff [0xfffe] bar 6: mem at 0xffffffffffffffff [0x1fffe] ... guest] # lspci guest] # dmesg | grep 00:05.0 Actual results: after step 4, fail to detect the vfio-pci assign PF/VF device in guest, I will paste the guest dmesg log later. # lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Communication controller: Red Hat, Inc Virtio console 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device # dmesg | grep 00:05.0 <---------output nothing Expected results: it should can do vfio-pci assgin the PF/VF to guest successfully. Additional info: # /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20131127.1_Server_x86_64.qcow2,if=none,id=drive-disk,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,vectors=0,bus=pci.0,addr=0x4,scsi=off,drive=drive-disk,id=system-disk,bootindex=1 -net none -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio -device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0
Created attachment 833015 [details] guest_dmesg_log.
vfio-pci assgin VF did not hit this issue, so this issue only specified the PF. -Generate the VFs. # lspci | grep -i BCM57810 08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) 08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) # modprobe vfio; modprobe vfio_pci; modprobe vfio_iommu_type1 # echo 4 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs # lspci | grep BCM57810 08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) 08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) 08:01.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function 08:01.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function 08:01.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function 08:01.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function -Unbind the VF of BCM57810 from bnx2x and bind it to vfio-pci. # lspci -n -s 0000:08:01.0 | awk '{ print $3 }' 14e4:16af # echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/new_id # echo 0000:08:01.0 > /sys/bus/pci/devices/0000\:08\:01.0/driver/unbind # echo 0000:08:01.0 > /sys/bus/pci/drivers/vfio-pci/bind -Boot guest the vfio-pci assgin the VF to guest. # /usr/libexec/qemu-kvm -M pc -cpu Opteron_G5 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection...-device vfio-pci,host=08:01.0,id=vf0 (qemu) info pci ... Bus 0, device 7, function 0: Ethernet controller: PCI device 14e4:16af BAR0: 64 bit prefetchable memory at 0xfe000000 [0xfe007fff]. BAR4: 64 bit prefetchable memory at 0xfe008000 [0xfe009fff]. id "vf0" (qemu) info qtree ... dev: i440FX-pcihost, id "" irq 0 bus: pci.0 type PCI dev: vfio-pci, id "vf0" host = 0000:08:01.0 x-intx-mmap-timeout-ms = 1100 x-vga = off bootindex = -1 addr = 07.0 romfile = <null> rombar = 1 multifunction = off command_serr_enable = on class Ethernet controller, addr 00:07.0, pci id 14e4:16af (sub 14e4:3106) bar 0: mem at 0xfe000000 [0xfe007fff] bar 4: mem at 0xfe008000 [0xfe009fff] ... -Check the vfio-pci assigned work well in guest. guest ]# lspci | grep BCM57810 00:07.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function guest ]# dmesg | grep 00:07.0 [ 0.231287] pci 0000:00:07.0: [14e4:16af] type 00 class 0x020000 [ 0.233003] pci 0000:00:07.0: reg 0x10: [mem 0xfe000000-0xfe007fff 64bit pref] [ 0.240003] pci 0000:00:07.0: reg 0x20: [mem 0xfe008000-0xfe009fff 64bit pref] [ 7.780319] bnx2x 0000:00:07.0: msix capability found [ 8.213705] bnx2x 0000:00:07.0: irq 43 for MSI/MSI-X [ 8.213843] bnx2x 0000:00:07.0: irq 44 for MSI/MSI-X # ifconfig -a ens7: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xfe000000-fe007fff # ethtool -i ens7 driver: bnx2x version: 1.78.17-0 firmware-version: bc 7.8.22 bus-info: 0000:00:07.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes Best Regards, sluo
This is very likely a side effect of a inoperable card due to the rom behavior as mentioned in bug 1037956. A fix for it is being reviewed upstream. *** This bug has been marked as a duplicate of bug 1037956 ***