Bug 1237034 - Error prompt while booting with vfio-pci device
Summary: Error prompt while booting with vfio-pci device
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.2
Hardware: ppc64le
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: David Gibson
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-30 08:58 UTC by Zhengtong
Modified: 2015-12-04 16:48 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-rhev-2.3.0-19.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-04 16:48:06 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2546 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2015-12-04 21:11:56 UTC

Description Zhengtong 2015-06-30 08:58:22 UTC
Description of problem:

While boot up guest with vfio-pci device , there will be error message: 
like "No such device"

Version-Release number of selected component (if applicable):

Host kernel: 
3.10.0-282.el7.ppc64le

[root@ibm-p8-kvm-02-qe ~]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.3.0 (qemu-kvm-rhev-2.3.0-5.el7), Copyright (c) 2003-2008 Fabrice Bellard


How reproducible:
100%

Steps to Reproduce:
1. load related modules:
#modprobe vfio
#modprobe vfio_spapr_eeh
#modprobe vfio_iommu_spapr_tce
#modprobe vfio_pci
2. unbind device from host and bind to vfio_pci bus
#lspci -n -s 0002:01:00.0
0002:01:00.0 0200: 10df:e220 (rev 10)
#lspci -n -s 0002:01:00.4
0002:01:00.4 0c04: 10df:e260 (rev 10)

echo "10df e220"  > /sys/bus/pci/drivers/vfio-pci/new_id

echo 0002:01:00.0 > /sys/bus/pci/devices/0002\:01\:00.0/driver/unbind
echo 0002:01:00.1 > /sys/bus/pci/devices/0002\:01\:00.1/driver/unbind
echo 0002:01:00.2 > /sys/bus/pci/devices/0002\:01\:00.2/driver/unbind
echo 0002:01:00.3 > /sys/bus/pci/devices/0002\:01\:00.3/driver/unbind
echo 0002:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.1 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.2 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.3 > /sys/bus/pci/drivers/vfio-pci/bind

echo "10df e260"  > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0002:01:00.4 > /sys/bus/pci/devices/0002\:01\:00.4/driver/unbind
echo 0002:01:00.5 > /sys/bus/pci/devices/0002\:01\:00.5/driver/unbind
echo 0002:01:00.4 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.5 > /sys/bus/pci/drivers/vfio-pci/bind



3. Boot up guest with vfio-pci device


/usr/libexec/qemu-kvm \
    -name vfio-test-xuhan \
    -machine pseries,accel=kvm,usb=off \
    -m 32768 \
    -realtime mlock=off \
    -cpu POWER8 \
    -smp 20,sockets=2,cores=10,threads=1 \
    -uuid 5125cf27-4b01-4493-b46d-734d08becc6b \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,path=monitor,server,nowait \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=utc \
    -boot strict=on \
    -device pci-ohci,id=usb,bus=pci.0,addr=0x1 \
    -device spapr-vscsi,id=scsi0,reg=0x2000 \
    -drive file=/home/xuhan/test/vfio_test_onppc/vdisk/RHEL-LE-7.1-20150219.1-Server-ppc64le.img,if=none,id=drive-scsi0-0-0-0,format=qcow2 \
    -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
    -drive file=/home/xuhan/test/vfio_test_onppc/iso/RHEL-LE-7.1-20150219.1-Server-ppc64le-dvd1.iso,if=none,id=cd_rom,media=cdrom,format=raw \
    -device scsi-cd,bus=scsi0.0,drive=cd_rom,id=cd_rom1,bootindex=2 \
    -netdev tap,id=hostnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=vnetvfio \
    -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5d:c7:9e,bus=pci.0,addr=0x2 \
    -chardev socket,id=charserial0,path=serial,server,nowait \
    -device spapr-vty,chardev=charserial0,reg=0x30000000 \
    -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1 \
    -device vfio-pci,host=0002:01:00.0,bus=vfiohost.0,addr=0x1 \
    -device usb-kbd,id=input0 \
    -device usb-mouse,id=input1 \
    -vnc 0.0.0.0:30 \
    -k en-us \
    -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x3 \
    -global spapr-nvram.reg=0x3000 \
    -monitor stdio

Actual results:

Guest could be boot up and there is target device shows up in guest. but there is error msg prompt while booting.
QEMU 2.3.0 monitor - type 'help' for more information

(qemu) qemu-kvm: -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1: Failed to create KVM VFIO device: No such device
qemu-kvm: -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1: Failed to create KVM VFIO device: No such device
qemu-kvm: -device vfio-pci,host=0002:01:00.0,bus=vfiohost.0,addr=0x1: Failed to create KVM VFIO device: No such device


Expected results:

Guest could be booted up without any error msg

Additional info:

Comment 2 David Gibson 2015-07-14 03:37:41 UTC
Adding Alex Williamson to CC, because I think his input will be useful.

Comment 3 David Gibson 2015-07-14 03:42:40 UTC
Ok, I've investigated this, and there seem to be 2 halves to the problem.

First, the vfio kvm device is built only on x86, not on powerpc.  AFAICT this device doesn't actually do anything yet, but is just a placeholder for future extensions.  It should work ok on powerpc as well, so it should probably be included there.

Second, qemu prints a scary warning when it's unable to create the vfio kvm device, even though this isn't a fatal error.  As you've seen the vfio device still works even with this error message.  We should probably tone down this error, which will appear both on powerpc and on older x86 kernels.

Both halves of this will require some upstream work.

Alex, can you confirm my observations above?  Do you any input on how to address either half of the problem?

Comment 4 Alex Williamson 2015-07-14 04:00:38 UTC
Sounds like a correct diagnosis.  On x86 the kvm-vfio device does actually do something, it registers vfio group fds with kvm, which can then use the vfio external user interface to determine whether the iommu managing the group promotes PCIe NoSnoop transactions to cache coherent.  This affects whether KVM fully emulates invalidate page instructions.  I'd therefore suggest that we do not really want to tone down the message on x86 as a failure to create the kvm-vfio device can indicate potential coherence issues with the VM.

It should be trivial to include support for the kvm-vfio device for power, it's already built for ARM and the kvm-arch callouts should be stubbed to make it just a little bit of baggage for future use.

Comment 5 David Gibson 2015-07-20 06:15:40 UTC
For the RHEL7.2 timeframe, it seems like the safer fix is just to suppress the error message in qemu.  I have a draft downstream patch to do this.

Zhengtong,  can you please try the kernel at https://brewweb.devel.redhat.com/taskinfo?taskID=9541733.  Our test systems don't have hardware suitable for vfio assignment.

If that works as expected I'll post downstream.

Meanwhile I'm working on an upstream patch to fix this on the kernel side.

Comment 6 David Gibson 2015-07-20 06:21:23 UTC
Upstream fix now posted

http://marc.info/?l=linux-kernel&m=143737274332400&w=2

Comment 7 Qunfang Zhang 2015-08-14 02:31:13 UTC
Hi, David

Seems we missed the requirement in comment 5. Could you help prepare another build for QE to test?  Sorry for that.

Thanks.

Comment 8 David Gibson 2015-08-14 03:46:36 UTC
Ok.

Please see http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9692144

Comment 9 Qunfang Zhang 2015-08-14 05:22:01 UTC
(In reply to David Gibson from comment #8)
> Ok.
> 
> Please see http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9692144

Thanks David.

Hi, Zhengtong

Could you help give a try with the above build to see whether the issue is gone? 

Thanks,
Qunfang

Comment 10 Zhengtong 2015-08-14 09:33:48 UTC
Hi, with the test version supplied by 
link http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9692144. the issue is gone.

[root@ibm-p8-kvm-02-qe vfio]# ./BE_guest_RHEL7.1-z.sh 
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) 

no. error prompt any more.

Comment 11 David Gibson 2015-08-18 06:07:47 UTC
Thanks for the information.

Comment 12 Miroslav Rezanina 2015-08-24 15:09:14 UTC
Fix included in qemu-kvm-rhev-2.3.0-19.el7

Comment 14 Xujun Ma 2015-09-01 08:33:42 UTC
Verified the issue on the latest version:

Version-Release number of selected component (if applicable):
Qemu-kvm-rhev: qemu-img-rhev-2.3.0-21.el7.ppc64le

Steps to Reproduce:
1. load related modules:
#modprobe vfio
#modprobe vfio_spapr_eeh
#modprobe vfio_iommu_spapr_tce
#modprobe vfio_pci
2. unbind device from host and bind to vfio_pci bus
#lspci -n -s 0002:01:00.0
0002:01:00.0 0200: 10df:e220 (rev 10)


echo "10df e220"  > /sys/bus/pci/drivers/vfio-pci/new_id

echo 0002:01:00.0 > /sys/bus/pci/devices/0002\:01\:00.0/driver/unbind
echo 0002:01:00.1 > /sys/bus/pci/devices/0002\:01\:00.1/driver/unbind
echo 0002:01:00.2 > /sys/bus/pci/devices/0002\:01\:00.2/driver/unbind
echo 0002:01:00.3 > /sys/bus/pci/devices/0002\:01\:00.3/driver/unbind
echo 0002:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.1 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.2 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.3 > /sys/bus/pci/drivers/vfio-pci/bind


3. Boot up guest with vfio-pci device


/usr/libexec/qemu-kvm \
    -name vfio-test-xuhan \
    -machine pseries,accel=kvm,usb=off \
    -m 32768 \
    -realtime mlock=off \
    -cpu POWER8 \
    -smp 20,sockets=2,cores=10,threads=1 \
    -uuid 5125cf27-4b01-4493-b46d-734d08becc6b \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,path=monitor,server,nowait \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=utc \
    -boot strict=on \
    -device pci-ohci,id=usb,bus=pci.0,addr=0x1 \
    -device spapr-vscsi,id=scsi0,reg=0x2000 \
    -drive file=/home/xuhan/test/vfio_test_onppc/vdisk/RHEL-LE-7.1-20150219.1-Server-ppc64le.img,if=none,id=drive-scsi0-0-0-0,format=qcow2 \
    -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
    -drive file=/home/xuhan/test/vfio_test_onppc/iso/RHEL-LE-7.1-20150219.1-Server-ppc64le-dvd1.iso,if=none,id=cd_rom,media=cdrom,format=raw \
    -device scsi-cd,bus=scsi0.0,drive=cd_rom,id=cd_rom1,bootindex=2 \
    -netdev tap,id=hostnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=vnetvfio \
    -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5d:c7:9e,bus=pci.0,addr=0x2 \
    -chardev socket,id=charserial0,path=serial,server,nowait \
    -device spapr-vty,chardev=charserial0,reg=0x30000000 \
    -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1 \
    -device vfio-pci,host=0002:01:00.0,bus=vfiohost.0,addr=0x1 \
    -device usb-kbd,id=input0 \
    -device usb-mouse,id=input1 \
    -vnc 0.0.0.0:30 \
    -k en-us \
    -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x3 \
    -global spapr-nvram.reg=0x3000 \
    -monitor stdio

Results:guest can be boot without any error message,and target device can be found in guest.

Comment 15 Xujun Ma 2015-09-02 09:08:35 UTC
Reproduced the issue on old version:

Version-Release number of selected component (if applicable):
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-5.ael7b.ppc64le.rpm


Steps to Reproduce:
1. load related modules:
#modprobe vfio
#modprobe vfio_spapr_eeh
#modprobe vfio_iommu_spapr_tce
#modprobe vfio_pci
2. unbind device from host and bind to vfio_pci bus
# lspci -n -s 0003:09:00.0
0003:09:00.0 0200: 14e4:1657 (rev 01)

echo "14e4 1657"  > /sys/bus/pci/drivers/vfio-pci/new_id

echo 0003:09:00.0 > /sys/bus/pci/devices/0003\:09\:00.0/driver/unbind
echo 0003:09:00.1 > /sys/bus/pci/devices/0003\:09\:00.1/driver/unbind
echo 0003:09:00.2 > /sys/bus/pci/devices/0003\:09\:00.2/driver/unbind
echo 0003:09:00.3 > /sys/bus/pci/devices/0003\:09\:00.3/driver/unbind


echo 0003:09:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0003:09:00.1 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0003:09:00.2 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0003:09:00.3 > /sys/bus/pci/drivers/vfio-pci/bind

3. Boot up guest with vfio-pci device
usr/libexec/qemu-kvm \
    -name vfio-test \
    -machine pseries,accel=kvm,usb=off \
    -m 32768 \
    -realtime mlock=off \
    -cpu POWER8 \
    -smp 20,sockets=2,cores=10,threads=1 \
    -uuid 5125cf27-4b01-4493-b46d-734d08becc6b \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,path=monitor,server,nowait \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=utc \
    -boot strict=on \
    -device pci-ohci,id=usb,bus=pci.0,addr=0x1 \
    -device spapr-vscsi,id=scsi0,reg=0x2000 \
    -drive file=/root/vm.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2 \
    -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0--
0,id=scsi0-0-0-0,bootindex=1 \
    -drive file=/root/RHEL-7.2-20150820.0-Server-ppc64le-dvd1.iso,if=none,id=cd__
rom,media=cdrom,format=raw \
    -device scsi-cd,bus=scsi0.0,drive=cd_rom,id=cd_rom1,bootindex=2 \
    -chardev socket,id=charserial0,path=serial,server,nowait \
    -device spapr-vty,chardev=charserial0,reg=0x30000000 \
    -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1 \
    -device vfio-pci,host=0003:09:00.0,bus=vfiohost.0,addr=0x1 \
    -device usb-kbd,id=input0 \
    -device usb-mouse,id=input1 \
    -vnc 0.0.0.0:30 \
    -k en-us \
    -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x3 \
    -global spapr-nvram.reg=0x3000 \
    -monitor stdio

Results:

root@ibm-p8-rhevm-05 ~]# sh vm.sh 
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) [41027.450651] pci 0003:09     : [PE# 002] Disabling 64-bit DMA bypass
[41027.450738] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.450995] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
qemu-kvm: -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1: Failed to create KVM VFIO device: No such device
[41027.451270] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.451327] pci 0003:09     : [PE# 002] Removing DMA window #1
[41027.453595] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
[41027.453729] pci 0003:09     : [PE# 002] Enabling 64-bit DMA bypass
[41027.473489] pci 0003:09     : [PE# 002] Disabling 64-bit DMA bypass
[41027.473567] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.473789] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
qemu-kvm: -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1: Failed to create KVM VFIO device: No such device
[41027.474079] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.474135] pci 0003:09     : [PE# 002] Removing DMA window #1
[41027.476385] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
[41027.476453] pci 0003:09     : [PE# 002] Enabling 64-bit DMA bypass
[41027.496925] pci 0003:09     : [PE# 002] Disabling 64-bit DMA bypass
[41027.497000] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.497279] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
qemu-kvm: -device vfio-pci,host=0003:09:00.0,bus=vfiohost.0,addr=0x1: Failed to create KVM VFIO device: No such device
[41027.651782] KVM guest htab at c000003ca8000000 (order 28), LPID 1
[41136.283292] hrtimer: interrupt took 6519 ns

There are some error mgs when boot guest.

Comment 17 errata-xmlrpc 2015-12-04 16:48:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html


Note You need to log in before you can comment on or make changes to this bug.