Adding Alex Williamson to CC, because I think his input will be useful.

Ok, I've investigated this, and there seem to be 2 halves to the problem.

First, the vfio kvm device is built only on x86, not on powerpc.  AFAICT this device doesn't actually do anything yet, but is just a placeholder for future extensions.  It should work ok on powerpc as well, so it should probably be included there.

Second, qemu prints a scary warning when it's unable to create the vfio kvm device, even though this isn't a fatal error.  As you've seen the vfio device still works even with this error message.  We should probably tone down this error, which will appear both on powerpc and on older x86 kernels.

Both halves of this will require some upstream work.

Alex, can you confirm my observations above?  Do you any input on how to address either half of the problem?

Sounds like a correct diagnosis.  On x86 the kvm-vfio device does actually do something, it registers vfio group fds with kvm, which can then use the vfio external user interface to determine whether the iommu managing the group promotes PCIe NoSnoop transactions to cache coherent.  This affects whether KVM fully emulates invalidate page instructions.  I'd therefore suggest that we do not really want to tone down the message on x86 as a failure to create the kvm-vfio device can indicate potential coherence issues with the VM.

It should be trivial to include support for the kvm-vfio device for power, it's already built for ARM and the kvm-arch callouts should be stubbed to make it just a little bit of baggage for future use.

For the RHEL7.2 timeframe, it seems like the safer fix is just to suppress the error message in qemu.  I have a draft downstream patch to do this.

Zhengtong,  can you please try the kernel at https://brewweb.devel.redhat.com/taskinfo?taskID=9541733.  Our test systems don't have hardware suitable for vfio assignment.

If that works as expected I'll post downstream.

Meanwhile I'm working on an upstream patch to fix this on the kernel side.

Upstream fix now posted

http://marc.info/?l=linux-kernel&m=143737274332400&w=2

Hi, David

Seems we missed the requirement in comment 5. Could you help prepare another build for QE to test?  Sorry for that.

Thanks.

Ok.

Please see http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9692144

(In reply to David Gibson from comment #8)
> Ok.
> 
> Please see http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9692144

Thanks David.

Hi, Zhengtong

Could you help give a try with the above build to see whether the issue is gone? 

Thanks,
Qunfang

Hi, with the test version supplied by 
link http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9692144. the issue is gone.

[root@ibm-p8-kvm-02-qe vfio]# ./BE_guest_RHEL7.1-z.sh 
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) 

no. error prompt any more.

Thanks for the information.

Fix included in qemu-kvm-rhev-2.3.0-19.el7

Verified the issue on the latest version:

Version-Release number of selected component (if applicable):
Qemu-kvm-rhev: qemu-img-rhev-2.3.0-21.el7.ppc64le

Steps to Reproduce:
1. load related modules:
#modprobe vfio
#modprobe vfio_spapr_eeh
#modprobe vfio_iommu_spapr_tce
#modprobe vfio_pci
2. unbind device from host and bind to vfio_pci bus
#lspci -n -s 0002:01:00.0
0002:01:00.0 0200: 10df:e220 (rev 10)


echo "10df e220"  > /sys/bus/pci/drivers/vfio-pci/new_id

echo 0002:01:00.0 > /sys/bus/pci/devices/0002\:01\:00.0/driver/unbind
echo 0002:01:00.1 > /sys/bus/pci/devices/0002\:01\:00.1/driver/unbind
echo 0002:01:00.2 > /sys/bus/pci/devices/0002\:01\:00.2/driver/unbind
echo 0002:01:00.3 > /sys/bus/pci/devices/0002\:01\:00.3/driver/unbind
echo 0002:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.1 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.2 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0002:01:00.3 > /sys/bus/pci/drivers/vfio-pci/bind


3. Boot up guest with vfio-pci device


/usr/libexec/qemu-kvm \
    -name vfio-test-xuhan \
    -machine pseries,accel=kvm,usb=off \
    -m 32768 \
    -realtime mlock=off \
    -cpu POWER8 \
    -smp 20,sockets=2,cores=10,threads=1 \
    -uuid 5125cf27-4b01-4493-b46d-734d08becc6b \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,path=monitor,server,nowait \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=utc \
    -boot strict=on \
    -device pci-ohci,id=usb,bus=pci.0,addr=0x1 \
    -device spapr-vscsi,id=scsi0,reg=0x2000 \
    -drive file=/home/xuhan/test/vfio_test_onppc/vdisk/RHEL-LE-7.1-20150219.1-Server-ppc64le.img,if=none,id=drive-scsi0-0-0-0,format=qcow2 \
    -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
    -drive file=/home/xuhan/test/vfio_test_onppc/iso/RHEL-LE-7.1-20150219.1-Server-ppc64le-dvd1.iso,if=none,id=cd_rom,media=cdrom,format=raw \
    -device scsi-cd,bus=scsi0.0,drive=cd_rom,id=cd_rom1,bootindex=2 \
    -netdev tap,id=hostnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=vnetvfio \
    -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5d:c7:9e,bus=pci.0,addr=0x2 \
    -chardev socket,id=charserial0,path=serial,server,nowait \
    -device spapr-vty,chardev=charserial0,reg=0x30000000 \
    -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1 \
    -device vfio-pci,host=0002:01:00.0,bus=vfiohost.0,addr=0x1 \
    -device usb-kbd,id=input0 \
    -device usb-mouse,id=input1 \
    -vnc 0.0.0.0:30 \
    -k en-us \
    -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x3 \
    -global spapr-nvram.reg=0x3000 \
    -monitor stdio

Results:guest can be boot without any error message,and target device can be found in guest.

Reproduced the issue on old version:

Version-Release number of selected component (if applicable):
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-5.ael7b.ppc64le.rpm


Steps to Reproduce:
1. load related modules:
#modprobe vfio
#modprobe vfio_spapr_eeh
#modprobe vfio_iommu_spapr_tce
#modprobe vfio_pci
2. unbind device from host and bind to vfio_pci bus
# lspci -n -s 0003:09:00.0
0003:09:00.0 0200: 14e4:1657 (rev 01)

echo "14e4 1657"  > /sys/bus/pci/drivers/vfio-pci/new_id

echo 0003:09:00.0 > /sys/bus/pci/devices/0003\:09\:00.0/driver/unbind
echo 0003:09:00.1 > /sys/bus/pci/devices/0003\:09\:00.1/driver/unbind
echo 0003:09:00.2 > /sys/bus/pci/devices/0003\:09\:00.2/driver/unbind
echo 0003:09:00.3 > /sys/bus/pci/devices/0003\:09\:00.3/driver/unbind


echo 0003:09:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0003:09:00.1 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0003:09:00.2 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0003:09:00.3 > /sys/bus/pci/drivers/vfio-pci/bind

3. Boot up guest with vfio-pci device
usr/libexec/qemu-kvm \
    -name vfio-test \
    -machine pseries,accel=kvm,usb=off \
    -m 32768 \
    -realtime mlock=off \
    -cpu POWER8 \
    -smp 20,sockets=2,cores=10,threads=1 \
    -uuid 5125cf27-4b01-4493-b46d-734d08becc6b \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,path=monitor,server,nowait \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=utc \
    -boot strict=on \
    -device pci-ohci,id=usb,bus=pci.0,addr=0x1 \
    -device spapr-vscsi,id=scsi0,reg=0x2000 \
    -drive file=/root/vm.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2 \
    -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0--
0,id=scsi0-0-0-0,bootindex=1 \
    -drive file=/root/RHEL-7.2-20150820.0-Server-ppc64le-dvd1.iso,if=none,id=cd__
rom,media=cdrom,format=raw \
    -device scsi-cd,bus=scsi0.0,drive=cd_rom,id=cd_rom1,bootindex=2 \
    -chardev socket,id=charserial0,path=serial,server,nowait \
    -device spapr-vty,chardev=charserial0,reg=0x30000000 \
    -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1 \
    -device vfio-pci,host=0003:09:00.0,bus=vfiohost.0,addr=0x1 \
    -device usb-kbd,id=input0 \
    -device usb-mouse,id=input1 \
    -vnc 0.0.0.0:30 \
    -k en-us \
    -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x3 \
    -global spapr-nvram.reg=0x3000 \
    -monitor stdio

Results:

root@ibm-p8-rhevm-05 ~]# sh vm.sh 
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) [41027.450651] pci 0003:09     : [PE# 002] Disabling 64-bit DMA bypass
[41027.450738] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.450995] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
qemu-kvm: -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1: Failed to create KVM VFIO device: No such device
[41027.451270] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.451327] pci 0003:09     : [PE# 002] Removing DMA window #1
[41027.453595] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
[41027.453729] pci 0003:09     : [PE# 002] Enabling 64-bit DMA bypass
[41027.473489] pci 0003:09     : [PE# 002] Disabling 64-bit DMA bypass
[41027.473567] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.473789] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
qemu-kvm: -device spapr-pci-vfio-host-bridge,id=vfiohost,iommu=1,index=0x1: Failed to create KVM VFIO device: No such device
[41027.474079] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.474135] pci 0003:09     : [PE# 002] Removing DMA window #1
[41027.476385] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
[41027.476453] pci 0003:09     : [PE# 002] Enabling 64-bit DMA bypass
[41027.496925] pci 0003:09     : [PE# 002] Disabling 64-bit DMA bypass
[41027.497000] pci 0003:09     : [PE# 002] Removing DMA window #0
[41027.497279] pci 0003:09     : [PE# 002] Setting up window#0 0..7fffffff pg=1000
qemu-kvm: -device vfio-pci,host=0003:09:00.0,bus=vfiohost.0,addr=0x1: Failed to create KVM VFIO device: No such device
[41027.651782] KVM guest htab at c000003ca8000000 (order 28), LPID 1
[41136.283292] hrtimer: interrupt took 6519 ns

There are some error mgs when boot guest.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html