This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 573850

Summary: PCI device assignment doesn't work out of the box, need to disable caps dropping
Product: [Fedora] Fedora Reporter: Sean Bruno <sbruno>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: low    
Version: 18CC: amturnip, berrange, bfay, clalance, crobinso, dallan, ddutile, gerd, hbrock, itamar, jforbes, laine, michael.hagmann, stephen_su, veillard, vinzstyle, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-07 16:43:35 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
dmidecode from a Lenovo T400
none
dmesg output with additional errors at the end of the log regarding PCI devices
none
Cleaner Dmesg output
none
Normal startup log of a F11 VM
none
Failure log when attempting to attach a PCI device none

Description Sean Bruno 2010-03-15 17:50:46 EDT
Description of problem:
Adding a PCI device to a virtual machine causes that virtual machine to be unable to startup

Version-Release number of selected component (if applicable):

libvirt-0.7.1-15.fc12.x86_64
libvirt-client-0.7.1-15.fc12.x86_64
libvirt-python-0.7.1-15.fc12.x86_64
python-virtinst-0.500.1-2.fc12.noarch
virt-manager-0.8.2-1.fc12.noarch

gpxe-roms-qemu-0.9.9-1.20091018git.fc12.noarch
qemu-common-0.11.0-13.fc12.x86_64
qemu-img-0.11.0-13.fc12.x86_64
qemu-system-x86-0.11.0-13.fc12.x86_64


How reproducible:
Every time

Steps to Reproduce:
1. Add a PCI device to a virtual machine (e.g. Wireless Card)
2. Attempt to startup virtual machine
  
Actual results:
Virtual Machine fails to start

Expected results:
Machine *should* start

Additional info:
Output Dialog from virt-manager:
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup
    self._backend.create()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 293, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Failure while reading monitor startup output: Connection reset by peer
Comment 1 Don Dutile 2010-03-15 18:42:43 EDT
Does the machine have an IOMMU for virtualization, i.e.,
Intel-VTd or AMD-IOMMU (not to be confused with GART-IOMMU)?

if so, is it enabled in the BIOS?

dmidecode dump & host boot log would be helpful.
Comment 2 Sean Bruno 2010-03-15 19:12:42 EDT
This is my "fancy" Lenovo T400 laptop, so all bet's are off.  Find the dmidecode output for your review.
Comment 3 Sean Bruno 2010-03-15 19:13:58 EDT
Created attachment 400328 [details]
dmidecode from a Lenovo T400
Comment 4 Sean Bruno 2010-03-15 19:14:32 EDT
Created attachment 400329 [details]
dmesg output with additional errors at the end of the log regarding PCI devices
Comment 5 Don Dutile 2010-03-16 17:34:55 EDT
From the dmesg log:

PCI-DMA: Intel(R) Virtualization Technology for Directed I/O


this implies VTd is enabled in bios & enabled in kernel.

what cmd did you use to start up the guest ?
Comment 6 Sean Bruno 2010-03-21 23:22:13 EDT
Not sure, to be honest.  I'm using the virtual machine manager so, it's not obvious what commands are being executed.

Virtualization *is* enabled in the BIOS.
Comment 7 Sean Bruno 2010-03-21 23:31:49 EDT
Created attachment 401626 [details]
Cleaner Dmesg output

Ok, this is a slightly cleaner version of the dmesg output from my system.

I attempted to attach the firewire controller on my laptop to the virtual machine and I received the same error from the virtual machine manager as before.
Comment 8 Cole Robinson 2010-04-05 12:29:10 EDT
Can you provide /var/log/libvirt/qemu/$your-vm.log? This should show the full KVM command line and any error output.
Comment 9 Sean Bruno 2010-04-07 18:26:25 EDT
Created attachment 405126 [details]
Normal startup log of a F11 VM
Comment 10 Sean Bruno 2010-04-07 18:26:53 EDT
Created attachment 405127 [details]
Failure log when attempting to attach a PCI device
Comment 11 Sean Bruno 2010-04-07 18:27:43 EDT
The failure log attached in Comment#10 is from my attempt to attach a PCI Firewire device to the VM
Comment 12 Gerd v. Egidy 2010-04-15 13:39:49 EDT
I have the same or a similar problem with FC13-beta (virt-manager-0.8.3-2.fc13.noarch):

the machine is vtd-enabled and I want to redirect a network card.

log output in /var/log/libvirt/qemu is like this:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M fedora-13 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name intranator -uuid 45364988-b8c2-4213-3ff0-ff35c1142122 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/intranator.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -drive file=/dev/vg_main/intragerd,if=none,id=drive-virtio-disk0,boot=on -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0 -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:4f:12:06,bus=pci.0,addr=0x5 -net tap,fd=58,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device pci-assign,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 
char device redirected to /dev/pts/1
get_real_device: /sys/bus/pci/devices/0000:07:00.0/config: Permission denied
pci-assign: Error: Couldn't get real device (hostdev0)!
Error initializing device pci-assign

my problem seems to be selinux-related as I see this in /var/log/messages:

kernel: type=1400 audit(1271352774.180:5): avc:  denied  { read } for  pid=7000 comm="qemu-kvm" name="0000:07:00.0" dev=sysfs ino=2369 scontext=system_u:system_r:svirt_t:s0:c224,c379 tcontext=system_u:object_r:sysfs_t:s0 tclass=lnk_file
Comment 13 Cole Robinson 2010-05-27 17:51:41 EDT
Gerd, if you can still reproduce that issue with the latest F13 packages (there is a libvirt update in updates-testing), please file a separate bug against F13 libvirt.

Sean's PCI error was:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.11 -m 2048 -smp 2 -name Fedora11 -uuid c220bf20-7718-870e-a092-b37bee9d3f6a -monitor unix:/var/lib/libvirt/qemu/Fedora11.monitor,server,nowait -boot c -drive file=/var/lib/libvirt/images/Fedora11.img,if=virtio,index=0,boot=on,format=raw -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=52:54:00:30:3b:c2,vlan=0,model=virtio,name=virtio.0 -net tap,fd=20,vlan=0,name=tap.0 -serial pty -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:0 -k en-us -vga cirrus -soundhw es1370 -pcidevice host=15:00.1 
char device redirected to /dev/pts/9
Failed to assign irq for "15:00.1": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?
Failed to initialize assigned device host=15:00.1

Sean, do you get that same error if you manually run qemu-kvm -pcidevice host=15:00.1 as root? Does disabling selinux with setenforce 0 make any difference when launching the VM via virt-manager?

Either way, this is not a virt-manager issue. Reassigning to libvirt for further triage.
Comment 14 Sean Bruno 2010-05-27 18:27:05 EDT
output from qemu-kvm -pcidevice host=15:00.1:

Failed to assign device "15:00.1" : Device or resource busy
Failed to deassign device "15:00.1" : Invalid argument
Failed to initialize assigned device host=15:00.1


Also noted this in the dmesg output:

firewire_ohci 0000:15:00.1: BAR 0: can't reserve mem region [0xf4301000-0xf43017ff]
kvm_vm_ioctl_assign_device: Could not get access to device regions
kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so cannot be deassigned
firewire_ohci 0000:15:00.1: BAR 0: can't reserve mem region [0xf4301000-0xf43017ff]
kvm_vm_ioctl_assign_device: Could not get access to device regions
kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so cannot be deassigned
Comment 15 Gerd v. Egidy 2010-05-27 18:37:21 EDT
I had to get vt-d to work so I did the following:
- switched to libvirt git from about two weeks ago
- run qemu-kvm as root
- added a patch to disable dropping the privileges before forking qemu-kvm (see my post to libvir-list@redhat.com)
- switched selinux to permissive mode

It's running now. I don't know which of these steps are still neccessary with the current updates or updates-testing. I probably will install a fresh F13 on a spare drive and be able to test it in a few days. Will report back then.
Comment 16 Cole Robinson 2010-05-27 18:41:11 EDT
Thanks Gerd. We definitely need a config option in libvirt to disable dropping emulator privs, so users can opt out (not to mention it's handy to diagnose things like this).
Comment 17 Sean Bruno 2010-05-27 18:45:37 EDT
also, here's my selinux status:

[root@home ~]# selinuxenabled ; echo $?
1
Comment 18 Cole Robinson 2010-07-12 15:37:10 EDT
Gerd, I was only able to get PCI device assignment working by following those steps as well. All other times gave me the error:

Failed to assign irq for "hostdev0": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?

Libvirt now has a config option for to change the capabilities dropping behavior, so rebuilding the package isn't required.. I've outlined the work around steps here:

https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#PCI_device_assignment

There has been some upstream qemu + libvirt work that works around the process capability issues: libvirt opens the PCI device FD and passes it to qemu. However, those patches likely won't be backported to F12 or F13, so I think this will only be properly fixed in F14 and beyond. Reassigning to rawhide.
Comment 19 Bug Zapper 2010-07-30 07:05:00 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 20 Fedora Admin XMLRPC Client 2011-09-22 13:51:16 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 21 Fedora Admin XMLRPC Client 2011-09-22 13:53:45 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 22 Fedora Admin XMLRPC Client 2011-09-22 13:59:34 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 23 amturnip 2011-11-26 12:13:15 EST
Virt-manager says 'Failed to assign device "hostdev0" : Operation not permitted' in Fedora 16.  Is it a symptom of the same problem?

$ rpm -q libvirt virt-manager qemu-kvm
libvirt-0.9.6-2.fc16.x86_64
virt-manager-0.9.0-7.fc16.noarch
qemu-kvm-0.15.1-3.fc16.x86_64

Virt-manager displays this message upon trying to start a virtual guest to which I had added a PCI Host Device:

Error starting domain: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
do_spice_init: starting 0.9.1
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
spice_server_add_interface: SPICE_INTERFACE_PLAYBACK
spice_server_add_interface: SPICE_INTERFACE_RECORD
Failed to assign device "hostdev0" : Operation not permitted
qemu-kvm: -device pci-assign,host=08:00.0,id=hostdev0,configfd=26,bus=pci.0,addr=0x8: Device 'pci-assign' could not be initialized

I think the host should be capable:

$ dmesg | grep Directed
[    1.287872] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O

The QEMU command line from /var/log/libvirt/qemu/$your-vm.log:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -S -M pc-0.14 -cpu qemu32 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name deb6 -uuid 5d29ba58-24aa-ea0f-11a5-88fcd18872d3 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/deb6.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/data/vm/disk/deb6,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:82:c9:f6,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -usb -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=08:00.0,id=hostdev0,configfd=26,bus=pci.0,addr=0x8 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Comment 24 Fedora Admin XMLRPC Client 2011-11-30 14:32:07 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 25 Fedora Admin XMLRPC Client 2011-11-30 14:35:36 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 26 Fedora Admin XMLRPC Client 2011-11-30 14:42:32 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 27 Fedora Admin XMLRPC Client 2011-11-30 14:53:28 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 28 Cole Robinson 2012-06-07 10:24:28 EDT
Moving to F16 given comment #23. But I'm curious, can anyone reproduce on F17?
Comment 29 Vincent Bolinard 2012-07-29 13:19:14 EDT
Hi,

I can reproduce on F17 :

Failed to assign device "hostdev0" : Device or resource busy
*** The driver 'pci-stub' is occupying your device 0000:08:06.0.
***
*** You can try the following commands to free it:
***
*** $ echo "1102 0002" > /sys/bus/pci/drivers/pci-stub/new_id
*** $ echo "0000:08:06.0" > /sys/bus/pci/drivers/pci-stub/unbind
*** $ echo "0000:08:06.0" > /sys/bus/pci/drivers/pci-stub/bind
*** $ echo "1102 0002" > /sys/bus/pci/drivers/pci-stub/remove_id
***
qemu-kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=24,bus=pci.0,addr=0x4: Device 'pci-assign' could not be initialized
Comment 30 stephen_su 2012-12-07 15:00:57 EST
I'm seeing the same on Centos 6.3. This is the output from libvirt log,

get_real_device: read failed, errno = 9
get_real_device: /sys/bus/pci/devices/0000:02:00.4/resource: Permission denied
qemu-kvm: -device pci-assign,host=02:00.4,id=hostdev0,configfd=25,bus=pci.0,addr=0x4: pci-assign: Error: Couldn't get real device (hostdev0)!
qemu-kvm: -device pci-assign,host=02:00.4,id=hostdev0,configfd=25,bus=pci.0,addr=0x4: Device 'pci-assign' could not be initialized

When I manually launch the VM, it looks like the error will not show up if the "configfg=" is not specify.
Comment 31 Fedora End Of Life 2013-01-16 17:06:09 EST
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 32 Cole Robinson 2013-01-27 15:37:08 EST
I still needed to drop capabilities, run as root:root, to get passthrough to work w/ F18 and my AMD box, so bumping target version.
Comment 33 Laine Stump 2013-02-07 16:43:35 EST
Whatever was the problem that required running as root and dropping caps when this bug was opened in 2010, it is no longer the case with F18.

(I know that the problem was fixed at one time, as it works properly in RHEL6.3/RHEL6.4, which is based on newer sources than the Fedora 12 this was reported against).

I've spent the last couple week diagnosing the current problem and writing a set of patches to overcome it. The problem now is caused by a *new* capability bit called CAP_COMPROMISE_KERNEL; if this capability isn't set in the qemu process, qemu will be unable to do setup a "pci-assign" device. CAP_COMPROMISE_KERNEL wasn't present in Fedora 17, but is in Fedora 18.

When I patch libvirt to properly set CAP_COMPROMISE_KERNEL in the qemu-kvm process, I can once again create pci passthrough devices without setting clear_emulator_capabilities=0 and without forcing qemu to run as root.

Since it's a separate issue though, I've filed a separate BZ: Bug 908888.