Description of problem: Adding a PCI device to a virtual machine causes that virtual machine to be unable to startup Version-Release number of selected component (if applicable): libvirt-0.7.1-15.fc12.x86_64 libvirt-client-0.7.1-15.fc12.x86_64 libvirt-python-0.7.1-15.fc12.x86_64 python-virtinst-0.500.1-2.fc12.noarch virt-manager-0.8.2-1.fc12.noarch gpxe-roms-qemu-0.9.9-1.20091018git.fc12.noarch qemu-common-0.11.0-13.fc12.x86_64 qemu-img-0.11.0-13.fc12.x86_64 qemu-system-x86-0.11.0-13.fc12.x86_64 How reproducible: Every time Steps to Reproduce: 1. Add a PCI device to a virtual machine (e.g. Wireless Card) 2. Attempt to startup virtual machine Actual results: Virtual Machine fails to start Expected results: Machine *should* start Additional info: Output Dialog from virt-manager: Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain vm.startup() File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup self._backend.create() File "/usr/lib64/python2.6/site-packages/libvirt.py", line 293, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: Failure while reading monitor startup output: Connection reset by peer
Does the machine have an IOMMU for virtualization, i.e., Intel-VTd or AMD-IOMMU (not to be confused with GART-IOMMU)? if so, is it enabled in the BIOS? dmidecode dump & host boot log would be helpful.
This is my "fancy" Lenovo T400 laptop, so all bet's are off. Find the dmidecode output for your review.
Created attachment 400328 [details] dmidecode from a Lenovo T400
Created attachment 400329 [details] dmesg output with additional errors at the end of the log regarding PCI devices
From the dmesg log: PCI-DMA: Intel(R) Virtualization Technology for Directed I/O this implies VTd is enabled in bios & enabled in kernel. what cmd did you use to start up the guest ?
Not sure, to be honest. I'm using the virtual machine manager so, it's not obvious what commands are being executed. Virtualization *is* enabled in the BIOS.
Created attachment 401626 [details] Cleaner Dmesg output Ok, this is a slightly cleaner version of the dmesg output from my system. I attempted to attach the firewire controller on my laptop to the virtual machine and I received the same error from the virtual machine manager as before.
Can you provide /var/log/libvirt/qemu/$your-vm.log? This should show the full KVM command line and any error output.
Created attachment 405126 [details] Normal startup log of a F11 VM
Created attachment 405127 [details] Failure log when attempting to attach a PCI device
The failure log attached in Comment#10 is from my attempt to attach a PCI Firewire device to the VM
I have the same or a similar problem with FC13-beta (virt-manager-0.8.3-2.fc13.noarch): the machine is vtd-enabled and I want to redirect a network card. log output in /var/log/libvirt/qemu is like this: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M fedora-13 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name intranator -uuid 45364988-b8c2-4213-3ff0-ff35c1142122 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/intranator.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -drive file=/dev/vg_main/intragerd,if=none,id=drive-virtio-disk0,boot=on -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0 -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:4f:12:06,bus=pci.0,addr=0x5 -net tap,fd=58,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device pci-assign,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 char device redirected to /dev/pts/1 get_real_device: /sys/bus/pci/devices/0000:07:00.0/config: Permission denied pci-assign: Error: Couldn't get real device (hostdev0)! Error initializing device pci-assign my problem seems to be selinux-related as I see this in /var/log/messages: kernel: type=1400 audit(1271352774.180:5): avc: denied { read } for pid=7000 comm="qemu-kvm" name="0000:07:00.0" dev=sysfs ino=2369 scontext=system_u:system_r:svirt_t:s0:c224,c379 tcontext=system_u:object_r:sysfs_t:s0 tclass=lnk_file
Gerd, if you can still reproduce that issue with the latest F13 packages (there is a libvirt update in updates-testing), please file a separate bug against F13 libvirt. Sean's PCI error was: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.11 -m 2048 -smp 2 -name Fedora11 -uuid c220bf20-7718-870e-a092-b37bee9d3f6a -monitor unix:/var/lib/libvirt/qemu/Fedora11.monitor,server,nowait -boot c -drive file=/var/lib/libvirt/images/Fedora11.img,if=virtio,index=0,boot=on,format=raw -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=52:54:00:30:3b:c2,vlan=0,model=virtio,name=virtio.0 -net tap,fd=20,vlan=0,name=tap.0 -serial pty -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:0 -k en-us -vga cirrus -soundhw es1370 -pcidevice host=15:00.1 char device redirected to /dev/pts/9 Failed to assign irq for "15:00.1": Operation not permitted Perhaps you are assigning a device that shares an IRQ with another device? Failed to initialize assigned device host=15:00.1 Sean, do you get that same error if you manually run qemu-kvm -pcidevice host=15:00.1 as root? Does disabling selinux with setenforce 0 make any difference when launching the VM via virt-manager? Either way, this is not a virt-manager issue. Reassigning to libvirt for further triage.
output from qemu-kvm -pcidevice host=15:00.1: Failed to assign device "15:00.1" : Device or resource busy Failed to deassign device "15:00.1" : Invalid argument Failed to initialize assigned device host=15:00.1 Also noted this in the dmesg output: firewire_ohci 0000:15:00.1: BAR 0: can't reserve mem region [0xf4301000-0xf43017ff] kvm_vm_ioctl_assign_device: Could not get access to device regions kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so cannot be deassigned firewire_ohci 0000:15:00.1: BAR 0: can't reserve mem region [0xf4301000-0xf43017ff] kvm_vm_ioctl_assign_device: Could not get access to device regions kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so cannot be deassigned
I had to get vt-d to work so I did the following: - switched to libvirt git from about two weeks ago - run qemu-kvm as root - added a patch to disable dropping the privileges before forking qemu-kvm (see my post to libvir-list) - switched selinux to permissive mode It's running now. I don't know which of these steps are still neccessary with the current updates or updates-testing. I probably will install a fresh F13 on a spare drive and be able to test it in a few days. Will report back then.
Thanks Gerd. We definitely need a config option in libvirt to disable dropping emulator privs, so users can opt out (not to mention it's handy to diagnose things like this).
also, here's my selinux status: [root@home ~]# selinuxenabled ; echo $? 1
Gerd, I was only able to get PCI device assignment working by following those steps as well. All other times gave me the error: Failed to assign irq for "hostdev0": Operation not permitted Perhaps you are assigning a device that shares an IRQ with another device? Libvirt now has a config option for to change the capabilities dropping behavior, so rebuilding the package isn't required.. I've outlined the work around steps here: https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#PCI_device_assignment There has been some upstream qemu + libvirt work that works around the process capability issues: libvirt opens the PCI device FD and passes it to qemu. However, those patches likely won't be backported to F12 or F13, so I think this will only be properly fixed in F14 and beyond. Reassigning to rawhide.
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle. Changing version to '14'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Virt-manager says 'Failed to assign device "hostdev0" : Operation not permitted' in Fedora 16. Is it a symptom of the same problem? $ rpm -q libvirt virt-manager qemu-kvm libvirt-0.9.6-2.fc16.x86_64 virt-manager-0.9.0-7.fc16.noarch qemu-kvm-0.15.1-3.fc16.x86_64 Virt-manager displays this message upon trying to start a virtual guest to which I had added a PCI Host Device: Error starting domain: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1 do_spice_init: starting 0.9.1 spice_server_add_interface: SPICE_INTERFACE_KEYBOARD spice_server_add_interface: SPICE_INTERFACE_MOUSE spice_server_add_interface: SPICE_INTERFACE_PLAYBACK spice_server_add_interface: SPICE_INTERFACE_RECORD Failed to assign device "hostdev0" : Operation not permitted qemu-kvm: -device pci-assign,host=08:00.0,id=hostdev0,configfd=26,bus=pci.0,addr=0x8: Device 'pci-assign' could not be initialized I think the host should be capable: $ dmesg | grep Directed [ 1.287872] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O The QEMU command line from /var/log/libvirt/qemu/$your-vm.log: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -S -M pc-0.14 -cpu qemu32 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name deb6 -uuid 5d29ba58-24aa-ea0f-11a5-88fcd18872d3 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/deb6.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/data/vm/disk/deb6,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:82:c9:f6,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -usb -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=08:00.0,id=hostdev0,configfd=26,bus=pci.0,addr=0x8 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Moving to F16 given comment #23. But I'm curious, can anyone reproduce on F17?
Hi, I can reproduce on F17 : Failed to assign device "hostdev0" : Device or resource busy *** The driver 'pci-stub' is occupying your device 0000:08:06.0. *** *** You can try the following commands to free it: *** *** $ echo "1102 0002" > /sys/bus/pci/drivers/pci-stub/new_id *** $ echo "0000:08:06.0" > /sys/bus/pci/drivers/pci-stub/unbind *** $ echo "0000:08:06.0" > /sys/bus/pci/drivers/pci-stub/bind *** $ echo "1102 0002" > /sys/bus/pci/drivers/pci-stub/remove_id *** qemu-kvm: -device pci-assign,host=08:06.0,id=hostdev0,configfd=24,bus=pci.0,addr=0x4: Device 'pci-assign' could not be initialized
I'm seeing the same on Centos 6.3. This is the output from libvirt log, get_real_device: read failed, errno = 9 get_real_device: /sys/bus/pci/devices/0000:02:00.4/resource: Permission denied qemu-kvm: -device pci-assign,host=02:00.4,id=hostdev0,configfd=25,bus=pci.0,addr=0x4: pci-assign: Error: Couldn't get real device (hostdev0)! qemu-kvm: -device pci-assign,host=02:00.4,id=hostdev0,configfd=25,bus=pci.0,addr=0x4: Device 'pci-assign' could not be initialized When I manually launch the VM, it looks like the error will not show up if the "configfg=" is not specify.
This message is a reminder that Fedora 16 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 16. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '16'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 16's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 16 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged to click on "Clone This Bug" and open it against that version of Fedora. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
I still needed to drop capabilities, run as root:root, to get passthrough to work w/ F18 and my AMD box, so bumping target version.
Whatever was the problem that required running as root and dropping caps when this bug was opened in 2010, it is no longer the case with F18. (I know that the problem was fixed at one time, as it works properly in RHEL6.3/RHEL6.4, which is based on newer sources than the Fedora 12 this was reported against). I've spent the last couple week diagnosing the current problem and writing a set of patches to overcome it. The problem now is caused by a *new* capability bit called CAP_COMPROMISE_KERNEL; if this capability isn't set in the qemu process, qemu will be unable to do setup a "pci-assign" device. CAP_COMPROMISE_KERNEL wasn't present in Fedora 17, but is in Fedora 18. When I patch libvirt to properly set CAP_COMPROMISE_KERNEL in the qemu-kvm process, I can once again create pci passthrough devices without setting clear_emulator_capabilities=0 and without forcing qemu to run as root. Since it's a separate issue though, I've filed a separate BZ: Bug 908888.