Bug 768857

what versions of qemu-kvm & libvirt did you test with ?

qemu-kvm-0.12.1.2-2.160.el6.x86_64
libvirt-0.8.7-18.el6.x86_64

After starting the guest and waiting a while I found this in syslog
kvm: 2550: cpu0 guest string pio down

Also the guest seems to be paused
virsh list --all
 Id Name                 State
----------------------------------
  1 rhel5-64-kvm         paused

I've manually upgraded the following packages to the 6.2 versions
glibc-2.12-1.47.el6.x86_64.rpm
glibc-common-2.12-1.47.el6.x86_64.rpm
glibc-devel-2.12-1.47.el6.x86_64.rpm
glibc-headers-2.12-1.47.el6.x86_64.rpm
libvirt-0.9.4-23.el6.x86_64.rpm
libvirt-client-0.9.4-23.el6.x86_64.rpm
libvirt-python-0.9.4-23.el6.x86_64.rpm
netcf-libs-0.1.9-2.el6.x86_64.rpm
qemu-img-0.12.1.2-2.209.el6.x86_64.rpm
qemu-kvm-0.12.1.2-2.209.el6.x86_64.rpm
sgabios-bin-0-0.3.20110621svn.el6.noarch.rpm
spice-server-0.8.2-5.el6.x86_64.rpm

Still the same, guest does not start.

Also when the guest switches from running to paused I sometimes get
single mode not supported
single mode not supported
level sensitive irq not supported
level sensitive irq not supported
in dmesg

I logged into the test machine; 04:00.[0,1] are the 82580's, not the 82576;
the latter are 02:00.[0,1].

root.bos.redhat.com:~> lspci | grep Ethernet
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)


So, if the description is correct, then 82576 do work, but 82580's do not.
I don't have experience with 82580's, but the problem could be reset-related.
To check, do a loop test that unbind's the driver to the 82580, resets it,
(do these steps by echo-ing to appropriate, device-specific sysfs files under /sys/bus/pci/devices/<BDF>/[driver/unbind,reset] ) and then re-bind the driver
back to the device.  if the device wedges after a number of these loops, then
the device has a reset/re-config issue (which is typical bug found while doing a device-assignment).

Created attachment 550878 [details]
pci-unbind-reset-bind.sh

Sorry Don my mistake. This problem here is the 82576 NIC not the 82580.
I tried what you suggested and did a unbind/reset/bind loop with 1000 iterations and that worked flawless.
Attaching the script I used.

To avoid any further misunderstanding the problem occurs with the following device 0000:02:00.0.

Very weird, after the whole unbind/reset/bind looping passthrough of both NICs seems to work. I tried several times now, including a reboot of the machine.

Don, should we close this and see if it happens again? I'd like to know why it didn't work before but I guess there's not much we can do now.

(In reply to comment #9)
> Very weird, after the whole unbind/reset/bind looping passthrough of both NICs
> seems to work. I tried several times now, including a reboot of the machine.
> 
> Don, should we close this and see if it happens again? I'd like to know why it
> didn't work before but I guess there's not much we can do now.

What does your kernel boot cmdline look like?
attach full boot-up dmesg log.

Created attachment 551533 [details]
dmesg.txt

root.bos.redhat.com:~> cat /proc/cmdline 
ro root=/dev/mapper/vg_dellpet41004-lv_root rd_LVM_LV=vg_dellpet41004/lv_root rd_LVM_LV=vg_dellpet41004/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200n81 ignore_loglevel no_console_suspend intel_iommu=on crashkernel=128M

I just tried to passthrough the 82576 to a guest and the guest first seemed stuck again but then started after 1-2 minutes... Don, I'll ping you so we can have a look at this together.

In /var/log/libvirt/libvirtd.log I found
06:27:04.486: 2044: error : daemonStreamEvent:208 : stream had I/O failure
06:27:04.486: 2044: error : virFDStreamUpdateCallback:111 : internal error stream is not open
06:27:05.437: 2044: error : qemuMonitorIO:583 : internal error End of file from monitor
06:35:45.510: 2044: error : daemonStreamHandleAbort:590 : stream aborted at client request
06:36:51.415: 2044: error : daemonStreamHandleAbort:590 : stream aborted at client request
06:47:13.709: 2044: error : daemonStreamEvent:208 : stream had I/O failure
06:47:13.710: 2044: error : virFDStreamUpdateCallback:111 : internal error stream is not open
06:47:14.775: 2044: error : qemuMonitorIO:583 : internal error End of file from monitor

(In reply to comment #12)
> I just tried to passthrough the 82576 to a guest and the guest first seemed
> stuck again but then started after 1-2 minutes... Don, I'll ping you so we can
> have a look at this together.
> 
> In /var/log/libvirt/libvirtd.log I found
> 06:27:04.486: 2044: error : daemonStreamEvent:208 : stream had I/O failure
> 06:27:04.486: 2044: error : virFDStreamUpdateCallback:111 : internal error
> stream is not open
> 06:27:05.437: 2044: error : qemuMonitorIO:583 : internal error End of file from
> monitor
> 06:35:45.510: 2044: error : daemonStreamHandleAbort:590 : stream aborted at
> client request
> 06:36:51.415: 2044: error : daemonStreamHandleAbort:590 : stream aborted at
> client request
> 06:47:13.709: 2044: error : daemonStreamEvent:208 : stream had I/O failure
> 06:47:13.710: 2044: error : virFDStreamUpdateCallback:111 : internal error
> stream is not open
> 06:47:14.775: 2044: error : qemuMonitorIO:583 : internal error End of file from
> monitor

With a device assigned to the guest, the guest is trying to do pxe boot
(can see this if you use virt-manager on host);  without the assigned device, 
no PXEboot -- I don't understand why that happens.
The 1->2 minute delay you are seeing is pxeboot waiting for a selection, and then times out to do a local boot.

I think I know what's going on now. The 82576 NIC has an option ROM for PXE boot. Now when the device is passed to the guest libvirt or kvm (not sure which) see this option ROM and decide to boot from it! The idea is not so bad but I don't see any option on virt-manager to disable this behaviour.
However I found the <rom bar='off'/> tag that I added to the xml manually and now the option ROM just gets ignored and everything works as expected.

I would suggest to add an option in virt-manager to disable any PCI option ROM detected for PCI devices that are passed to the guest.

Created attachment 555077 [details]
virt-manager.jpg

This might be a good place to add the "disable PCI option ROM" option.

Sounds like we should change the component to virt-manager.

Upstream now:

commit 82754ddc84041ece7a8462e1b14860eda4ea022b
Author: Cole Robinson <crobinso>
Date:   Mon Feb 10 14:24:22 2014 -0500

    Expose hostdev rombar in UI and cli (bz 768857)