Red Hat Bugzilla – Bug 506923
[RHEL5.4 Xen]: AMD PCI passthrough failed with Xen FV guest
Last modified: 2009-12-14 16:27:01 EST
Description of problem: Can not boot xen fv guest with pci passthrough config. Version-Release number of selected component (if applicable): libvirt 0.6.3-10.el5 rhel-5.4 (2.6.18-154.el5xen) How reproducible: Always Steps to Reproduce: # virsh nodedev-dettach pci_14e4_167a Device pci_14e4_167a dettached # virsh nodedev-reset pci_14e4_167a Device pci_14e4_167a reset # virsh edit demo Domain demo XML configuration edited. <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address bus='0x02' slot='0x00' function='0x00'/> </source> </hostdev> # virsh start demo error: Failed to start domain demo error: POST operation failed: xend_post: error from xen daemon: (xend.err "Error creating domain: failed to assign device: maybe the platform doesn't support VT-d, or VT-d isn't enabled properly?") Actual results: Failed to start guest domain. Expected results: Guest domain can be started with pci passthrough config. Additional info: It's fine on Xen PV guest and KVM
Unlikely to be a libvirt bug, since the error is coming from XenD and indicates XenD has understood the PCI config, but simply doesn't support it. Can you provide the output from 'dmesg' on the host OS immediately after it has finished booting.
Can you also provide the 'xm dmesg' output, since we need to see the hypervisor logs too
Created attachment 348633 [details] dmesg info on the host os
Created attachment 348635 [details] xm dmesg info
The reason you get the error message from Xen, is that it does not have VT-D support active. This appears to be confirmed by the hypervisor dmesg logs (XEN) I/O virtualisation disabled I think you need to reboot, and add 'iommmu=1' on the grub command line for the *hypervisor*. Then attach the new 'xm dmesg' and 'dmesg' output
Created attachment 348659 [details] dmesg with iommu=1
Created attachment 348660 [details] xm dmesg with iommu=1 To boot hypervisor with 'iommu=1', also appears in the hypervisor dmesg logs. I don't know if it's the bug on AMD machine. (XEN) I/O virtualisation disabled
Both those logs indicate you put 'iommu=1' on the Dom0 kernel command line. It needs to be on the *hypervisor* command line. ie, in grub.conf - it needs to be set for the xen.gz line, not the vmlinuz kernel title Red Hat Enterprise Linux Server (2.6.18-153.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-153.el5 iommu=1 module /vmlinuz-2.6.18-153.el5xen ro root=/dev/HostVG/RootLV rhgb quiet module /initrd-2.6.18-153.el5xen.img
Created attachment 348667 [details] dmesg info with iommu=1
Created attachment 348668 [details] xm dmesg info with iommu=1 oh, sorry for my mistake. But, unfortunately get the same error this time. (XEN) I/O virtualisation disabled
This is the important bit: (XEN) AMD-Vi: IOMMU not found! (XEN) I/O virtualisation disabled Your hardware is not VT-D capable. Therefore you can't use PCI passthrough for HVM guests, and thus the XenD error message is correct.
Strangely, if my hardware do not support VT-D, Why KVM can use PCI passthrough?
I got the same errors on Intel machine. error: Failed to start domain demo error: POST operation failed: xend_post: error from xen daemon: (xend.err "Error creating domain: failed to assign device: maybe the platform doesn't support VT-d, or VT-d isn't enabled properly?")
Don, perhaps you can help out here...
From the `xm dmesg' log it looks like this is an AMD platform so I probably can't help much. Given that you've specified `iommu=1' on the command line the only other issue I can think of is the BIOS, are you sure the IOMMU hasn't been disabled in the BIOS?
Sorry Don, did not realize it was non-Intel...
Right. Let's stick to the AMD side of things here (i.e. let's ignore comment #13, if that is also a problem, please open a separate bug). As danpb pointed out, the hypervisor doesn't think there is any IOMMU in the machine, so everything PCI-passthrough from that point on is going to fail. It's possible that the hypervisor just doesn't have support for this hardware, it's possible it is a bug. nzhang, can you please give us detailed information about the machine (dmidecode, what kind of processors, platform, etc), including where we can find it in RHTS if necessary? Bhvana, once we have some more detailed information, can you take a look? Thanks, Chris Lalancette
Chris, sure I can help. Once you confirm it is AMD specific (which it appears to be), please check AMD conf group. Bhavna
Bhavna, Thanks. I *think* it should be open to everybody, so you should be able to see the comments and the attachments. The problem basically boils down to the fact that on this particular piece of hardware, KVM pci-passthrough seems to work, but with Xen the HV says: IOMMU disabled (or something to that effect; you can look at the "xm dmesg info with iommu=1" attachment for full xm dmesg output). It's possible that it just hasn't been implemented for this platform yet, it's possible it's a bug. Besides the information that's already attached here (dmesg, xm dmesg, dmidecode, lspci, /proc/cpuinfo), do you need any other information? Thanks, Chris Lalancette
Chris, there is enough information, thanks. The system is a K8 dual core Athlon X2 system, defintely no IOMMU hw support. Xen IOMMU driver is doing the right thing as designed: For Xen, passing through pci devices on FV guest is only allowed after enabling iommu. The reason is that, FV guest device driver will program passthru device using guest physical address so we need iommu to translate dma address from guest physical address into host physical address otherwise the passthru devices won't be able to work. Therefore, xen disables pci-passthru for fv guest on non-iommu systems. For pv guest, guest driver uses host address directly so pci-passthru works even without iommu. In this case, dma translation is not necessary, enabling iommu only enhances I/O isolation. KVM driver is also behaving as defined. KVM is designed such that the guest can use the device without DMA. Adding Chris as this is a known issue, just how KVM is designed.
Bhavna, Thanks for the update. Then this is just a limitation of the software, not really a bug. I'll close this as NOTABUG. Chris Lalancette
Bhavna and Chris, Why did fail on Intel platform that I mentioned on Comments #13? nzhang
(In reply to comment #25) > Bhavna and Chris, > Why did fail on Intel platform that I mentioned on Comments #13? I'm not sure. That is either another, different limitation, or another bug. In either case, please open up a new Bugzilla with the details of that configuration and the problems you are facing there, and we can address that problem in the new BZ. Chris Lalancette