Bug 506923 - [RHEL5.4 Xen]: AMD PCI passthrough failed with Xen FV guest
[RHEL5.4 Xen]: AMD PCI passthrough failed with Xen FV guest
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
All Linux
low Severity medium
: rc
: ---
Assigned To: Xen Maintainance List
Virtualization Bugs
: Reopened
Depends On:
  Show dependency treegraph
Reported: 2009-06-19 06:33 EDT by Nan Zhang
Modified: 2009-12-14 16:27 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-07-27 10:41:06 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
dmesg info on the host os (16.73 KB, text/plain)
2009-06-19 07:03 EDT, Nan Zhang
no flags Details
xm dmesg info (5.16 KB, text/plain)
2009-06-19 07:05 EDT, Nan Zhang
no flags Details
dmesg with iommu=1 (16.74 KB, text/plain)
2009-06-19 09:05 EDT, Nan Zhang
no flags Details
xm dmesg with iommu=1 (5.16 KB, text/plain)
2009-06-19 09:10 EDT, Nan Zhang
no flags Details
dmesg info with iommu=1 (16.73 KB, text/plain)
2009-06-19 09:50 EDT, Nan Zhang
no flags Details
xm dmesg info with iommu=1 (5.30 KB, text/plain)
2009-06-19 09:52 EDT, Nan Zhang
no flags Details
dmidecode info (22.22 KB, text/plain)
2009-06-25 06:37 EDT, Nan Zhang
no flags Details
cpu info (782 bytes, text/plain)
2009-06-25 06:39 EDT, Nan Zhang
no flags Details
pci device info (20.12 KB, text/plain)
2009-06-25 06:41 EDT, Nan Zhang
no flags Details

  None (edit)
Description Nan Zhang 2009-06-19 06:33:54 EDT
Description of problem:
Can not boot xen fv guest with pci passthrough config.

Version-Release number of selected component (if applicable):
libvirt 0.6.3-10.el5
rhel-5.4 (2.6.18-154.el5xen)

How reproducible:

Steps to Reproduce:
# virsh nodedev-dettach pci_14e4_167a
Device pci_14e4_167a dettached

# virsh nodedev-reset pci_14e4_167a
Device pci_14e4_167a reset

# virsh edit demo
Domain demo XML configuration edited.

        <hostdev mode='subsystem' type='pci' managed='yes'>
           <address bus='0x02' slot='0x00' function='0x00'/>

# virsh start demo
error: Failed to start domain demo
error: POST operation failed: xend_post: error from xen daemon: (xend.err
"Error creating domain: failed to assign device: maybe the platform doesn't
support VT-d, or VT-d isn't enabled properly?")
Actual results:
Failed to start guest domain.

Expected results:
Guest domain can be started with pci passthrough config.

Additional info:
It's fine on Xen PV guest and KVM
Comment 1 Daniel Berrange 2009-06-19 06:42:11 EDT
Unlikely to be a libvirt bug, since the error is coming from XenD and indicates XenD has understood the PCI config, but simply doesn't support it.

Can you provide the output from 'dmesg' on the host OS immediately after it has finished booting.
Comment 2 Daniel Berrange 2009-06-19 06:57:36 EDT
Can you also provide the 'xm dmesg' output, since we need to see the hypervisor logs too
Comment 3 Nan Zhang 2009-06-19 07:03:28 EDT
Created attachment 348633 [details]
dmesg info on the host os
Comment 4 Nan Zhang 2009-06-19 07:05:28 EDT
Created attachment 348635 [details]
xm dmesg info
Comment 5 Daniel Berrange 2009-06-19 08:04:42 EDT
The reason you get the error message from Xen, is that it does not have VT-D support active. This appears to be confirmed by the hypervisor dmesg logs

(XEN) I/O virtualisation disabled

I think you need to reboot, and add 'iommmu=1' on the grub command line for the *hypervisor*. Then attach the new 'xm dmesg' and 'dmesg' output
Comment 6 Nan Zhang 2009-06-19 09:05:55 EDT
Created attachment 348659 [details]
dmesg with iommu=1
Comment 7 Nan Zhang 2009-06-19 09:10:58 EDT
Created attachment 348660 [details]
xm dmesg with iommu=1

To boot hypervisor with 'iommu=1', also appears in the hypervisor dmesg logs. I don't know if it's the bug on AMD machine.

(XEN) I/O virtualisation disabled
Comment 8 Daniel Berrange 2009-06-19 09:16:22 EDT
Both those logs indicate you put 'iommu=1' on the Dom0 kernel command line. It needs to be on the *hypervisor* command line. 

ie, in grub.conf - it needs to be set for the xen.gz line, not the vmlinuz kernel

title Red Hat Enterprise Linux Server (2.6.18-153.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-153.el5 iommu=1
        module /vmlinuz-2.6.18-153.el5xen ro root=/dev/HostVG/RootLV rhgb quiet
        module /initrd-2.6.18-153.el5xen.img
Comment 9 Nan Zhang 2009-06-19 09:50:27 EDT
Created attachment 348667 [details]
dmesg info with iommu=1
Comment 10 Nan Zhang 2009-06-19 09:52:12 EDT
Created attachment 348668 [details]
xm dmesg info with iommu=1

oh, sorry for my mistake. But, unfortunately get the same error this time.

(XEN) I/O virtualisation disabled
Comment 11 Daniel Berrange 2009-06-19 09:59:25 EDT
This is the important bit:

(XEN) AMD-Vi: IOMMU not found!
(XEN) I/O virtualisation disabled

Your hardware is not VT-D capable. Therefore you can't use PCI passthrough for HVM guests, and thus the XenD error message is correct.
Comment 12 Nan Zhang 2009-06-19 10:16:17 EDT
Strangely, if my hardware do not support VT-D, Why KVM can use PCI passthrough?
Comment 13 Nan Zhang 2009-06-21 05:01:37 EDT
I got the same errors on Intel machine.

error: Failed to start domain demo
error: POST operation failed: xend_post: error from xen daemon: (xend.err
"Error creating domain: failed to assign device: maybe the platform doesn't
support VT-d, or VT-d isn't enabled properly?")
Comment 14 Bill Burns 2009-06-24 07:30:03 EDT
Don, perhaps you can help out here...
Comment 15 Don Dugger 2009-06-24 16:07:29 EDT
From the `xm dmesg' log it looks like this is an AMD platform so I probably can't help much.  Given that you've specified `iommu=1' on the command line the only other issue I can think of is the BIOS, are you sure the IOMMU hasn't been disabled in the BIOS?
Comment 16 Bill Burns 2009-06-24 20:29:10 EDT
Sorry Don, did not realize it was non-Intel...
Comment 17 Chris Lalancette 2009-06-25 03:01:03 EDT
Right.  Let's stick to the AMD side of things here (i.e. let's ignore comment #13, if that is also a problem, please open a separate bug).  As danpb pointed out, the hypervisor doesn't think there is any IOMMU in the machine, so everything PCI-passthrough from that point on is going to fail.  It's possible that the hypervisor just doesn't have support for this hardware, it's possible it is a bug.  nzhang, can you please give us detailed information about the machine (dmidecode, what kind of processors, platform, etc), including where we can find it in RHTS if necessary?

Bhvana, once we have some more detailed information, can you take a look?

Chris Lalancette
Comment 21 Bhavna Sarathy 2009-06-25 09:47:26 EDT
Chris, sure I can help.  Once you confirm it is AMD specific (which it appears to be), please check AMD conf group.
Comment 22 Chris Lalancette 2009-06-25 10:31:34 EDT
    Thanks.  I *think* it should be open to everybody, so you should be able to see the comments and the attachments.  The problem basically boils down to the fact that on this particular piece of hardware, KVM pci-passthrough seems to work, but with Xen the HV says:

IOMMU disabled

(or something to that effect; you can look at the "xm dmesg info with iommu=1" attachment for full xm dmesg output).  It's possible that it just hasn't been implemented for this platform yet, it's possible it's a bug.  Besides the information that's already attached here (dmesg, xm dmesg, dmidecode, lspci, /proc/cpuinfo), do you need any other information?

Chris Lalancette
Comment 23 Bhavna Sarathy 2009-06-29 10:41:51 EDT
Chris, there is enough information, thanks.   The system is a K8 dual core 
Athlon X2 system, defintely no IOMMU hw support.

Xen IOMMU driver is doing the right thing as designed:

For Xen, passing through pci devices on FV guest is only allowed after enabling iommu. The reason is that, FV guest device driver will program passthru device using guest physical address so we need iommu to translate dma address from guest physical address into host physical address otherwise the passthru devices won't be able to work. Therefore, xen disables pci-passthru for fv guest  on non-iommu systems. 

For pv guest, guest driver uses host address directly so pci-passthru works even without iommu. In this case, dma translation is not necessary, enabling iommu only enhances I/O isolation.

KVM driver is also behaving as defined.  KVM is designed such that the guest can use the device without DMA.  Adding Chris as this is a known issue, just how KVM is designed.
Comment 24 Chris Lalancette 2009-07-09 07:56:01 EDT
    Thanks for the update.  Then this is just a limitation of the software, not really a bug.  I'll close this as NOTABUG.

Chris Lalancette
Comment 25 Nan Zhang 2009-07-22 03:34:53 EDT
Bhavna and Chris,
    Why did fail on Intel platform that I mentioned on Comments #13?

Comment 26 Chris Lalancette 2009-07-27 10:41:06 EDT
(In reply to comment #25)
> Bhavna and Chris,
>     Why did fail on Intel platform that I mentioned on Comments #13?

I'm not sure.  That is either another, different limitation, or another bug.  In either case, please open up a new Bugzilla with the details of that configuration and the problems you are facing there, and we can address that problem in the new BZ.

Chris Lalancette

Note You need to log in before you can comment on or make changes to this bug.