"00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)" -- 00:09.0 has a "Bridge:" in the name -- seems unusual?

Hi Chris Lalancette,
Please supply the info of "lspci -tv" and "lspci -vvvv -xxxx" so I can have enough info to know what's wrong here.

Created attachment 360884 [details]
Output of lspci -tv

Created attachment 360885 [details]
Output of lspci -vvvv -xxxx

(In reply to comment #0)
> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
From the "lspci -vvvv -xxxx" info in Comment #3, we can see 00:09.0 has an non-page-aligned MMIO BAR:
	Region 3: Memory at b0005c00 (32-bit, non-prefetchable) [size=16].

Actually in the host:
00:02.0 has an MMIO BAR with base b0005000;
00:08.0 has an MMIO BAR with base b0005800 and b0005400;
00:09.0 has an MMIO BAR with base b0005c00.
We can see such 3 devices/functions have BARs included in the same page.
This means it's unsafe to assign them to guests, including both pv guest and hvm guest, because hypervisor can only perform the mmio permission control at a granularity of page, e.g., if we allow 00:09.0 to be assigned to a pv or an hvm guest, the guest can access the MMIO bars of 00:02.0 and 00:08.0 that are not assigned to it at all! 

So I think such kind of device is not assignable: when we try to do that, the guest construction would fail and xend would complain "pci: 0000:00:09.0: non-page-aligned MMIO BAR found."
That's why RHEL5.4 Xen and upstream Xen don't show the device in the output of "xm pci-list-assignable-devices".

> Jirka mentioned that xm pci-list-assignable-devices may only show devices that
> have page-aligned MMIO bars, because those are the only ones you can
> passthrough to an HVM guest.  However, if this is supposed to be common
> functionality between PV and HVM guests, we need to show these devices as
> assignable for PV guests.  I would suggest that we show all of the devices, but
> mark the ones that we can't passthrough to an HVM guest with a short message
> describing exactly that.  
As I stated above, this applies to both pv guest and hvm guest, so I don't think we should show devices with non-page-aligned MMIO BAR.

(In reply to comment #4)
> (In reply to comment #0)
> > 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
> From the "lspci -vvvv -xxxx" info in Comment #3, we can see 00:09.0 has an
> non-page-aligned MMIO BAR:
>  Region 3: Memory at b0005c00 (32-bit, non-prefetchable) [size=16].
> 
> Actually in the host:
> 00:02.0 has an MMIO BAR with base b0005000;
> 00:08.0 has an MMIO BAR with base b0005800 and b0005400;
> 00:09.0 has an MMIO BAR with base b0005c00.
> We can see such 3 devices/functions have BARs included in the same page.
> This means it's unsafe to assign them to guests, including both pv guest and
> hvm guest, because hypervisor can only perform the mmio permission control at a
> granularity of page, e.g., if we allow 00:09.0 to be assigned to a pv or an hvm
> guest, the guest can access the MMIO bars of 00:02.0 and 00:08.0 that are not
> assigned to it at all! 

Hm, OK, that is interesting.

> 
> So I think such kind of device is not assignable: when we try to do that, the
> guest construction would fail and xend would complain "pci: 0000:00:09.0:
> non-page-aligned MMIO BAR found."

But no, I definitely was able to pass it through to the PV guest (although I might have disabled strict checking at that point; I can't remember).

Given all of that, I agree, we shouldn't necessarily fix this one for PV guests.  I think we have a workaround for people who used to do this (by disabling strict checking), and since it's unsafe, we don't need to list it in pci-list-assignable.  I'll close as WONTFIX.


Chris Lalancette

(In reply to comment #5)
> > So I think such kind of device is not assignable: when we try to do that, the
> > guest construction would fail and xend would complain "pci: 0000:00:09.0:
> > non-page-aligned MMIO BAR found."
> But no, I definitely was able to pass it through to the PV guest (although I
> might have disabled strict checking at that point; I can't remember).
I think even if we set pci-dev-assign-strict-check=no, we're still unable to assign the device to PV guest as xend would prevent us with the "pci: 0000:00:09.0: non-page-aligned MMIO BAR found". I think pci-dev-assign-strict-check=no can only workaround the co-assignment issue for pv guest. So you may want to double check it. :-)
(I don't have such an 'IOV-unfriendly' BIOS so I can't try it myself.)

> Given all of that, I agree, we shouldn't necessarily fix this one for PV
> guests.  I think we have a workaround for people who used to do this (by
> disabling strict checking), and since it's unsafe, we don't need to list it in
> pci-list-assignable.  I'll close as WONTFIX.
> Chris Lalancette  
Yes, in old RHEL Xen (e.g., RHEL 5.3 Xen or 5.2 Xen), we could assign such kind of device to PV guest, but we didn't realize we were at the risk of being attacked from guest.


I think the best solution is: BIOS vendors to not assign non-page-aligned MMIO BAR to device if VT-d is enabled. Actually I guess most of BIOS developer's manuals/specifications should have already contained such a suggestion. And BIOS vendors should supply bios updates to fix the existing BIOSes.

To cope with the existing bad BIOSes, we can let dom0 fix the non-page-aligned MMIO BAR when it boots up. In upstream linux-2.6.18-xen.hg, there are the code and kernel parameters: it
should be http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev=reassign_resources. Maybe they could be backed port into RHEL Xen'd Dom0. The line-of-code might be 1k~1.5k by my rough estimation. Maybe you would be interested in this. :-)

This bug was closed during 5.5 development and it's being removed from the internal tracking bugs (which are now for 5.6).

Clearing out old flags for reporting purposes.

Chris Lalancette