(In reply to oakwhiz from comment #0)
> Is it possible to have this patch added to the Fedora kernel?

At the moment, no.  Mostly because it's still in the middle of being discussed and isn't in the upstream tree yet.  We'll keep an eye on it and see if it's applicable for backport once it hits the mainline kernel tree.

The patch is not expected to be accepted upstream but downstreams can obviously choose to carry it.  The argument upstream is that issues that arise from a user overriding known, hardware advertised device isolation can be subtle and incredibly difficult to debug.  The path forward to allowing configurations that are currently prevented is to work with the hardware vendors to determine whether devices are isolated and encourage future products to support PCI ACS so that the hardware advertises this isolation automatically.

Created attachment 913028 [details]
ACS override patch for 3.14.8-200.fc20.x86_64

ACS override patch for 3.14.8-200.fc20.x86_64
This patch seems like it works on the latest Fedora kernel. I think.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.17.2-200.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21.

If you experience different issues, please open a new bug report for those.

I am observing the same vfio error message after upgrading a system from Fedora 20 to Fedora 21.

My scenario is the passthrough of an Intel RMS25KB080 controller, installed in an Intel S1200V3RPL mainboard (in the PCIe x8 Gen 2.x slot).

With Fedora 20 I was able to passthrough without getting this error, and the controller was working fine attached to a guest.

However, after I upgraded the host system to Fedora 21, I am now seeing this error when I attempt to start the guest with 'sudo virsh start', e.g. 

"group <n> is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver"

I would prefer not to have to build and maintain a patched kernel :( What further information should I provide, and should I create a new bug for this specific hardware scenario?

Thanks,

Alex

(In reply to Alex from comment #5)
> What further information should I provide, and should I create a new bug 
> for this specific hardware scenario?

lspci -nnvvv and processor model

Created attachment 977932 [details]
Output from 'lspci -nnvvv'

Attached 'lspci_output.txt', contains the output from 'lspci -nnvvv' on a system with Intel RMS25KB080 controller, installed in an Intel S1200V3RPL mainboard. The device address of the controller being passed through is '02:00.0'.

(In reply to Alex Williamson from comment #6)
> (In reply to Alex from comment #5)
> > What further information should I provide, and should I create a new bug 
> > for this specific hardware scenario?
> 
> lspci -nnvvv and processor model

CPU processor model: Intel Xeon CPU E3-1220v3 @ 3.10GHz

(In reply to Alex from comment #7)
> Created attachment 977932 [details]
> Output from 'lspci -nnvvv'
> 
> Attached 'lspci_output.txt', contains the output from 'lspci -nnvvv' on a
> system with Intel RMS25KB080 controller, installed in an Intel S1200V3RPL
> mainboard. The device address of the controller being passed through is
> '02:00.0'.

Sorry to say, but Xeon E3-1200 series are the only Xeons (afaik) that do not (and apparently will not) support isolation on the processor root ports:

https://www-ssl.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e3-1200v3-spec-update.pdf

---
HSW2. Intel® Virtualization Technology (Intel® VT) Clarification

Section 3.1 will be modified to include the following paragraph:

It is recommended to avoid device direct assignment to Virtual Machines in virtualized environments with this processor due to the lack of Access Control Services (ACS) support in PCI-Express root ports. Some Operating Systems may check for ACS support and potentially disable direct device assignment (that is, affects SR-IOV setup/configuration within the server) as well.
---

My assumption is that this applies to v1 and v2 E3-1200 processors as well.  Unless you can get Intel to change their mind, you're going to need to install the device elsewhere or patch your kernel and risk unintended peer-to-peer between devices.  We have quirks for most of the PCH root ports, but Intel seems to have no intention of supporting isolation on client processor (Core-i5/7) or Xeon E3-1200 series processor root ports.

(In reply to Alex Williamson from comment #9)
> (In reply to Alex from comment #7)
> > Created attachment 977932 [details]
> > Output from 'lspci -nnvvv'
> > 
> > Attached 'lspci_output.txt', contains the output from 'lspci -nnvvv' on a
> > system with Intel RMS25KB080 controller, installed in an Intel S1200V3RPL
> > mainboard. The device address of the controller being passed through is
> > '02:00.0'.
> 
> Sorry to say, but Xeon E3-1200 series are the only Xeons (afaik) that do not
> (and apparently will not) support isolation on the processor root ports:
> 
> https://www-ssl.intel.com/content/dam/www/public/us/en/documents/
> specification-updates/xeon-e3-1200v3-spec-update.pdf
> 
> My assumption is that this applies to v1 and v2 E3-1200 processors as well. 
> Unless you can get Intel to change their mind, you're going to need to
> install the device elsewhere or patch your kernel and risk unintended
> peer-to-peer between devices.  We have quirks for most of the PCH root
> ports, but Intel seems to have no intention of supporting isolation on
> client processor (Core-i5/7) or Xeon E3-1200 series processor root ports.

I'm confused now.. this was working fine in Fedora 20 for several months (no error on virsh start), how could this be the case if it's a hardware issue? In fact, my plan was to go back to Fedora 20 to get it working again.

Thanks

Alex.

(In reply to Alex from comment #10)
> 
> I'm confused now.. this was working fine in Fedora 20 for several months (no
> error on virsh start), how could this be the case if it's a hardware issue?
> In fact, my plan was to go back to Fedora 20 to get it working again.

Perhaps you were using legacy KVM device assignment on FC20 (pci-assign).  That driver leaves ACS requirements to userspace, which mostly gets it wrong and allows it to be easily disabled anyway.  The new driver, vfio-pci has kernel enforced isolation requirements because the kernel must protect itself.  You can switch back to legacy assignment by specifying the driver option in your xml, but you can expect that at some point legacy KVM device assignment will go away.

(In reply to Alex Williamson from comment #11)
> (In reply to Alex from comment #10)
> > 
> > I'm confused now.. this was working fine in Fedora 20 for several months (no
> > error on virsh start), how could this be the case if it's a hardware issue?
> > In fact, my plan was to go back to Fedora 20 to get it working again.
> 
> Perhaps you were using legacy KVM device assignment on FC20 (pci-assign). 
> That driver leaves ACS requirements to userspace, which mostly gets it wrong
> and allows it to be easily disabled anyway.  The new driver, vfio-pci has
> kernel enforced isolation requirements because the kernel must protect
> itself.  You can switch back to legacy assignment by specifying the driver
> option in your xml, but you can expect that at some point legacy KVM device
> assignment will go away.

Aha.. based on your advice I did a quick web search and came across this doc for libvirt: https://libvirt.org/formatdomain.html#elementsHostDev

In this doc, regarding the 'name' attribute of the 'driver' sub-element of 'hostdev', it states: "When not specified, the default is 'vfio' on systems where the VFIO driver is available and loaded, and 'kvm' on older systems, or those where the VFIO driver hasn't been loaded [Since 1.1.3] (prior to that the default was always 'kvm')."

So I guess this is the change from F20 to F21 that is causing the error. I hadn't touched the xml definition for the guest, but the default value has changed in libvirt; and with my system not supporting the vfio driver properly, I hit the error.

Anyway I just tried a workaround by adding "<device name='kvm'/>" to the 'hostdev' in the xml definition, and "virsh start" is now working :)

So to summarise..

- If I want to stay on F21, my options are:

1) Use the "<device name='kvm'/>" workaround (aka legacy device assignment)
2) Try the disk controller in a PCH-connected pcie slot, instead of a CPU-connected pcie slot? I think this would mean an x4 connection instead of an x8 connection, for the mainboard I have..
3) Use the ACS override patch
4) Upgrade the system to use hardware with ACS support; presumably this would mean a Xeon E5 CPU/mainboard instead of an E3, or something like that...?

- The disadvantages of legacy device assignment, compared to ACS/vfio are:

1) Legacy method doesn't support EFI systems & secure boot etc
2) Legacy method gives lower system performance/less stability?
3) Legacy method will be removed at some point

Thanks

Alex.

I just wanted to give a quick update following my previous comment, with some results after trying options 1 and 2:

- Option 1 worked - with this option, I didn't need to relocate the disk controller to a different (PCH) slot; I used 'virsh edit' to modify the guest definition and insert the "<device name='kvm'/>" workaround;

- Option 2 worked - with this option, I didn't need the "<device name='kvm'/>" workaround; I relocated the disk controller from a CPU-connected pcie slot to a PCH-connected pcie slot.

Thanks for your help,

Alex.

This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Hi, 
By updating to kernel (3.10.0-514.21.1.el7.x86_64),I found the following error when try to deploy VM.

Libvirt cannot start the VM because of the following errors:
2017-11-01T14:07:44.930993Z qemu-kvm: -device vfio-pci,host=04:10.6,id=hostdev0,bus=pci.0,addr=0x4: vfio: error, group 2 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

With old kernel, 3.10.0-514.10.2.el7.x86_64 has no such issue.


I suspect the ACS override patch for AMD is missing in current kernel package. Here are the list iommmu group for working kernel:

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.3
/sys/kernel/iommu_groups/3/devices/0000:00:02.4
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/6/devices/0000:00:08.0
/sys/kernel/iommu_groups/7/devices/0000:00:09.0
/sys/kernel/iommu_groups/7/devices/0000:00:09.2
/sys/kernel/iommu_groups/8/devices/0000:00:10.0
/sys/kernel/iommu_groups/9/devices/0000:00:11.0
/sys/kernel/iommu_groups/10/devices/0000:00:12.0
/sys/kernel/iommu_groups/11/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:18.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.1
/sys/kernel/iommu_groups/12/devices/0000:00:18.2
/sys/kernel/iommu_groups/12/devices/0000:00:18.3
/sys/kernel/iommu_groups/12/devices/0000:00:18.4
/sys/kernel/iommu_groups/12/devices/0000:00:18.5
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/15/devices/0000:03:00.1
/sys/kernel/iommu_groups/16/devices/0000:03:00.2
/sys/kernel/iommu_groups/17/devices/0000:03:00.3
/sys/kernel/iommu_groups/18/devices/0000:04:10.2
/sys/kernel/iommu_groups/19/devices/0000:04:10.6
/sys/kernel/iommu_groups/20/devices/0000:04:10.3
/sys/kernel/iommu_groups/21/devices/0000:04:10.7
/sys/kernel/iommu_groups/22/devices/0000:04:10.0
/sys/kernel/iommu_groups/23/devices/0000:04:10.4
/sys/kernel/iommu_groups/24/devices/0000:04:10.1
/sys/kernel/iommu_groups/25/devices/0000:04:10.5
[root@nfvis ~]# 

Here is one with issues:

/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.1
/sys/kernel/iommu_groups/2/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:03:00.1
/sys/kernel/iommu_groups/2/devices/0000:03:00.2
/sys/kernel/iommu_groups/2/devices/0000:03:00.3
/sys/kernel/iommu_groups/2/devices/0000:04:10.0
/sys/kernel/iommu_groups/2/devices/0000:04:10.1
/sys/kernel/iommu_groups/2/devices/0000:04:10.2
/sys/kernel/iommu_groups/2/devices/0000:04:10.3
/sys/kernel/iommu_groups/2/devices/0000:04:10.4
/sys/kernel/iommu_groups/2/devices/0000:04:10.5
/sys/kernel/iommu_groups/2/devices/0000:04:10.6
/sys/kernel/iommu_groups/2/devices/0000:04:10.7

0000:03:xx.x is for physical nic
0000:04:xx.x is for virtual nic

Any suggestion??

Any other information required??