Root caused this bug, there are 2 issues at here for both RHEL6.2 kernel and linux upstream.
1). When the device with ATS capability is assigned to guest, KVM will enable ATS as expect, however, the device info did not be added into list before do context mapping, so enable device fail. code path is:
   kvm_assign_device
           \
   iommu_attach_device
           \
   intel_iommu_attach_device
           \
   domain_add_dev_info
           \
   domain_context_mapping
           \
   domain_context_mapping_one
           \
   iommu_enable_dev_iotlb(info)
           \  /*pci_enable_ats be called only if info is not Null*/

2). ATS status did not be restore after to FLR at all; this issue also happen on system S3/S4 case, when system sleep into S3 or S4, and then system resume, pci_restore_state() did not restore the ATS status for device.

I cook two patches for these two issues fix, will attach them in this BZ.

Created attachment 547583 [details]
add device info before mapping context

Add device info into list before do context mapping, because device info will be used by iommu_enable_dev_iotlb function.

Created attachment 547584 [details]
enable ATS when restore

When system go to S3 or S4 sleep and then return, some register of PCI device does not be restored correctly, such as ATS capability. The same problem happen on pci reset function.

This patch enable ATS at the device state restore if PCIe device has ATS capability.

(In reply to comment #2) 
> I cook two patches for these two issues fix, will attach them in this BZ.

Will you be submitting these upstream?  Please let us know when they are accepted.  Thanks

(In reply to comment #5)
> (In reply to comment #2) 
> > I cook two patches for these two issues fix, will attach them in this BZ.
> Will you be submitting these upstream?  Please let us know when they are
> accepted.  Thanks

Sure, I will submit to upstream and indicate commit here if they are accepted.

Since RHEL 6.3 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

(In reply to comment #8)
> Since RHEL 6.3 External Beta has begun, and this bug remains
> unresolved, it has been rejected as it is not proposed as
> exception or blocker.
> Red Hat invites you to ask your support representative to
> propose this request, if appropriate and relevant, in the
> next release of Red Hat Enterprise Linux.

There are two patches to try fix this issue, the 1st is iommu side and the 2nd is pci side. 

The 2nd patch has been included in linux upstream commit: 1900ca132f53c3d51e6e6b94ea8912530223c63a. 
 "PCI: Enable ATS at the device state restore"

The 1st patch did not be checked into linux yet, I'm continuing push it.

I think RHEL6.3 can merge the 2nd patch in firstly. But maybe it's still a long time for the 1st patch, so delay this bug to next release of RHEL.

Need 1st patch to be upstream before inclusion in RHEL6.

RHEL6.3 is taking release blockers only. 
This bz is not a release blocker for 6.3,
and is targetted for 6.4.

Please continue to push upstream inclusion, so we can do backport once included.

Now all fix patches is in kernel upstream tree, which include 3 patches commit:

commit 109b9b0408e5f1dd327a44f446841a9fbe0bcd83
Author: David Woodhouse <dwmw2>
Date:   Fri May 25 17:43:02 2012 +0100

    intel-iommu: Move duplicate list-mangling code into unlink_domain_info() helper

commit e2ad23d04c1304431ab5176c89b7b476ded2d995
Author: David Woodhouse <dwmw2>
Date:   Fri May 25 17:42:54 2012 +0100

    intel-iommu: Add device info into list before doing context mapping

commit 1900ca132f53c3d51e6e6b94ea8912530223c63a
Author: Hao, Xudong <xudong.hao>
Date:   Sat Dec 17 21:24:40 2011 +0800

    PCI: Enable ATS at the device state restore

(In reply to comment #11)
> Now all fix patches is in kernel upstream tree, which include 3 patches
> commit:
> 
> commit 109b9b0408e5f1dd327a44f446841a9fbe0bcd83
> Author: David Woodhouse <dwmw2>
> Date:   Fri May 25 17:43:02 2012 +0100
> 
>     intel-iommu: Move duplicate list-mangling code into unlink_domain_info()
> helper
> 
> commit e2ad23d04c1304431ab5176c89b7b476ded2d995
> Author: David Woodhouse <dwmw2>
> Date:   Fri May 25 17:42:54 2012 +0100
> 
>     intel-iommu: Add device info into list before doing context mapping
> 
> commit 1900ca132f53c3d51e6e6b94ea8912530223c63a
> Author: Hao, Xudong <xudong.hao>
> Date:   Sat Dec 17 21:24:40 2011 +0800
> 
>     PCI: Enable ATS at the device state restore

What we need to do to support this new functionality (ATS hasn't been supported in RHEL5 or RHEL6 to date) is determine the necessary patches for RHEL6.
For example, the last commit above is to a file that doesn't even exist in RHEL6.  That requires cherry-picking 10 patches previous to it before commit 1900ca132 can be applied.
Additionally, there are over 50 patches in upstream intel-iommu.c btwn the
above ATS patches and the rhel6 version of that file.  Many of those 50 patches aren't needed, but we have to go through them to figure out which of those 50 patches are needed before the first two patches above will enable ATS correctly.

Turns out RHEL 6 does support ATS, it's just that the support is tied to the config parameter CONFIG_PCI_IOV (a lot of the differences between the RHEL 6 kernel and upstrem in this area are related to refactoring to put the ATS code in separate files under the CONFIG_PCI_ATS parameter so it is separately configurable).  Intel has tested the 3 patches and they fix this BZ, I'll be posting the patches to RHML shortly.

This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

Hi Yongjie,
It seems we don't have the card in-house. Do you commit to test the bug in rhel6.4 testing cycle? Thanks.

(In reply to comment #16)
> Hi Yongjie,
> It seems we don't have the card in-house. Do you commit to test the bug in
> rhel6.4 testing cycle? Thanks.

Yes, sure. I'll do a testing for RHEL6.4 to verify this bug.

BTW, there might be a bug for the driver 'myri10ge' in RHEL6.3/6.4. It can't driver my Myricom NIC, which has nothing about virtualization.
So, I'll verify this bug with RHEL6.4 as my host and RHEL6.2 as a guest.

(Odd, for reasons that are obscure to me I can't see comment #15 where Kexin asked me for info.  Anyway...)

I think the risk associated with this patch is not that great.  The bug is fixed by 3 patches that:

1)  Add the device info at the right time.  This is current code that is just moved to the appropriate place.

2)  Consolidate some common code.

3)  Restore state after an S3 resume, the correct thing to do.

Also, we have tested it, even through S3, with no problems so I think this should be a safe change.

Patch(es) available on kernel-2.6.32-315.el6

ATS is enabled in the assigned Myricom NIC in a RHEL6.4 guest on a RHEL6.4 host.

Verified by Intel Virtualization Team.

RPM packages I used:
kernel-2.6.32-343.el6.x86_64
qemu-kvm-0.12.1.2-2.335.el6.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0496.html