RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2073872 - Devices lost PCI ATS support capability in windows vm
Summary: Devices lost PCI ATS support capability in windows vm
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.0
Hardware: x86_64
OS: Windows
high
medium
Target Milestone: rc
: ---
Assignee: Kostiantyn Kostiuk
QA Contact: jinl
Jiri Herrmann
URL:
Whiteboard:
: 2089545 (view as bug list)
Depends On:
Blocks: 2001312
TreeView+ depends on / blocked
 
Reported: 2022-04-11 02:25 UTC by jinl
Modified: 2023-08-07 12:58 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
.PCIe ATS devices do not work on Windows VMs When you configure a PCIe Address Translation Services (ATS) device in the XML configuration of virtual machine (VM) with a Windows guest operating system, the guest does not enable the ATS device after booting the VM. This is because Windows currently does not support ATS on `virtio` devices. For more information, see the link:https://access.redhat.com/solutions/7025324[Red Hat KnowledgeBase].
Clone Of:
Environment:
Last Closed: 2023-07-18 13:58:37 UTC
Type: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-118495 0 None None None 2022-04-11 02:29:31 UTC
Red Hat Knowledge Base (Solution) 7025324 0 None None None 2023-07-21 00:42:49 UTC

Description jinl 2022-04-11 02:25:23 UTC
Description of problem:
Devices lost PCI ATS support capability in windows vm

Version-Release number of selected component (if applicable):
kernel-5.14.0-70.9.1.el9_0.x86_64
qemu-kvm-6.2.0-11.el9_0.2.x86_64
libvirt-8.0.0-8.el9_0.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot windows vm with Qemu cli:
/usr/libexec/qemu-kvm \
-name guest=win2019,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-9-win2019/master-key.aes"}' \
-machine pc-q35-rhel9.0.0,usb=off,dump-guest-core=off,kernel_irqchip=split,memory-backend=pc.ram \
-accel kvm \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on,hv-evmcs=on \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=4,threads=1 \
-uuid 06554e8d-c2b6-4cb1-953e-5f719ed60dfc \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on \
-device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.3,addr=0x0 \
-device virtio-scsi-pci,iommu_platform=on,ats=on,id=scsi0,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/home/win2019.qcow2","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-4-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-4-storage","backing":null}' \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-4-format,id=scsi0-0-0-0,bootindex=1,write-cache=on \
-device ide-cd,bus=ide.1,id=sata0-0-1 \
-device ide-cd,bus=ide.2,id=sata0-0-2 \
-device ide-cd,bus=ide.3,id=sata0-0-3 \
-netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 \
-device virtio-net-pci,iommu_platform=on,ats=on,netdev=hostnet0,id=net0,mac=52:54:00:03:52:9c,bus=pci.1,addr=0x0 \
-add-fd set=3,fd=33 \
-chardev file,id=charserial0,path=/dev/fdset/3,append=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,audiodev=audio1 \
-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-set device.net0.aer=on \
-set device.scsi0.aer=on \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

2.connect vncviewer to VM
Check virtio-net-pci device ATS ability:

Actual results:
No PCI ATS support property 
Device Manager -> Red Hat
VirtIO Ethernet Adapter -> Properties -> Details -> no PCI ATS support

Expected results:
Device Manager -> Red Hat
VirtIO Ethernet Adapter -> Properties -> Details
-> PCI ATS support


Additional info:
Change machine type from pc-q35-rhel9.0.0 to pc-q35-rhel8.5.0, device has 
PCI ATS support back.
Device ATS ability in RHEL VM works well with pc-q35-rhel9.0.0

Comment 1 jinl 2022-04-11 08:54:28 UTC
Additional info:
Devices properties which also lost:
PCI ACS capability register
PCI ACS compatible up hierarchy
PCI ACS support
PACI ARI support
PCI ECRC errors
PCI atomics supported
PCI correctable error mask
PCI error reporting
PCI firmware error handing
PCI uncorrectable error mask
PCI uncorrectable error severity

Comment 2 Peter Xu 2022-04-19 14:26:57 UTC
What about the same qemu cmdline with Linux guest?

Since it's specific issue with virtio-net/vhost, copying Jason.

Comment 3 jason wang 2022-04-20 03:24:52 UTC
(In reply to jinl from comment #1)
> Additional info:
> Devices properties which also lost:
> PCI ACS capability register
> PCI ACS compatible up hierarchy
> PCI ACS support
> PACI ARI support
> PCI ECRC errors
> PCI atomics supported
> PCI correctable error mask
> PCI error reporting
> PCI firmware error handing
> PCI uncorrectable error mask
> PCI uncorrectable error severity

Is the device identified as a PCI-E device or not in the guest?

Thanks

Comment 4 jinl 2022-04-24 07:59:45 UTC
(In reply to Peter Xu from comment #2)
> What about the same qemu cmdline with Linux guest?
Linux guest works well with the same qemu cmdline.

Comment 5 jinl 2022-04-24 08:38:47 UTC
(In reply to jason wang from comment #3)
> (In reply to jinl from comment #1)
> > Additional info:
> > Devices properties which also lost:
> > PCI ACS capability register
> > PCI ACS compatible up hierarchy
> > PCI ACS support
> > PACI ARI support
> > PCI ECRC errors
> > PCI atomics supported
> > PCI correctable error mask
> > PCI error reporting
> > PCI firmware error handing
> > PCI uncorrectable error mask
> > PCI uncorrectable error severity
> 
> Is the device identified as a PCI-E device or not in the guest?
> 
> Thanks

virtio-net-pci device is under pcie-root-port, so is identified as a PCI-E device in the guest.

Comment 6 jinl 2022-06-02 10:34:01 UTC
hit the same issue with RHEL 9.1 host

Comment 7 Yvugenfi@redhat.com 2022-07-11 12:33:00 UTC
Related BZ#2089545

Comment 8 CongLi 2022-07-13 03:41:06 UTC
*** Bug 2089545 has been marked as a duplicate of this bug. ***

Comment 9 Yvugenfi@redhat.com 2022-10-03 13:05:47 UTC
MS support ticket: 2210030030001461

Comment 10 jinl 2022-10-26 03:59:31 UTC
hit the same issue with win11 22H2
host env:
qemu-kvm-7.1.0-1.el9.x86_64
kernel-5.14.0-171.el9.x86_64

guest cmd:
/usr/libexec/qemu-kvm \
-name guest=win11,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-9-win11/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win11_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-rhel9.0.0,usb=off,smm=on,kernel_irqchip=split,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu Icelake-Server,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,avx512ifma=on,sha-ni=on,rdpid=on,fsrm=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,mpx=off,intel-pt=off,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on,hv-evmcs=on \
-global driver=cfi.pflash01,property=secure,value=on \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=4,threads=1 \
-uuid 28807dcd-5c36-4ac8-ba2c-4d4cedbe998a \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=22,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"eim":"on","device-iotlb":true}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.3","addr":"0x0"}' \
-device '{"driver":"virtio-scsi-pci","iommu_platform":true,"ats":true,"id":"scsi0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/home/win10.qcow2","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-4-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-4-storage","backing":null}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-4-format","id":"scsi0-0-0-0","bootindex":1,"write-cache":"on"}' \
-device '{"driver":"ide-cd","bus":"ide.1","id":"sata0-0-1"}' \
-device '{"driver":"ide-cd","bus":"ide.2","id":"sata0-0-2"}' \
-device '{"driver":"ide-cd","bus":"ide.3","id":"sata0-0-3"}' \
-netdev tap,fd=24,vhost=on,vhostfd=26,id=hostnet0 \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"netdev":"hostnet0","id":"net0","mac":"52:54:00:dc:d0:4c","bus":"pci.1","addr":"0x0"}' \
-add-fd set=0,fd=23,opaque=serial0-source \
-chardev file,id=charserial0,path=/dev/fdset/0,append=on \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/9-win11-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-crb","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.4","addr":"0x0"}' \

Comment 15 Yvugenfi@redhat.com 2023-03-13 11:22:33 UTC
Issue summary:
================

Windows does not detect ATS property for Ethernet PCI controller when
firmware masks PCIE native hotplug capability in _OSC control


Versions of windows affected:
=============================

Server versions: Windows server 2016, windows server 2019, windows server 2022
Desktop versions: Windows 10, Windows 11.

From our testing, the versions of Windows earlier than those mentioned
above do not report/check for ATS availability even when the firmware
does not mask PCIE native hotplug in _OSC. It seems reporting ATS
status is only available in the above newer versions of the OS.

How to reproduce the issue?
============================

We have used the QEMU hypervisor to test for this issue.
Please refer to the following RedHat BZ for the QEMU commandline used:
https://bugzilla.redhat.com/show_bug.cgi?id=2089545

Additionally, one can use the following scripts:

https://github.com/ani-sinha/misc/blob/master/test-ats.sh
https://github.com/ani-sinha/misc/blob/master/test-ats-win11.sh

(Please adjust the locations of the images/binaries as per your host).

Note that the ethernet controller is attached to the pcie root port
which in turn is attached to the pci root complex.
The ATS property is for the pcie root port as mentioned in Intel VTD spec (1).

"--global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on/off" will
mask/unmask the PCIE native hotplug reporting in _OSC from the
firmware in QEMU and thus the issue can be observed easily. Please see
(2). "on" will mask the bit, "off" will unmask.

"-snapshot" command line has been used with the disk image to make
sure there are no caching effect within the guest OS.
Further, we have tested by removing and re-adding the ethernet
controller from within the guest OS.
The results are the same.


References:

(1) Intel Virtualization technology for Directed I/O, section 8.5 -
Root Port ATS capability reporting structure.
(2) ACPI spec 5.1, section 6.2.11.3 - OSC Implementation Example for
PCI Host Bridge Devices


Note You need to log in before you can comment on or make changes to this bug.