Description of problem: Devices lost PCI ATS support capability in windows vm Version-Release number of selected component (if applicable): kernel-5.14.0-70.9.1.el9_0.x86_64 qemu-kvm-6.2.0-11.el9_0.2.x86_64 libvirt-8.0.0-8.el9_0.x86_64 How reproducible: 100% Steps to Reproduce: 1. Boot windows vm with Qemu cli: /usr/libexec/qemu-kvm \ -name guest=win2019,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-9-win2019/master-key.aes"}' \ -machine pc-q35-rhel9.0.0,usb=off,dump-guest-core=off,kernel_irqchip=split,memory-backend=pc.ram \ -accel kvm \ -cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on,hv-evmcs=on \ -m 8192 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \ -overcommit mem-lock=off \ -smp 4,sockets=1,dies=1,cores=4,threads=1 \ -uuid 06554e8d-c2b6-4cb1-953e-5f719ed60dfc \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=34,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on \ -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \ -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.3,addr=0x0 \ -device virtio-scsi-pci,iommu_platform=on,ats=on,id=scsi0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/home/win2019.qcow2","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-4-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-4-storage","backing":null}' \ -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-4-format,id=scsi0-0-0-0,bootindex=1,write-cache=on \ -device ide-cd,bus=ide.1,id=sata0-0-1 \ -device ide-cd,bus=ide.2,id=sata0-0-2 \ -device ide-cd,bus=ide.3,id=sata0-0-3 \ -netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 \ -device virtio-net-pci,iommu_platform=on,ats=on,netdev=hostnet0,id=net0,mac=52:54:00:03:52:9c,bus=pci.1,addr=0x0 \ -add-fd set=3,fd=33 \ -chardev file,id=charserial0,path=/dev/fdset/3,append=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -audiodev '{"id":"audio1","driver":"none"}' \ -vnc 0.0.0.0:1,audiodev=audio1 \ -device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \ -device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \ -set device.net0.aer=on \ -set device.scsi0.aer=on \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2.connect vncviewer to VM Check virtio-net-pci device ATS ability: Actual results: No PCI ATS support property Device Manager -> Red Hat VirtIO Ethernet Adapter -> Properties -> Details -> no PCI ATS support Expected results: Device Manager -> Red Hat VirtIO Ethernet Adapter -> Properties -> Details -> PCI ATS support Additional info: Change machine type from pc-q35-rhel9.0.0 to pc-q35-rhel8.5.0, device has PCI ATS support back. Device ATS ability in RHEL VM works well with pc-q35-rhel9.0.0
Additional info: Devices properties which also lost: PCI ACS capability register PCI ACS compatible up hierarchy PCI ACS support PACI ARI support PCI ECRC errors PCI atomics supported PCI correctable error mask PCI error reporting PCI firmware error handing PCI uncorrectable error mask PCI uncorrectable error severity
What about the same qemu cmdline with Linux guest? Since it's specific issue with virtio-net/vhost, copying Jason.
(In reply to jinl from comment #1) > Additional info: > Devices properties which also lost: > PCI ACS capability register > PCI ACS compatible up hierarchy > PCI ACS support > PACI ARI support > PCI ECRC errors > PCI atomics supported > PCI correctable error mask > PCI error reporting > PCI firmware error handing > PCI uncorrectable error mask > PCI uncorrectable error severity Is the device identified as a PCI-E device or not in the guest? Thanks
(In reply to Peter Xu from comment #2) > What about the same qemu cmdline with Linux guest? Linux guest works well with the same qemu cmdline.
(In reply to jason wang from comment #3) > (In reply to jinl from comment #1) > > Additional info: > > Devices properties which also lost: > > PCI ACS capability register > > PCI ACS compatible up hierarchy > > PCI ACS support > > PACI ARI support > > PCI ECRC errors > > PCI atomics supported > > PCI correctable error mask > > PCI error reporting > > PCI firmware error handing > > PCI uncorrectable error mask > > PCI uncorrectable error severity > > Is the device identified as a PCI-E device or not in the guest? > > Thanks virtio-net-pci device is under pcie-root-port, so is identified as a PCI-E device in the guest.
hit the same issue with RHEL 9.1 host
Related BZ#2089545
*** Bug 2089545 has been marked as a duplicate of this bug. ***
MS support ticket: 2210030030001461
hit the same issue with win11 22H2 host env: qemu-kvm-7.1.0-1.el9.x86_64 kernel-5.14.0-171.el9.x86_64 guest cmd: /usr/libexec/qemu-kvm \ -name guest=win11,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-9-win11/master-key.aes"}' \ -blockdev '{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win11_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-q35-rhel9.0.0,usb=off,smm=on,kernel_irqchip=split,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \ -accel kvm \ -cpu Icelake-Server,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,avx512ifma=on,sha-ni=on,rdpid=on,fsrm=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,mpx=off,intel-pt=off,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on,hv-evmcs=on \ -global driver=cfi.pflash01,property=secure,value=on \ -m 8192 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \ -overcommit mem-lock=off \ -smp 4,sockets=1,dies=1,cores=4,threads=1 \ -uuid 28807dcd-5c36-4ac8-ba2c-4d4cedbe998a \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=22,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"eim":"on","device-iotlb":true}' \ -device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \ -device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \ -device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \ -device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \ -device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \ -device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \ -device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \ -device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \ -device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \ -device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \ -device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \ -device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \ -device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \ -device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.3","addr":"0x0"}' \ -device '{"driver":"virtio-scsi-pci","iommu_platform":true,"ats":true,"id":"scsi0","bus":"pci.2","addr":"0x0"}' \ -blockdev '{"driver":"file","filename":"/home/win10.qcow2","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-4-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-4-storage","backing":null}' \ -device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-4-format","id":"scsi0-0-0-0","bootindex":1,"write-cache":"on"}' \ -device '{"driver":"ide-cd","bus":"ide.1","id":"sata0-0-1"}' \ -device '{"driver":"ide-cd","bus":"ide.2","id":"sata0-0-2"}' \ -device '{"driver":"ide-cd","bus":"ide.3","id":"sata0-0-3"}' \ -netdev tap,fd=24,vhost=on,vhostfd=26,id=hostnet0 \ -device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"netdev":"hostnet0","id":"net0","mac":"52:54:00:dc:d0:4c","bus":"pci.1","addr":"0x0"}' \ -add-fd set=0,fd=23,opaque=serial0-source \ -chardev file,id=charserial0,path=/dev/fdset/0,append=on \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \ -chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/9-win11-swtpm.sock \ -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \ -device '{"driver":"tpm-crb","tpmdev":"tpm-tpm0","id":"tpm0"}' \ -device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -vnc 0.0.0.0:0,audiodev=audio1 \ -device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.4","addr":"0x0"}' \
Issue summary: ================ Windows does not detect ATS property for Ethernet PCI controller when firmware masks PCIE native hotplug capability in _OSC control Versions of windows affected: ============================= Server versions: Windows server 2016, windows server 2019, windows server 2022 Desktop versions: Windows 10, Windows 11. From our testing, the versions of Windows earlier than those mentioned above do not report/check for ATS availability even when the firmware does not mask PCIE native hotplug in _OSC. It seems reporting ATS status is only available in the above newer versions of the OS. How to reproduce the issue? ============================ We have used the QEMU hypervisor to test for this issue. Please refer to the following RedHat BZ for the QEMU commandline used: https://bugzilla.redhat.com/show_bug.cgi?id=2089545 Additionally, one can use the following scripts: https://github.com/ani-sinha/misc/blob/master/test-ats.sh https://github.com/ani-sinha/misc/blob/master/test-ats-win11.sh (Please adjust the locations of the images/binaries as per your host). Note that the ethernet controller is attached to the pcie root port which in turn is attached to the pci root complex. The ATS property is for the pcie root port as mentioned in Intel VTD spec (1). "--global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on/off" will mask/unmask the PCIE native hotplug reporting in _OSC from the firmware in QEMU and thus the issue can be observed easily. Please see (2). "on" will mask the bit, "off" will unmask. "-snapshot" command line has been used with the disk image to make sure there are no caching effect within the guest OS. Further, we have tested by removing and re-adding the ethernet controller from within the guest OS. The results are the same. References: (1) Intel Virtualization technology for Directed I/O, section 8.5 - Root Port ATS capability reporting structure. (2) ACPI spec 5.1, section 6.2.11.3 - OSC Implementation Example for PCI Host Bridge Devices