IIUC, on x86_64 we already have the intel-iommu device available. Is there some benefit from also having the virtio-iommu on x86_64 as well ?  Presumably the intel-iommu is more portable for guest OS since it is not reliant on the virtio transport.

So if we have 2 options on x86_674, what criteria should apps use to decide which to use ?

Hi Daniel. In the current state of the driver and qemu device I don't see any great added value compared to the intel-iommu as the perf will be worse than with the intel-iommu. See this old presentation, slide 22:

http://events17.linuxfoundation.org/sites/events/files/slides/viommu_arm.pdf

So at the moment apps should stick to the intel-iommu.

However, Jerry, our kernel IOMMU maintainer said he has some long standing requests to enable the feature for customer and mostly for Intel to start "playing" with it. I think potential customer are interested in next "promised" virtualization related optimizations that would be more likely implemented in the virtio-iommu driver rather than in a native iommu driver. Today most people now look at looming virtual shared virtual memory  support (ie. PASID). However PASID may be enabled along with intel-iommu first. This vSVM work is in progress and also depends on /dev/iommu.

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Verify with qemu-kvm-7.0.0-7.el9.src.rpm
guest kernel: kernel-5.14.0-104.mr955_220602_1540.el9.src.rpm

1)with intel host:
qemu command line:
/usr/libexec/qemu-kvm \
-name guest=v1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-5-v1/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/v1_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-rhel9.0.0,usb=off,smm=on,dump-guest-core=off,kernel_irqchip=split,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on \
-global driver=cfi.pflash01,property=secure,value=on \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=4,threads=1 \
-uuid ffe7972d-b950-4e96-9a27-a7414786848d \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=21,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"virtio-iommu","bus":"pcie.0","addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x1d.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x1d"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x1d.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x1d.0x2"}' \
-device '{"driver":"virtio-scsi-pci","iommu_platform":true,"ats":true,"id":"scsi0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/home/rhel8.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-1-format","id":"scsi0-0-0-0","bootindex":1}' \
-netdev tap,fd=24,vhost=on,vhostfd=28,id=hostnet0 \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"netdev":"hostnet0","id":"net0","mac":"52:54:00:e8:32:0e","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device '{"driver":"bochs-display","id":"video0","vgamem":16777216,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.3","addr":"0x0"}' \
-global virtio-net-pci.aer=on \
-global virtio-scsi-pci.aer=on \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

check dmesg:
[root@bootp-73-199-57 ~]# dmesg | grep iommu
[    0.388509] iommu: Default domain type: Translated 
[    0.389109] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.583798] virtio_iommu virtio0: input address: 64 bits
[    0.584329] virtio_iommu virtio0: page mask: 0xfffffffffffff000
[    0.588694] ehci-pci 0000:00:1d.7: Adding to iommu group 0
[    0.589248] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    0.606243] uhci_hcd 0000:00:1d.0: Adding to iommu group 0
[    0.614426] uhci_hcd 0000:00:1d.1: Adding to iommu group 0
[    0.622630] uhci_hcd 0000:00:1d.2: Adding to iommu group 0
[    1.032554] pcieport 0000:00:02.0: Adding to iommu group 1
[    1.033088] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    1.037331] pcieport 0000:00:02.1: Adding to iommu group 1
[    1.040324] pcieport 0000:00:02.2: Adding to iommu group 1
[    1.043271] pcieport 0000:00:02.3: Adding to iommu group 1
[    1.046349] virtio-pci 0000:01:00.0: Adding to iommu group 1
[    1.047832] virtio-pci 0000:02:00.0: Adding to iommu group 1
[    1.049222] virtio-pci 0000:03:00.0: Adding to iommu group 1
[    1.530908] ahci 0000:00:1f.2: Adding to iommu group 2
[    1.531691] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    3.296466] lpc_ich 0000:00:1f.0: Adding to iommu group 2
[    3.298084] i801_smbus 0000:00:1f.3: Adding to iommu group 2
[    3.393899] bochs-drm 0000:00:01.0: Adding to iommu group 3
[    3.394717] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA

check iommu group:
[root@bootp-73-199-57 ~]# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/3/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.3
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.1
/sys/kernel/iommu_groups/1/devices/0000:00:02.2
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:1f.2
/sys/kernel/iommu_groups/2/devices/0000:00:1f.0
/sys/kernel/iommu_groups/2/devices/0000:00:1f.3
/sys/kernel/iommu_groups/0/devices/0000:00:1d.1
/sys/kernel/iommu_groups/0/devices/0000:00:1d.2
/sys/kernel/iommu_groups/0/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:1d.7

2)with AMD host
qemu command line:
/usr/libexec/qemu-kvm \
-name guest=virtio-iommu-m,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-4-virtio-iommu-m/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/virtio-iommu-m_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-rhel9.0.0,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on \
-global driver=cfi.pflash01,property=secure,value=on \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=4,threads=1 \
-uuid cf73da8b-c0f2-41c4-af15-7174371eda91 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"virtio-iommu","bus":"pcie.0","addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x1d.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x1d"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x1d.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x1d.0x2"}' \
-device '{"driver":"virtio-scsi-pci","iommu_platform":true,"ats":true,"id":"scsi0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/home/RHEL-9.1.0-20220629.0-x86_64-ovmf.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-1-format","id":"scsi0-0-0-0","bootindex":1}' \
-netdev tap,fd=37,vhost=on,vhostfd=38,id=hostnet0 \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"netdev":"hostnet0","id":"net0","mac":"52:54:00:25:9e:ab","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,audiodev=audio1 \
-device '{"driver":"bochs-display","id":"video0","vgamem":16777216,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.3","addr":"0x0"}' \
-global virtio-net-pci.aer=on \
-global virtio-scsi-pci.aer=on \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

check dmesg:
[    0.532669] iommu: Default domain type: Translated 
[    0.533149] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.777849] virtio_iommu virtio0: input address: 64 bits
[    0.778735] virtio_iommu virtio0: page mask: 0xfffffffffffff000
[    0.785983] ehci-pci 0000:00:1d.7: Adding to iommu group 0
[    0.786922] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    0.818243] uhci_hcd 0000:00:1d.0: Adding to iommu group 0
[    0.838821] uhci_hcd 0000:00:1d.1: Adding to iommu group 0
[    0.858907] uhci_hcd 0000:00:1d.2: Adding to iommu group 0
[    1.394244] pcieport 0000:00:02.0: Adding to iommu group 1
[    1.395927] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    1.405418] pcieport 0000:00:02.1: Adding to iommu group 1
[    1.411870] pcieport 0000:00:02.2: Adding to iommu group 1
[    1.417895] pcieport 0000:00:02.3: Adding to iommu group 1
[    1.423859] virtio-pci 0000:01:00.0: Adding to iommu group 1
[    1.426363] virtio-pci 0000:02:00.0: Adding to iommu group 1
[    1.428792] virtio-pci 0000:03:00.0: Adding to iommu group 1
[    2.034911] ahci 0000:00:1f.2: Adding to iommu group 2
[    2.037388] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA
[    4.315824] lpc_ich 0000:00:1f.0: Adding to iommu group 2
[    4.349424] i801_smbus 0000:00:1f.3: Adding to iommu group 2
[    4.412281] bochs-drm 0000:00:01.0: Adding to iommu group 3
[    4.414419] iommu: Failed to allocate default IOMMU domain of type 11 for group (null) - Falling back to IOMMU_DOMAIN_DMA

check iommu group:
/sys/kernel/iommu_groups/3/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.3
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.1
/sys/kernel/iommu_groups/1/devices/0000:00:02.2
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:1f.2
/sys/kernel/iommu_groups/2/devices/0000:00:1f.0
/sys/kernel/iommu_groups/2/devices/0000:00:1f.3
/sys/kernel/iommu_groups/0/devices/0000:00:1d.1
/sys/kernel/iommu_groups/0/devices/0000:00:1d.2
/sys/kernel/iommu_groups/0/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:1d.7

Based on the Comment 6 test results, set this bug as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7967