Bug 2178885

Yeah, this is because 'virsh capabilities' looks whether '/sys/kernel/iommu_groups/' exists (and is not empty). The virt-host-validate, on the other hand, that particular check looks whether there's HW support (and whether it's enabled). I'm not sure how to check for SMMU presence on ARM though.

Patch posted upstream:

https://listman.redhat.com/archives/libvir-list/2023-March/238985.html

Merged upstream as:

commit 2c13a2a7c9c368ea81eccd4ba12d9cf34bdd331b
Author:     Michal Prívozník <mprivozn>
AuthorDate: Wed Mar 22 16:33:32 2023 +0100
Commit:     Michal Prívozník <mprivozn>
CommitDate: Fri Mar 24 14:44:38 2023 +0100

    virt-host-validate: Detect SMMU support on ARMs
    
    In vir-host-validate we do two checks related to IOMMU:
    
      1) hardware support, and
      2) kernel support.
    
    While users are usually interested in the latter, the former also
    makes sense. And for the former (hardware support) we have this
    huge if-else block for nearly every architecture, except ARM.
    
    Now, IOMMU is called SMMU in ARM world, and while there's
    certainly a definitive way of detecting SMMU support (e.g. via
    dumping some registers in asm), we can work around this - just
    like we do for Intel and AMD - and check for an ACPI table
    presence.
    
    In ARM world, there's I/O Remapping Table (IORT) which describes
    SMMU capabilities on given host and is exposed in sysfs
    (regardless of arm_smmu module).
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2178885
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Ján Tomko <jtomko>

v9.1.0-286-g2c13a2a7c9

Eric,

can you please confirm whether this approach (checking if
/sys/firmware/acpi/tables/IORT exists on the system) is the correct
one for detecting IOMMU support at the hardware level?

Thanks!

(In reply to Andrea Bolognani from comment #5)
> Eric,
> 
> can you please confirm whether this approach (checking if
> /sys/firmware/acpi/tables/IORT exists on the system) is the correct
> one for detecting IOMMU support at the hardware level?
> 
> Thanks!

Hum I am afraid this is not sufficient. The IORT table can exist and simply describe the routing between PCI requester ID and ITS MSI controller for example, ie. it describes other stuff than SMMU routing (as opposed to DMAR table). So the IORT may exist without any SMMU. You need to look inside the IORT and scan for the SMMU node. Can't you look at the groups in sysfs? Adding Jerry in cc.

(In reply to Eric Auger from comment #6)
> (In reply to Andrea Bolognani from comment #5)
> > Eric,
> > 
> > can you please confirm whether this approach (checking if
> > /sys/firmware/acpi/tables/IORT exists on the system) is the correct
> > one for detecting IOMMU support at the hardware level?
> > 
> > Thanks!
> 
> Hum I am afraid this is not sufficient. The IORT table can exist and simply
> describe the routing between PCI requester ID and ITS MSI controller for
> example, ie. it describes other stuff than SMMU routing (as opposed to DMAR
> table). So the IORT may exist without any SMMU. You need to look inside the
> IORT and scan for the SMMU node.

Yeah, I worried that we'd need to do this. Although, I expected there to be very few pieces of HW that do have IORT table without SMMU.

> Can't you look at the groups in sysfs?

Nope. There are two checks that virt-host-validate does:

  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS

The first one checks for HW support, the other for kernel support. Now, we want the first one to PASS even if for instance IOMMU wasn't enabled in kernel or corresponding kernel module isn't loaded (and thus there's no sysfs hierarchy). Now, the latter can't exist without the former, but the former can exist without the latter.

> Adding Jerry in cc.

Thanks.

I just posted a merge request to beaker system scan this weekend to start populating system inventory with data for arm systems with SMMUs, so this is fresh in my memory. Like Eric for a system with acpi tables you need to look for nodes of type 3 (SMMUv1 and SMMUv2), or 4 (SMMUv3).

Does this code need to check other architectures as well? I think unless someone is forcing disable, SMMU should be enabled by the kernel on aarch64 systems if it is there. It is also enabled by the kernel if it is there on AMD x86 systems. Intel x86 systems, for historical reasons, need intel_iommu=on for the iommu to be enabled by the kernel. In the x86 cases, the IVRS table (AMD) and the DMAR table (Intel) will signify that there is an iommu enabled by the firmware.

Speaking of arm and smmu, does anyone know of an arm system in beaker using devicetree that has an smmu that they can point me to?

I should have clarified in comment #8 that the when talking about whether the SMMU or IOMMU is enabled that was in relation to our RHEL kernel configurations.

Just to set the context here: libvirt offers this 'virt-host-validate' binary, which aims on verifying that all virtualization bits are enabled on the host (and if not then suggest how to enable them). As a part of that, the tool checks for IOMMU (well SMMU on arm). The source code can be found here:

https://gitlab.com/libvirt/libvirt/-/blob/master/tools/virt-host-validate-common.c#L328

as noted earlier, there are two different checks done by the tool:

  a) IOMMU presence (well, ennoblement) at HW level, and
  b) IOMMU availability in kernel.

Now, for different architectures, the tool checks for different symptoms. For instance, on Intel, plain presence of DMAR ACPI table is sufficient to satisfy a). On AMD it's IVRS ACPI table that satisfies a). BTW: "presence" of these tables is checked as presence of corresponding /sys/firmware/acpi/tables/* file. It's my understanding after reading both specifications that either table can be virtually empty in which case the host lacks IOMMU capabilities. But I believe there are virtually zero BIOS vendors who would implement an empty DMAR/IVRS table. So I guess the question then is whether IORT table falls into the same category. If not, then we really need to parse it (ugrh).

(In reply to Michal Privoznik from comment #11)
> Just to set the context here: libvirt offers this 'virt-host-validate'
> binary, which aims on verifying that all virtualization bits are enabled on
> the host (and if not then suggest how to enable them). As a part of that,
> the tool checks for IOMMU (well SMMU on arm). The source code can be found
> here:
> 
> https://gitlab.com/libvirt/libvirt/-/blob/master/tools/virt-host-validate-
> common.c#L328
> 
> as noted earlier, there are two different checks done by the tool:
> 
>   a) IOMMU presence (well, ennoblement) at HW level, and
>   b) IOMMU availability in kernel.
> 
> Now, for different architectures, the tool checks for different symptoms.
> For instance, on Intel, plain presence of DMAR ACPI table is sufficient to
> satisfy a). On AMD it's IVRS ACPI table that satisfies a). BTW: "presence"
> of these tables is checked as presence of corresponding
> /sys/firmware/acpi/tables/* file. It's my understanding after reading both
> specifications that either table can be virtually empty in which case the
> host lacks IOMMU capabilities. But I believe there are virtually zero BIOS
> vendors who would implement an empty DMAR/IVRS table. So I guess the
> question then is whether IORT table falls into the same category. If not,
> then we really need to parse it (ugrh).

No you really need to parse it. IORT describes many other stuff than just the SMMU topology (MSI controller, PCIe root complex, ...). You need to parse it and look for smmu* nodes. That's the only way to make sure there is a SMMU described in the system.

IVRS requires there to be at least one IVHD block describing an IOMMU. DMAR requires at least one DRHD as well. I imagine those being empty would be considered broken firmware. 

The problem like Eric mentioned is that the IORT table can decscribe other things, so there won't necessarily be SMMU nodes.

(and Eric just posted at the same time :))

Alright. I've posted patches that parse IORT table here:

https://listman.redhat.com/archives/libvir-list/2023-April/239251.html

v2:

https://listman.redhat.com/archives/libvir-list/2023-April/239275.html

Merged upstream as:

d335f49c70b3db959e6c11ab9ccbbf5de3079965 virt-host-validate: Detect SMMU presence on ARMs by parsing IORT table
fc216db4fb789cbd30917be036d0b94d965bdf7f tests: Introduce viracpitest
1a5c864e9f2d53024f17a6be92d75b1d5c8225fb util: Introduce virAcpi module

v9.2.0-195-gd335f49c70

Change status to verified based on latest test result in comment#21

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6409