qemu-2.7.0-7.fc25.aarch64

Created attachment 1241897 [details]
Log file.

Created attachment 1241899 [details]
guestfs-6e3g0js3v3zp0l1e.log

qemu command line and error

Not surprisingly, this can be reproduced by creating an aarch64 guest on an x86 host. switching back to virtio-mmio addresses of course eliminates the error.

The problem is

  -machine virt-2.7,...,gic-version=3

libvirt automatically adds

  <feature>
    <gic version='3'/>
  </feature>

to the guest XML passed in by libguestfs because QEMU
advertises emulated GICv3 support:

  $ /usr/libexec/qemu-kvm -M none -qmp stdio
  {"QMP": {"version": {"qemu": {"micro": 0, "minor": 8, "major": 2},
                       "package": "(qemu-kvm-rhev-2.8.0-2.el7)"},
           "capabilities": []}}
  {"execute": "qmp_capabilities"}
  {"return": {}}
  {"execute": "query-gic-capabilities"}
  {"return": [{"emulated": true, "version": 3, "kernel": false},
              {"emulated": true, "version": 2, "kernel": true}]}

but apparently the emulated GICv3, unlike the in-kernel
one, doesn't support MSI.

Adding

  <feature>
    <gic version='2'/>
  </feature>

to the guest XML when using TCG should work around the issue
without having to give up PCI, but the proper way to fix it
would be for QEMU to grow MSI support in the emulated GICv3.

A minimal reproducer that doesn't require libguestfs or
libvirt is:

  $ /usr/libexec/qemu-kvm \
      -nographic -nodefaults \
      -M virt,accel=tcg,gic-version=3 \
      -device ioh3420

Reassigning to qemu-kvm-rhev.

The MSI controller used along with GICv3 is the GICv3 ITS. Pure QEMU emulated model is not supported at the moment. Only KVM ITS in-kernel model is supported. If this is an important feature this is a task I could carry on.

In case you use GICv2, MSI should be supported since the MSI controller being used is different: it is a GICv2M.

Thanks

Eric

(In reply to Eric Auger from comment #6)
> The MSI controller used along with GICv3 is the GICv3 ITS. Pure QEMU
> emulated model is not supported at the moment. Only KVM ITS in-kernel model
> is supported.

Is it though?

  $ /usr/libexec/qemu-kvm \
      -nographic -nodefaults \
      -M virt,accel=kvm,gic-version=3 -cpu host \
      -device ioh3420
  qemu-kvm: -device ioh3420: MSI is not supported by interrupt controller
  qemu-kvm: -device ioh3420: Device initialization failed

This is on a GICv3-equipped ThunderX host. I was under the
impression such a combination would work, but that doesn't
seem to be the case after all.

It should but you need at least a 4.8 kernel (in-kernel ITS) and a QEMU 2.8 (ITS KVM device + related ACPI tables)

Thanks

Eric

(In reply to Eric Auger from comment #8)
> It should but you need at least a 4.8 kernel (in-kernel ITS) and a QEMU 2.8
> (ITS KVM device + related ACPI tables)

I tried with kernel 4.9, QEMU master and libvirt 3.0.0 on a
GICv3 machine and I didn't have any problem running a guest,
so it seems the KVM situation will sort itself out as newer
packages make it into distributions.

For TCG guests though, we'll have to either stick with the
emulated GICv2 or implement MSI support in the emulated
GICv3.

Eric, any update on the progress?

Bug 1449837 has been reported yesterday and it's an example
of people being affected by the lack of GICv3 MSI controller
emulation in QEMU.

See my comment there for how we will probably end up working
around the issue in the short run.

Job not started as of now as I work on the smmu emulation model at the moment. Developing the emulation code for ITS is a huge work and we should not expect this to come soon.

Okay, thanks for the update.

When the MSI controller will eventually be implemented, will
it show up in the device list like arm-gicv2m already does?

I'm asking because libvirt will want to decide between using
GICv2 and GICv3 based on the availability of the MSI
controller, so it will need a way to figure out whether the
QEMU binary supports it, and it showing up in the device
list would do the trick.

I guess so. I don't see any reason why it wouldn't.

No longer a bug?

If the reproducer in comment 5 is correct, then I believe the
bug still exists even upstream:

$ /usr/bin/qemu-system-aarch64 -nographic -nodefaults -M virt,accel=tcg,gic-version=3 -device ioh3420
qemu-system-aarch64: -device ioh3420: MSI is not supported by interrupt controller

Package: qemu-system-aarch64-core-2.10.0-0.4.rc4.fc28.aarch64
Host: Linux mustang.home.annexia.org 4.13.0-0.rc6.git0.1.fc28.aarch64 #1 SMP Mon Aug 21 14:16:58 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

For completeness however I also tried the reproducer on RHEL 7.4
with the batcave repo enabled:

$ /usr/libexec/qemu-kvm \
>       -nographic -nodefaults \
>       -M virt,accel=tcg,gic-version=3 \
>       -device ioh3420
qemu-kvm: -device ioh3420: MSI is not supported by interrupt controller

Package: qemu-kvm-rhev-2.9.0-22.el7a.aarch64
Host: Linux harm.home.annexia.org 4.5.0-15.4.2.el7.aarch64 #1 SMP Wed Mar 22 15:58:27 EDT 2017 aarch64 aarch64 aarch64 GNU/Linux

Nothing has changed on QEMU side and there is still no emulated device for ITS. So if you launch qemu directly with virt,gic_version=3 that's normal you still hit the issue. I think the problem was worked around at libvirt level by choosing by default gic_version=2 and hence using gicv2m MSI controller instead. This latter has some emulated code.

For reference, the libvirt bug used to track the workaround
is Bug 1450433.

We decided to close this BZ as there is a workaround at libvirt level, consisting in using GICv2 + GICv2m MSI controller and it is TCG only. The main limitation when using GICv2m is number of vcpus is capped at 8.