Bug 1414081

Summary: qemu-kvm-ma: Emulated GICv3 doesn't support MSI
Product: Red Hat Enterprise Linux 7 Reporter: Richard W.M. Jones <rjones>
Component: qemu-kvm-maAssignee: Eric Auger <eric.auger>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: abologna, chayang, drjones, eric.auger, hhuang, jcm, juzhang, laine, libvirt-maint, rbalakri, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: aarch64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-05 15:17:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 910269, 1173755    
Attachments:
Description Flags
Log file.
none
guestfs-6e3g0js3v3zp0l1e.log none

Description Richard W.M. Jones 2017-01-17 17:13:00 UTC
Description of problem:

libguestfs has a test in the test suite where we force TCG.  However
that fails with the error:

Original error from libvirt: internal error: process exited while connecting to monitor: 2017-01-17T17:10:51.315581Z qemu-system-aarch64: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: MSI is not supported by interrupt controller
2017-01-17T17:10:51.316906Z qemu-system-aarch64: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: Device initialization failed [code=1 int1=-1]
FAIL qemu-force-tcg.sh (exit status: 1)

Version-Release number of selected component (if applicable):

libvirt 3.0.0 rc2

How reproducible:

100%

Steps to Reproduce:
1. In libguestfs directory, do:
   $ make -C tests/qemu check TESTS=qemu-force-tcg.sh

Comment 1 Richard W.M. Jones 2017-01-17 17:15:02 UTC
qemu-2.7.0-7.fc25.aarch64

Comment 2 Richard W.M. Jones 2017-01-17 17:19:56 UTC
Created attachment 1241897 [details]
Log file.

Comment 3 Richard W.M. Jones 2017-01-17 17:27:09 UTC
Created attachment 1241899 [details]
guestfs-6e3g0js3v3zp0l1e.log

qemu command line and error

Comment 4 Laine Stump 2017-01-17 18:22:19 UTC
Not surprisingly, this can be reproduced by creating an aarch64 guest on an x86 host. switching back to virtio-mmio addresses of course eliminates the error.

Comment 5 Andrea Bolognani 2017-01-18 14:13:16 UTC
The problem is

  -machine virt-2.7,...,gic-version=3

libvirt automatically adds

  <feature>
    <gic version='3'/>
  </feature>

to the guest XML passed in by libguestfs because QEMU
advertises emulated GICv3 support:

  $ /usr/libexec/qemu-kvm -M none -qmp stdio
  {"QMP": {"version": {"qemu": {"micro": 0, "minor": 8, "major": 2},
                       "package": "(qemu-kvm-rhev-2.8.0-2.el7)"},
           "capabilities": []}}
  {"execute": "qmp_capabilities"}
  {"return": {}}
  {"execute": "query-gic-capabilities"}
  {"return": [{"emulated": true, "version": 3, "kernel": false},
              {"emulated": true, "version": 2, "kernel": true}]}

but apparently the emulated GICv3, unlike the in-kernel
one, doesn't support MSI.

Adding

  <feature>
    <gic version='2'/>
  </feature>

to the guest XML when using TCG should work around the issue
without having to give up PCI, but the proper way to fix it
would be for QEMU to grow MSI support in the emulated GICv3.

A minimal reproducer that doesn't require libguestfs or
libvirt is:

  $ /usr/libexec/qemu-kvm \
      -nographic -nodefaults \
      -M virt,accel=tcg,gic-version=3 \
      -device ioh3420

Reassigning to qemu-kvm-rhev.

Comment 6 Eric Auger 2017-01-18 16:17:58 UTC
The MSI controller used along with GICv3 is the GICv3 ITS. Pure QEMU emulated model is not supported at the moment. Only KVM ITS in-kernel model is supported. If this is an important feature this is a task I could carry on.

In case you use GICv2, MSI should be supported since the MSI controller being used is different: it is a GICv2M.

Thanks

Eric

Comment 7 Andrea Bolognani 2017-01-18 16:46:18 UTC
(In reply to Eric Auger from comment #6)
> The MSI controller used along with GICv3 is the GICv3 ITS. Pure QEMU
> emulated model is not supported at the moment. Only KVM ITS in-kernel model
> is supported.

Is it though?

  $ /usr/libexec/qemu-kvm \
      -nographic -nodefaults \
      -M virt,accel=kvm,gic-version=3 -cpu host \
      -device ioh3420
  qemu-kvm: -device ioh3420: MSI is not supported by interrupt controller
  qemu-kvm: -device ioh3420: Device initialization failed

This is on a GICv3-equipped ThunderX host. I was under the
impression such a combination would work, but that doesn't
seem to be the case after all.

Comment 8 Eric Auger 2017-01-18 17:00:43 UTC
It should but you need at least a 4.8 kernel (in-kernel ITS) and a QEMU 2.8 (ITS KVM device + related ACPI tables)

Thanks

Eric

Comment 12 Andrea Bolognani 2017-01-19 14:03:51 UTC
(In reply to Eric Auger from comment #8)
> It should but you need at least a 4.8 kernel (in-kernel ITS) and a QEMU 2.8
> (ITS KVM device + related ACPI tables)

I tried with kernel 4.9, QEMU master and libvirt 3.0.0 on a
GICv3 machine and I didn't have any problem running a guest,
so it seems the KVM situation will sort itself out as newer
packages make it into distributions.

For TCG guests though, we'll have to either stick with the
emulated GICv2 or implement MSI support in the emulated
GICv3.

Comment 13 Andrea Bolognani 2017-05-11 16:11:25 UTC
Eric, any update on the progress?

Bug 1449837 has been reported yesterday and it's an example
of people being affected by the lack of GICv3 MSI controller
emulation in QEMU.

See my comment there for how we will probably end up working
around the issue in the short run.

Comment 14 Eric Auger 2017-05-12 07:15:01 UTC
Job not started as of now as I work on the smmu emulation model at the moment. Developing the emulation code for ITS is a huge work and we should not expect this to come soon.

Comment 15 Andrea Bolognani 2017-05-12 08:24:03 UTC
Okay, thanks for the update.

When the MSI controller will eventually be implemented, will
it show up in the device list like arm-gicv2m already does?

I'm asking because libvirt will want to decide between using
GICv2 and GICv3 based on the availability of the MSI
controller, so it will need a way to figure out whether the
QEMU binary supports it, and it showing up in the device
list would do the trick.

Comment 16 Eric Auger 2017-05-12 10:29:16 UTC
I guess so. I don't see any reason why it wouldn't.

Comment 17 Jon Masters 2017-08-31 19:51:43 UTC
No longer a bug?

Comment 18 Richard W.M. Jones 2017-08-31 20:14:38 UTC
If the reproducer in comment 5 is correct, then I believe the
bug still exists even upstream:

$ /usr/bin/qemu-system-aarch64 -nographic -nodefaults -M virt,accel=tcg,gic-version=3 -device ioh3420
qemu-system-aarch64: -device ioh3420: MSI is not supported by interrupt controller

Package: qemu-system-aarch64-core-2.10.0-0.4.rc4.fc28.aarch64
Host: Linux mustang.home.annexia.org 4.13.0-0.rc6.git0.1.fc28.aarch64 #1 SMP Mon Aug 21 14:16:58 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

For completeness however I also tried the reproducer on RHEL 7.4
with the batcave repo enabled:

$ /usr/libexec/qemu-kvm \
>       -nographic -nodefaults \
>       -M virt,accel=tcg,gic-version=3 \
>       -device ioh3420
qemu-kvm: -device ioh3420: MSI is not supported by interrupt controller

Package: qemu-kvm-rhev-2.9.0-22.el7a.aarch64
Host: Linux harm.home.annexia.org 4.5.0-15.4.2.el7.aarch64 #1 SMP Wed Mar 22 15:58:27 EDT 2017 aarch64 aarch64 aarch64 GNU/Linux

Comment 19 Eric Auger 2017-09-01 07:20:34 UTC
Nothing has changed on QEMU side and there is still no emulated device for ITS. So if you launch qemu directly with virt,gic_version=3 that's normal you still hit the issue. I think the problem was worked around at libvirt level by choosing by default gic_version=2 and hence using gicv2m MSI controller instead. This latter has some emulated code.

Comment 20 Andrea Bolognani 2017-09-02 09:13:57 UTC
For reference, the libvirt bug used to track the workaround
is Bug 1450433.

Comment 22 Eric Auger 2018-02-05 15:17:59 UTC
We decided to close this BZ as there is a workaround at libvirt level, consisting in using GICv2 + GICv2m MSI controller and it is TCG only. The main limitation when using GICv2m is number of vcpus is capped at 8.