Bug 1165290 - No KVM virtualization on APM Mustang and other boards (3.17.4/f21 and 3.18-rc/f22)
Summary: No KVM virtualization on APM Mustang and other boards (3.17.4/f21 and 3.18-rc...
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: aarch64
OS: Unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
: 1208768 1263499 (view as bug list)
Depends On:
Blocks: ARM64, F-ExcludeArch-aarch64
TreeView+ depends on / blocked
Reported: 2014-11-18 18:31 UTC by Marcin Juszkiewicz
Modified: 2015-09-28 11:31 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2015-05-10 19:44:27 UTC
Type: Bug

Attachments (Terms of Use)
Hacky workaround from Alexander Graf (3.59 KB, application/mbox)
2014-11-18 18:31 UTC, Marcin Juszkiewicz
no flags Details
DeviceTreeBlob with fixed gicv entry (13.95 KB, application/octet-stream)
2015-03-31 15:51 UTC, Marcin Juszkiewicz
no flags Details
DeviceTreeBlob with fixed gicv entry and 1GbE as eth0 (17.56 KB, text/plain)
2015-04-08 10:08 UTC, Marcin Juszkiewicz
no flags Details

Description Marcin Juszkiewicz 2014-11-18 18:31:26 UTC
Created attachment 958690 [details]
Hacky workaround from Alexander Graf

Description of problem:

There was a kernel change [1] which ended in disabling virtualization on APM Mustang when kernel with 64KB pages is used (which is Fedora default). 

Hacky workaround was created by Alexander Graf from SUSE (attached)

1. https://git.kernel.org/cgit/linux/kernel/git/kvmarm/kvmarm.git/commit/virt/kvm/arm/vgic.c?id=63afbe7a0ac184ef8485dac4914e87b211b5bfaa

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. boot kernel on Mustang

Actual results:

[    0.343796] kvm [1]: GICV size 0x2000 not a multiple of page size 0x10000
[    0.343802] kvm [1]: error: no compatible GIC info found
[    0.343909] kvm [1]: error initializing Hyp mode: -6

Expected results:

[    0.343799] kvm [1]: interrupt-controller@780c0000 IRQ25
[    0.343903] kvm [1]: timer IRQ30
[    0.343912] kvm [1]: Hyp mode initialized successfully

Additional info:

Comment 1 Marcin Juszkiewicz 2014-11-18 18:34:23 UTC
IRC log from #kvm-arm from 17 November 2014 (CET timestamps)

09:29 < hrw> hi
09:30 < hrw> can someone boot 64KB pages kernel on Juno or Mustang and get kvm?
09:30 < hrw> [    0.343796] kvm [1]: GICV size 0x2000 not a multiple of page size 0x10000
09:30 < hrw> [    0.343802] kvm [1]: error: no compatible GIC info found
09:30 < hrw> [    0.343909] kvm [1]: error initializing Hyp mode: -6
09:30 < hrw> is what I got on 3.18-rc4 64K from fedora/rawhide
09:32 < hrw> bbiab
10:27 < suihkulokki> hrw: using 3.18-rc5 and mustang I get the same error with 64KB pages - 4KB pages work ok
10:31 < chazy> hrw: suihkulokki: see this commit: https://git.kernel.org/cgit/linux/kernel/git/kvmarm/kvmarm.git/commit/virt/kvm/arm/vgic.c?id=63afbe7a0ac184ef8485dac4914e87b211b5bfaa
10:32 < chazy> iirc, you can fix the DT to report things properly so that this check will pass and everything works (at least on Seattle), but for APM mustang I did think carefully about the proper fix.
10:33 < chazy> if you want to hack on it, and don’t care about stability etc., you can simply remove those two PAGE_ALIGNED checks (now moved to vgic-v2.c) and I believe things will work, but beware, you’re venturing into  danger land.
10:37 < suihkulokki> chazy: I think hrw and fedora would be more interested in fixing the dtb
10:39 < chazy> suihkulokki: I realize that, but I think they would have to work with APM on that one.  Not sure what the long term plan for juno was on this.
12:41 < agraf> chazy: I have a local patch that relaxes this check slightly
12:42 < agraf> chazy: http://paste.debian.net/132128/
12:42 < agraf> chazy: need to make this a kernel module option that defaults to off
12:42 < agraf> chazy: but you get the idea ;)
12:43 < agraf> chazy: the problem is that on juno qemu needs to align the vgic at the same offset as real hardware has it
12:43 < hrw> chazy: seattle has gicv3 iirc
12:44 < hrw> suihkulokki: at RH we tend to switch for acpi asap. now probably only pci is not fully tested
12:44 < agraf> hrw: nope, it’s v2+msi
12:45 < hrw> ah, right. Thunder is v3
12:45 < agraf> yup, thunder is the first v3 hw you can get fwiw
12:48 < hrw> chazy: I can also go to 3.17 kernel and not have that check ;D

Comment 2 Marcin Juszkiewicz 2014-11-25 16:54:34 UTC
3.18.0-0.rc6.git0.1.fc22.aarch64 is affected too

Comment 3 Brendan Conoboy 2014-11-26 19:40:04 UTC
FYI, we think this will be resolved by a firmware update.  We are working with APM to achieve this.

Comment 4 Marcin Juszkiewicz 2014-12-04 17:14:06 UTC
Fresh install of Fedora 21 also lacks nationalization support:

[root@pinkiepie virt]# virsh create hrw-f21.xml 
błąd: Utworzenie domeny z hrw-f21.xml nie powiodło się
błąd: unsupported configuration: Domain requires KVM, but it is not available. Check that virtualisation is enabled in the host BIOS, and host configuration is setup to load the kvm modules.

[root@pinkiepie virt]# dmesg|grep -i kvm
[    0.303959] kvm [1]: GICV size 0x2000 not a multiple of page size 0x10000
[    0.303965] kvm [1]: error: no compatible GIC info found
[    0.304070] kvm [1]: error initializing Hyp mode: -6
[root@pinkiepie virt]# uname -a
Linux pinkiepie 3.17.4-301.fc21.aarch64 #1 SMP Sun Nov 30 20:41:43 UTC 2014 aarch64 aarch64 aarch64 GNU/Linux

Comment 8 Jaroslav Reznik 2015-03-03 16:31:03 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:

Comment 9 Marcin Juszkiewicz 2015-03-31 15:51:23 UTC
Created attachment 1009121 [details]
DeviceTreeBlob with fixed gicv entry

Grab fixed DTB, put it in /boot/ directory and add one line into GRUB2 config:

devicetree /boot/mustang.dtb

Then kernel will boot and will have KVM support (tested with 4.0-rc5).

Comment 10 Michael Tremer 2015-03-31 16:17:58 UTC
Can confirm that KVM is working again with that DTB file on kernel 4.0.0-0.rc6.git0.1.fc23.aarch64

Comment 12 Richard W.M. Jones 2015-04-08 09:10:37 UTC
Still broken in Rawhide (without workarounds).

Comment 13 Marcin Juszkiewicz 2015-04-08 09:20:08 UTC
Richard: issue is in firmware not in kernel. As long as we do not have fixed firmware available bug will happen.

Comment 14 Richard W.M. Jones 2015-04-08 09:23:21 UTC
So ..  Anyone give me a clue exactly how to implement the workaround?

I dropped the mustang.dtb file into /boot and added:

devicetree /boot/mustang.dtb

to the end of /boot/grub2/grub.cfg, and rebooted, but I still see
the same error:

$ dmesg | grep -i kvm
[    0.725483] kvm [1]: GICV size 0x2000 not a multiple of page size 0x10000
[    0.725576] kvm [1]: error initializing Hyp mode: -6

Comment 15 Richard W.M. Jones 2015-04-08 09:48:36 UTC
I also tried appending the devicetree line to
/boot/efi/EFI/fedora/grub.cfg but that also doesn't appear to
make any difference.

Comment 16 Richard W.M. Jones 2015-04-08 09:56:00 UTC
The answer is you have to edit /boot/efi/EFI/fedora/grub.cfg,
find the latest kernel section in that file, and add the
devicetree line to that kernel section.  You end up with:

        linux /vmlinuz-4.0.0-0.rc6.git2.1.fc23.aarch64 root=UUID=9d1bb62f-5d50-458e-be91-36af494ccfa0 ro  console=ttyS0,115200 earlyprintk=uart8250-32bit,0x1c020000 LANG=en_GB.UTF-8
        initrd /initramfs-4.0.0-0.rc6.git2.1.fc23.aarch64.img
        devicetree /mustang.dtb

This fixed KVM for me.  However it also broke stable network
addressing.  It seems that the first NIC now gets a randomized
MAC address each time.

Comment 17 Marcin Juszkiewicz 2015-04-08 10:02:01 UTC
Have to check and reorder Ethernet devices.

Comment 18 Marcin Juszkiewicz 2015-04-08 10:08:48 UTC
Created attachment 1012161 [details]
DeviceTreeBlob with fixed gicv entry and 1GbE as eth0

This DTB has 1GbE as eth0 (was eth2 in previous DTB), working USB and KVM:

13:46 hrw@pinkiepie-rawhide:~$ dmesg|grep kvm -i
[    0.871608] kvm [1]: interrupt-controller@780c0000 IRQ5
[    0.877167] kvm [1]: timer IRQ3
[    0.880426] kvm [1]: Hyp mode initialized successfully
13:47 hrw@pinkiepie-rawhide:~$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 125f:de7a A-DATA Technology Co., Ltd. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 1234:2088 Brain Actuated Technologies 
Bus 001 Device 002: ID 05e3:0606 Genesys Logic, Inc. USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
13:47 hrw@pinkiepie-rawhide:~$ ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet  netmask  broadcast
        inet6 fdd2:2a70:985d::c0a  prefixlen 128  scopeid 0x0<global>
        inet6 fdd2:2a70:985d:0:201:73ff:fe02:b73  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::201:73ff:fe02:b73  prefixlen 64  scopeid 0x20<link>
        ether 00:01:73:02:0b:73  txqueuelen 1000  (Ethernet)
        RX packets 223  bytes 25152 (24.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 198  bytes 47959 (46.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Comment 19 Marcin Juszkiewicz 2015-04-08 21:52:12 UTC
*** Bug 1208768 has been marked as a duplicate of this bug. ***

Comment 20 Itaru Kitayama 2015-05-07 23:42:46 UTC
Version 1.1.0 firmware fixed this issue on Mustang. KVM is properly initialised
at boot time.

Comment 21 Richard W.M. Jones 2015-05-10 19:24:49 UTC
I can confirm also that 1.1.0 firmware fixes the problem, and
has a stable MAC address.  I suggest we can just close this bug
unless anyone objects?

Comment 22 Marcin Juszkiewicz 2015-05-10 19:44:27 UTC
As bug was not in a kernel but in DTB provided by firmware and latest official firmware (available to all users of affected platform) fixed problem I am closing this bug.

Comment 23 Cole Robinson 2015-09-16 14:11:18 UTC
*** Bug 1263499 has been marked as a duplicate of this bug. ***

Comment 24 Itaru Kitayama 2015-09-28 06:20:11 UTC
Changing the page size from 64KB (Fedora's default) to 4KB seems to fix this issue
(firmware version 1.1.0-rh-0.15), hoping new firmware .20 from APM handles the 64KB kernel as well.

Comment 25 Marcin Juszkiewicz 2015-09-28 11:31:23 UTC
Itaru: 4K kernels are something which is not supported. 

You should/have to upgrade your firmware.

Note You need to log in before you can comment on or make changes to this bug.