Created attachment 958690 [details]
Hacky workaround from Alexander Graf
Description of problem:
There was a kernel change  which ended in disabling virtualization on APM Mustang when kernel with 64KB pages is used (which is Fedora default).
Hacky workaround was created by Alexander Graf from SUSE (attached)
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. boot kernel on Mustang
[ 0.343796] kvm : GICV size 0x2000 not a multiple of page size 0x10000
[ 0.343802] kvm : error: no compatible GIC info found
[ 0.343909] kvm : error initializing Hyp mode: -6
[ 0.343799] kvm : interrupt-controller@780c0000 IRQ25
[ 0.343903] kvm : timer IRQ30
[ 0.343912] kvm : Hyp mode initialized successfully
IRC log from #kvm-arm from 17 November 2014 (CET timestamps)
09:29 < hrw> hi
09:30 < hrw> can someone boot 64KB pages kernel on Juno or Mustang and get kvm?
09:30 < hrw> [ 0.343796] kvm : GICV size 0x2000 not a multiple of page size 0x10000
09:30 < hrw> [ 0.343802] kvm : error: no compatible GIC info found
09:30 < hrw> [ 0.343909] kvm : error initializing Hyp mode: -6
09:30 < hrw> is what I got on 3.18-rc4 64K from fedora/rawhide
09:32 < hrw> bbiab
10:27 < suihkulokki> hrw: using 3.18-rc5 and mustang I get the same error with 64KB pages - 4KB pages work ok
10:31 < chazy> hrw: suihkulokki: see this commit: https://git.kernel.org/cgit/linux/kernel/git/kvmarm/kvmarm.git/commit/virt/kvm/arm/vgic.c?id=63afbe7a0ac184ef8485dac4914e87b211b5bfaa
10:32 < chazy> iirc, you can fix the DT to report things properly so that this check will pass and everything works (at least on Seattle), but for APM mustang I did think carefully about the proper fix.
10:33 < chazy> if you want to hack on it, and don’t care about stability etc., you can simply remove those two PAGE_ALIGNED checks (now moved to vgic-v2.c) and I believe things will work, but beware, you’re venturing into danger land.
10:37 < suihkulokki> chazy: I think hrw and fedora would be more interested in fixing the dtb
10:39 < chazy> suihkulokki: I realize that, but I think they would have to work with APM on that one. Not sure what the long term plan for juno was on this.
12:41 < agraf> chazy: I have a local patch that relaxes this check slightly
12:42 < agraf> chazy: http://paste.debian.net/132128/
12:42 < agraf> chazy: need to make this a kernel module option that defaults to off
12:42 < agraf> chazy: but you get the idea ;)
12:43 < agraf> chazy: the problem is that on juno qemu needs to align the vgic at the same offset as real hardware has it
12:43 < hrw> chazy: seattle has gicv3 iirc
12:44 < hrw> suihkulokki: at RH we tend to switch for acpi asap. now probably only pci is not fully tested
12:44 < agraf> hrw: nope, it’s v2+msi
12:45 < hrw> ah, right. Thunder is v3
12:45 < agraf> yup, thunder is the first v3 hw you can get fwiw
12:48 < hrw> chazy: I can also go to 3.17 kernel and not have that check ;D
3.18.0-0.rc6.git0.1.fc22.aarch64 is affected too
FYI, we think this will be resolved by a firmware update. We are working with APM to achieve this.
Fresh install of Fedora 21 also lacks nationalization support:
[root@pinkiepie virt]# virsh create hrw-f21.xml
błąd: Utworzenie domeny z hrw-f21.xml nie powiodło się
błąd: unsupported configuration: Domain requires KVM, but it is not available. Check that virtualisation is enabled in the host BIOS, and host configuration is setup to load the kvm modules.
[root@pinkiepie virt]# dmesg|grep -i kvm
[ 0.303959] kvm : GICV size 0x2000 not a multiple of page size 0x10000
[ 0.303965] kvm : error: no compatible GIC info found
[ 0.304070] kvm : error initializing Hyp mode: -6
[root@pinkiepie virt]# uname -a
Linux pinkiepie 3.17.4-301.fc21.aarch64 #1 SMP Sun Nov 30 20:41:43 UTC 2014 aarch64 aarch64 aarch64 GNU/Linux
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.
More information and reason for this action is here:
Created attachment 1009121 [details]
DeviceTreeBlob with fixed gicv entry
Grab fixed DTB, put it in /boot/ directory and add one line into GRUB2 config:
Then kernel will boot and will have KVM support (tested with 4.0-rc5).
Can confirm that KVM is working again with that DTB file on kernel 4.0.0-0.rc6.git0.1.fc23.aarch64
Still broken in Rawhide (without workarounds).
Richard: issue is in firmware not in kernel. As long as we do not have fixed firmware available bug will happen.
So .. Anyone give me a clue exactly how to implement the workaround?
I dropped the mustang.dtb file into /boot and added:
to the end of /boot/grub2/grub.cfg, and rebooted, but I still see
the same error:
$ dmesg | grep -i kvm
[ 0.725483] kvm : GICV size 0x2000 not a multiple of page size 0x10000
[ 0.725576] kvm : error initializing Hyp mode: -6
I also tried appending the devicetree line to
/boot/efi/EFI/fedora/grub.cfg but that also doesn't appear to
make any difference.
The answer is you have to edit /boot/efi/EFI/fedora/grub.cfg,
find the latest kernel section in that file, and add the
devicetree line to that kernel section. You end up with:
linux /vmlinuz-4.0.0-0.rc6.git2.1.fc23.aarch64 root=UUID=9d1bb62f-5d50-458e-be91-36af494ccfa0 ro console=ttyS0,115200 earlyprintk=uart8250-32bit,0x1c020000 LANG=en_GB.UTF-8
This fixed KVM for me. However it also broke stable network
addressing. It seems that the first NIC now gets a randomized
MAC address each time.
Have to check and reorder Ethernet devices.
Created attachment 1012161 [details]
DeviceTreeBlob with fixed gicv entry and 1GbE as eth0
This DTB has 1GbE as eth0 (was eth2 in previous DTB), working USB and KVM:
13:46 hrw@pinkiepie-rawhide:~$ dmesg|grep kvm -i
[ 0.871608] kvm : interrupt-controller@780c0000 IRQ5
[ 0.877167] kvm : timer IRQ3
[ 0.880426] kvm : Hyp mode initialized successfully
13:47 hrw@pinkiepie-rawhide:~$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 125f:de7a A-DATA Technology Co., Ltd.
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 1234:2088 Brain Actuated Technologies
Bus 001 Device 002: ID 05e3:0606 Genesys Logic, Inc. USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
13:47 hrw@pinkiepie-rawhide:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.76 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fdd2:2a70:985d::c0a prefixlen 128 scopeid 0x0<global>
inet6 fdd2:2a70:985d:0:201:73ff:fe02:b73 prefixlen 64 scopeid 0x0<global>
inet6 fe80::201:73ff:fe02:b73 prefixlen 64 scopeid 0x20<link>
ether 00:01:73:02:0b:73 txqueuelen 1000 (Ethernet)
RX packets 223 bytes 25152 (24.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 198 bytes 47959 (46.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
*** Bug 1208768 has been marked as a duplicate of this bug. ***
Version 1.1.0 firmware fixed this issue on Mustang. KVM is properly initialised
at boot time.
I can confirm also that 1.1.0 firmware fixes the problem, and
has a stable MAC address. I suggest we can just close this bug
unless anyone objects?
As bug was not in a kernel but in DTB provided by firmware and latest official firmware (available to all users of affected platform) fixed problem I am closing this bug.
*** Bug 1263499 has been marked as a duplicate of this bug. ***
Changing the page size from 64KB (Fedora's default) to 4KB seems to fix this issue
(firmware version 1.1.0-rh-0.15), hoping new firmware .20 from APM handles the 64KB kernel as well.
Itaru: 4K kernels are something which is not supported.
You should/have to upgrade your firmware.