Bug 1454281 - Enable vm.allocate_pgste sysctl before running qemu-kvm on s390x
Summary: Enable vm.allocate_pgste sysctl before running qemu-kvm on s390x
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.4-Alt
Hardware: s390x
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: David Hildenbrand
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-22 11:47 UTC by Thomas Huth
Modified: 2017-11-09 11:28 UTC (History)
9 users (show)

Fixed In Version: qemu-kvm-2.9.0-22.el7a
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-09 11:28:46 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:3169 normal SHIPPED_LIVE qemu-kvm enhancement update 2017-11-09 16:13:13 UTC
Red Hat Bugzilla 1290589 None CLOSED ship sysctl file enabling vm.allocate_pgste for s390x kvm 2019-07-11 07:40:42 UTC

Internal Links: 1290589

Description Thomas Huth 2017-05-22 11:47:09 UTC
Description of problem:
When you try to run qemu-kvm on s390x with the 4.11 Pegas kernel on the host, the guest can not be started an QEMU complains with:

ioctl(KVM_CREATE_VM) failed: 22 Invalid argument
Host kernel setup problem detected. Please verify:
- for kernels supporting the switch_amode or user_mode parameters, whether
  user space is running in primary address space
- for kernels supporting the vm.allocate_pgste sysctl, whether it is enabled
failed to initialize KVM: Invalid argument

Version-Release number of selected component (if applicable):
kernel-4.11.0-3.el7.s390x
qemu-kvm-rhev-2.9.0-4.el7.s390x

How reproducible:
100%

Steps to Reproduce:
1. Install the 4.11 pegas kernel and qemu-kvm on the host
2. Try to start a guest like this:
   /usr/libexec/qemu-kvm -nographic -enable-kvm -hda guest.qcow2

Actual results:
Guest can not be started

Expected results:
Guest can be started successfully

Additional info:
After manually running
 sysctl vm.allocate_pgste=1
the error message is gone and the guest can be started automatically. So we likely need to create a setting in /etc/sysctl.d with this setting when the qemu-kvm RPM package gets installed (or maybe figure out another keener way to deal with this problem...).

Comment 2 Thomas Huth 2017-05-29 07:55:28 UTC
FWIW, the QEMU package in Fedora apparently ships with a sysctl file already, see BZ 1290589.

Comment 3 David Hildenbrand 2017-06-12 12:52:44 UTC
We are currently discussing an upstream solution that will avoid having to set this sysctl. It will most likely consist of

a) A gcc patch that allows to set specific ELF file header
b) A QEMU patch that will set this ELF file header using gcc on the qemu-system-s390x binary
c) A kernel patch that will look out for this ELF file header and setup pgste accordingly.

Folks at IBM are currently working on this.

Comment 4 David Hildenbrand 2017-08-21 09:27:09 UTC
We now have the kernel patch upstream:

commit 23fefe119ceb5fb0c7d3321010620010a4eddb18
Author: Martin Schwidefsky <schwidefsky@de.ibm.com>
Date:   Wed Jun 7 14:10:24 2017 +0200

    s390/kvm: avoid global config of vm.alloc_pgste=1

While the QEMU part will be easy, we are still missing a GCC patch.

As discussed during the zKVM meeting, it might make sense to temporarily set vm.allocate_pgste=1 like fedora does, to make working with zKVM easier during development. This can then be reverted once we have proper gcc + qemu support in place.

Comment 6 David Hildenbrand 2017-08-23 08:02:19 UTC
We have a new ld linker option upstream (binutils):
https://sourceware.org/binutils/docs/ld/S_002f390-ELF.html

commit b4cbbe8f7294070cc93a71ace78f134965ddad82
Author: Andreas Krebbel <krebbel@linux.vnet.ibm.com>
Date:   Thu Jun 8 17:24:50 2017 +0200

    S/390: Add support for pgste marker


In summary, to get rid of this temporary solution, we need:
- binutils >= 2.29 (or relevant backport)
- kernels >= 4.12 (or relevant backport)

QEMU patch is currently being discussed upstream:
https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04160.html

Comment 7 Miroslav Rezanina 2017-08-29 08:22:31 UTC
Fix included in qemu-kvm-2.9.0-22.el7_4

Comment 9 Qunfang Zhang 2017-09-20 07:42:54 UTC
Reproduced this bug with qemu-kvm-2.9.0-21.el7a.s390x:

Host kernel: kernel-4.11.0-30.el7a.s390x

# /usr/libexec/qemu-kvm 
ioctl(KVM_CREATE_VM) failed: 22 Invalid argument
Host kernel setup problem detected. Please verify:
- for kernels supporting the switch_amode or user_mode parameters, whether
  user space is running in primary address space
- for kernels supporting the vm.allocate_pgste sysctl, whether it is enabled
failed to initialize KVM: Invalid argument
Back to tcg accelerator.
VNC server running on ::1:5900
#

Result: Qemu-kvm command line could not boot up.

Verified this issue with qemu-kvm-2.9.0-22.el7a.s390x:

Host kernel: kernel-4.11.0-30.el7a.s390x

# /usr/libexec/qemu-kvm -no-shutdown
VNC server running on ::1:5900

Result: Qemu-kvm command line will not quit.

Besides, I installed a guest and it could finish successfully.

/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -sandbox off -machine s390-ccw-virtio -nodefaults -vga none -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_0No0v2/monitor-qmpmonitor1-20170920-033800-mlIN5A31,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_0No0v2/monitor-catch_monitor-20170920-033800-mlIN5A31,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_0No0v2/serial-serial0-20170920-033800-mlIN5A31,server,nowait -device sclpconsole,chardev=serial_id_serial0 -device virtio-scsi-ccw,id=virtio_scsi_ccw0 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=unsafe,format=qcow2,file=/root/staf-kvm-devel/vt_test_images/ALT-Server-7.4-s390x-virtio-scsi.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-ccw,mac=9a:47:48:49:4a:4b,id=idNUNb5u,netdev=idAgqOHS -netdev tap,id=idAgqOHS,vhost=on,vhostfd=23,fd=22 -m 802 -smp 2,cores=1,threads=1,sockets=2 -cpu host -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=unsafe,media=cdrom,file=/root/staf-kvm-devel/workspace/var/lib/avocado/data/avocado-vt/isos/linux/RHEL-ALT-7.4-20170913.1-Server-s390x-dvd1.iso -device scsi-cd,id=cd1,drive=drive_cd1 -drive id=drive_unattended,if=none,snapshot=off,aio=threads,cache=unsafe,media=cdrom,file=/root/staf-kvm-devel/vt_test_images/rhel-alt-74-s390x/ks.iso -device scsi-cd,id=unattended,drive=drive_unattended -kernel /root/staf-kvm-devel/vt_test_images/rhel-alt-74-s390x/kernel.img -append ksdevice=link inst.repo=cdrom:/dev/sr0 inst.ks=cdrom:/dev/sr1:/ks.cfg nicdelay=60 biosdevname=0 net.ifnames=0 console=ttysclp0 -initrd /root/staf-kvm-devel/vt_test_images/rhel-alt-74-s390x/initrd.img -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot strict=on -no-shutdown -enable-kvm


Hi, David

Since I just test this bug on a beaker system which should be a zVM, and I install kvm guest on that environment.  Could I call it VERIFIED pass now?

Thanks,
Qunfang

Comment 10 David Hildenbrand 2017-09-20 12:52:37 UTC
Hi Qunfang,

yes, that should be enough. Testing under z/VM is sufficient.

As soon as you can run

$ /usr/libexec/qemu-kvm --enable-kvm

Without getting said error, the fix works. (If it works, you will get "VNC server running on ::1:5900")

And that is the case when you can run a KVM guest!

Thanks for verifying!

David

Comment 11 Qunfang Zhang 2017-09-21 02:26:49 UTC
Okay, thanks for confirmation. :)

Comment 13 errata-xmlrpc 2017-11-09 11:28:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3169


Note You need to log in before you can comment on or make changes to this bug.