RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2116496 - Can't run when memory backing with hugepages and backend type memfd
Summary: Can't run when memory backing with hugepages and backend type memfd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.1
Hardware: s390x
OS: Linux
low
low
Target Milestone: rc
: 9.2
Assignee: Thomas Huth
QA Contact: smitterl
Jiri Herrmann
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-08 15:59 UTC by smitterl
Modified: 2023-05-09 07:43 UTC (History)
5 users (show)

Fixed In Version: qemu-kvm-7.2.0-1.el9
Doc Type: Bug Fix
Doc Text:
.VMs on IBM Z no longer fail to start when using `memfd` memory backing Previously, on IBM Z hosts, virtual machines (VMs) failed to boot if they were configured to use the `memfd` type of hugepage memory backing, for example as follows: ---- <memoryBacking> <hugepages/> <source type='memfd'/> </memoryBacking> ---- With this update, the underlying cause has been fixed, and the affected VMs now start correctly.
Clone Of:
: 2117149 (view as bug list)
Environment:
Last Closed: 2023-05-09 07:20:04 UTC
Type: Bug
Target Upstream Version: 7.2
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2117149 0 low CLOSED Can't run when memory backing with hugepages and backend type memfd 2023-09-10 07:56:36 UTC
Red Hat Issue Tracker RHELPLAN-130523 0 None None None 2022-08-08 16:05:00 UTC
Red Hat Product Errata RHSA-2023:2162 0 None None None 2023-05-09 07:20:28 UTC

Internal Links: 2117149

Description smitterl 2022-08-08 15:59:45 UTC 
Description of problem:
A machine with memory backing via hugepages using the memfd backend doesn't run.

Version-Release number of selected component (if applicable):
qemu-kvm-7.0.0-9.el9.s390x

How reproducible:
100%


Steps to Reproduce:
0. Make sure hugepages are enabled for KVM (modprobe kvm hpage=1)
1. Start a machine with -object {"qom-type":"memory-backend-memfd","id":"s390.ram","hugetlb":true,"hugetlbsize":1048576,"x-use-canonical-path-for-ramblock-id":false,"prealloc":true,"size":1073741824}
OR equivalently, in libvirt define
<memoryBacking>
  <hugepages/>
  <source type='memfd'/>
</memoryBacking>

Actual results:
The machine is immediately paused; there's an error: kvm run failed Bad address
PSW=mask 0000000180000000 addr 000000003fe00510
R00=0000000000000000 R01=0000000000000000 R02=0000000000000000 R03=0000000000000000
R04=0000000000000000 R05=0000000000000000 R06=0000000000000000 R07=0000000000000000
R08=0000000000000000 R09=0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
C00=00000000000000e0 C01=0000000000000000 C02=0000000000000000 C03=0000000000000000
C04=0000000000000000 C05=0000000000000000 C06=0000000000000000 C07=0000000000000000
C08=0000000000000000 C09=0000000000000000 C10=0000000000000000 C11=0000000000000000
C12=0000000000000000 C13=0000000000000000 C14=00000000c2000000 C15=0000000000000000


Expected results:
The machine runs.

Additional info:
a) The machine runs successfully on x86_64 (used qemu-kvm-6.2.0-11.el9_0.3.x86_64).
b) It reproduces with qemu-kvm-6.2.0-11.el9_0.2.s390x.
c) Not considering critical at this point as we can still use the memory-backend-file instead of memory-backend-memfd successfully.
d) Using memory-backend-memfd without hugepages works, too.
Comment 1 Thomas Huth 2022-08-09 06:29:11 UTC 
Just to be sure: Did you load your kvm kernel module with "hpage=1" ?
Comment 2 smitterl 2022-08-09 07:30:34 UTC 
(In reply to Thomas Huth from comment #1)
> Just to be sure: Did you load your kvm kernel module with "hpage=1" ?

Yes :)
Comment 3 Thomas Huth 2022-08-09 09:56:02 UTC 
FWIW, I can reproduce the issue. With memory-backend-file (which is working), the QEMU command line looks like this:

/usr/libexec/qemu-kvm \
-name guest=rhel9,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-4-rhel9/master-key.aes"}' \
-machine s390-ccw-virtio-rhel9.0.0,usb=off,dump-guest-core=off,memory-backend=s390.ram \
-accel kvm \
-cpu gen15a-base,aen=on,cmmnt=on,vxpdeh=on,aefsi=on,diag318=on,csske=on,mepoch=on,msa9=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,deflate=on,edat2=on,etoken=on,vx=on,ipter=on,mepochptff=on,ap=on,vxeh=on,vxpd=on,esop=on,msa9_pckmo=on,vxeh2=on,esort=on,apqi=on,apft=on,els=on,iep=on,apqci=on,cte=on,ais=on,bpb=on,gs=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on \
-m 4096 \
-object '{"qom-type":"memory-backend-file","id":"s390.ram","mem-path":"/dev/hugepages/libvirt/qemu/4-rhel9","x-use-canonical-path-for-ramblock-id":false,"prealloc":true,"size":4294967296}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 6ad23529-478b-44d6-9140-3d5bda239a53 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device '{"driver":"virtio-serial-ccw","id":"virtio-serial0","devno":"fe.0.0002"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/rhel9.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \
-netdev tap,fd=33,vhost=on,vhostfd=35,id=hostnet0 \
-device '{"driver":"virtio-net-ccw","netdev":"hostnet0","id":"net0","mac":"52:54:00:c9:ff:7d","devno":"fe.0.0001"}' \
-chardev socket,id=charchannel0,fd=31,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev pty,id=charconsole0 \
-device '{"driver":"sclpconsole","chardev":"charconsole0","id":"console0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0003"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-ccw","rng":"objrng0","id":"rng0","devno":"fe.0.0004"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

With memory-backend-memfd (which is not working), the command line looks like this:

/usr/libexec/qemu-kvm \
-name guest=rhel9,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-5-rhel9/master-key.aes"}' \
-machine s390-ccw-virtio-rhel9.0.0,usb=off,dump-guest-core=off,memory-backend=s390.ram \
-accel kvm \
-cpu gen15a-base,aen=on,cmmnt=on,vxpdeh=on,aefsi=on,diag318=on,csske=on,mepoch=on,msa9=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,deflate=on,edat2=on,etoken=on,vx=on,ipter=on,mepochptff=on,ap=on,vxeh=on,vxpd=on,esop=on,msa9_pckmo=on,vxeh2=on,esort=on,apqi=on,apft=on,els=on,iep=on,apqci=on,cte=on,ais=on,bpb=on,gs=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on \
-m 4096 \
-object '{"qom-type":"memory-backend-memfd","id":"s390.ram","hugetlb":true,"hugetlbsize":1048576,"x-use-canonical-path-for-ramblock-id":false,"prealloc":true,"size":4294967296}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 6ad23529-478b-44d6-9140-3d5bda239a53 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device '{"driver":"virtio-serial-ccw","id":"virtio-serial0","devno":"fe.0.0002"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/rhel9.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \
-netdev tap,fd=33,vhost=on,vhostfd=35,id=hostnet0 \
-device '{"driver":"virtio-net-ccw","netdev":"hostnet0","id":"net0","mac":"52:54:00:c9:ff:7d","devno":"fe.0.0001"}' \
-chardev socket,id=charchannel0,fd=31,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev pty,id=charconsole0 \
-device '{"driver":"sclpconsole","chardev":"charconsole0","id":"console0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0003"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-ccw","rng":"objrng0","id":"rng0","devno":"fe.0.0004"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

The only difference is basicall the line with "-object '{"qom-type":"memory-backend-...'.
Comment 4 Thomas Huth 2022-08-09 10:14:02 UTC 
For reproducing without libvirt, these QEMU command lines can be used, too:

- memory-backend-file (working):

  qemu-system-s390x --accel kvm -m 4G -nographic -hda rhel8.qcow2 \
   -M s390-ccw-virtio,memory-backend=s390.ram \
   -object '{"qom-type":"memory-backend-file","id":"s390.ram","mem-path":"/dev/hugepages/","x-use-canonical-path-for-ramblock-id":false,"prealloc":true,"size":4294967296}'

- memory-backend-memfd (non-working):

  qemu-system-s390x --accel kvm -m 4G -nographic -hda rhel8.qcow2 \
   -M s390-ccw-virtio,memory-backend=s390.ram \
  -object '{"qom-type":"memory-backend-memfd","id":"s390.ram","hugetlb":true,"hugetlbsize":1048576,"x-use-canonical-path-for-ramblock-id":false,"prealloc":true,"size":4294967296}'
Comment 5 Thomas Huth 2022-08-09 10:21:59 UTC 
Seems like s390_set_max_pagesize() in the QEMU sources does not activate the huge pages since qemu_maxrampagesize() only returns 4096 on s390x ... and looking a little bit closer, this happens because host_memory_backend_pagesize() is only able to deal with objects of type memory-backend-file...
Comment 6 Thomas Huth 2022-08-10 06:47:41 UTC 
I suggested a patch upstream here: https://lore.kernel.org/qemu-devel/20220810063204.3589543-1-thuth@redhat.com/
Comment 7 Thomas Huth 2022-09-22 08:47:36 UTC 
We'll get this fixed with the rebase to QEMU 7.2.
Comment 9 smitterl 2022-12-19 10:26:34 UTC 
Pre-verified with:
qemu-kvm-7.2.0-1.el9.s390x

Automated test updated to pass on s390x:
https://github.com/autotest/tp-libvirt/pull/4683
Comment 17 errata-xmlrpc 2023-05-09 07:20:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2162
Note You need to log in before you can comment on or make changes to this bug.