2132176 – libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted [rhel-8.7.0.z]

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2132176 - libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted [rhel-8.7.0.z]

Summary: libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	8.4
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Michal Privoznik
QA Contact:	liang cong
Docs Contact:
URL:
Whiteboard:
Depends On:	2123196 2151869
Blocks:
TreeView+	depends on / blocked

Reported:	2022-10-04 21:18 UTC by RHEL Program Management Team
Modified:	2023-01-12 09:19 UTC (History)
CC List:	10 users (show)
Fixed In Version:	libvirt-8.0.0-10.1.module+el8.7.0+17192+cbc2449b
Doc Type:	Bug Fix
Doc Text:	Cause: When libvirt is restarted after a hugetlbfs was mounted and a guest is running, libvirt tries to create guest specific path in the new hugetlbfs mount point. Because of a bug in namespace code this fails which results in the guest being killed by libvirt. Consequence: Guest is killed on libvirtd restart. Fix: Twofold. Firstly, the namespace code was fixed so that creating this guest specific path now succeeds. Secondly, the creation is postponed until really needed (memory hotplug). Result: Guests can now survive libvirtd restart.
Clone Of:	2123196
Environment:
Last Closed:	2023-01-12 09:18:55 UTC
Type:	---
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	LIBVIRTAT-13734	None	None	None	2022-12-01 07:30:20 UTC
Red Hat Issue Tracker	RHELPLAN-135639	None	None	None	2022-10-05 10:54:00 UTC
Red Hat Product Errata	RHSA-2023:0099	None	None	None	2023-01-12 09:19:07 UTC

Comment 2 liang cong 2022-10-08 10:00:09 UTC

Find an issue on:
# rpm -q libvirt qemu-kvm
libvirt-8.0.0-11.el8_rc.8c593088fd.x86_64
qemu-kvm-6.2.0-20.module+el8.7.0+16689+53d59bc2.1.x86_64


1. Define a guest with below memorybacking xml.
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>

2. Mount 1G hugepage path
mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G

3. Start vm
# virsh start vm1
Domain vm1 started

4. Prepare memory device hotplug xml like below:
# cat dimm1G.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>1048576</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>

5. Attach 1G memory device described in step 4.
# virsh attach-device vm1 dimm1G.xml 
error: Failed to attach device from dimm1G.xml
error: internal error: unable to execute QEMU command 'object-add': can't open backing store /dev/hugepages1G/libvirt/qemu/3-vm1 for guest RAM: Permission denied


@mprivozn could you help to check this issue?

Comment 3 liang cong 2022-10-09 09:30:06 UTC

Additional info for comment#2 :
more info about reproduce steps could be seen at bug#2123196comment#34

Comment 4 Michal Privoznik 2022-10-13 10:42:27 UTC

See bug 2123196#c38 for more info. Just for the record here, this is a slightly different issue and it's tracked as bug 2134009.

Comment 5 liang cong 2022-10-21 02:38:33 UTC

Pre-verified on scratch build:
# rpm -q libvirt qemu-kvm
libvirt-8.0.0-11.el8_rc.8c593088fd.x86_64
qemu-kvm-6.2.0-20.module+el8.7.0+16689+53d59bc2.1.x86_64

Verify steps:
1. Define a guest with below memorybacking xml.
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>

2. Start the VM and stop libvirt

# virsh start vm1 && systemctl stop libvirtd
Domain vm1 started

Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
  libvirtd-ro.socket
  libvirtd-admin.socket

3. Mount 1G hugepage path
# mkdir /dev/hugepages1G
# mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G


4. Do virsh list and guest still in running state.

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

5. Prepare memory device hotplug xml like below:
# cat dimm1G.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>1048576</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>


6. Hotplug dimm memory device:
# virsh attach-device vm1 dimm1G.xml 
Device attached successfully

7. Prepare memory device with 2M hugepage source hotplug xml like below:
# cat dimm2M.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>2048</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>

8. Hotplug dimm memory device:
# virsh attach-device vm1 dimm2M.xml 
Device attached successfully


9. Shutoff vm
# virsh destroy vm1
Domain vm1 destroyed


10. Restart libvirtd
# systemctl restart libvirtd

11. Start vm
# virsh start vm1
Domain 'vm1' started


Also check the below scenarios:
Steps:
1. memory backing 2M guest vm start -> stop libvirt -> mount 1G path -> start libvirt -> hotplug 1G dimm -> restart vm -> restart libvirtd -> hotplug 1G dimm
2. mount 1G path -> memory backing 2M guest vm start -> restart libvirtd -> hogplug 1G dimm -> restart libvirtd -> restart vm ->hogplug 1G dimm

Tested with these settings：remember_owner=1 or 0, memfd memory backing, default memory backing, 1G hugepage memory backing, 1G hugepage path as /mnt/hugepages1G

Comment 10 liang cong 2022-11-16 07:15:29 UTC

Verified on build:
# rpm -q libvirt qemu-kvm
libvirt-8.0.0-10.1.module+el8.7.0+17192+cbc2449b.x86_64
qemu-kvm-6.2.0-20.module+el8.7.0+16905+efca5d32.2.x86_64

Verify steps:
1. Define a guest with below memorybacking xml.
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>

2. Start the VM and stop libvirt

# virsh start vm1 && systemctl stop libvirtd
Domain vm1 started

Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
  libvirtd-ro.socket
  libvirtd-admin.socket

3. Mount 1G hugepage path
# mkdir /dev/hugepages1G
# mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G


4. Do virsh list and guest still in running state.

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

5. Prepare memory device hotplug xml like below:
# cat dimm1G.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>1048576</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>


6. Hotplug dimm memory device:
# virsh attach-device vm1 dimm1G.xml 
Device attached successfully

7. Prepare memory device with 2M hugepage source hotplug xml like below:
# cat dimm2M.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>2048</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>

8. Hotplug dimm memory device:
# virsh attach-device vm1 dimm2M.xml 
Device attached successfully


9. Shutoff vm
# virsh destroy vm1
Domain vm1 destroyed


10. Restart libvirtd
# systemctl restart libvirtd

11. Start vm
# virsh start vm1
Domain 'vm1' started


Also check the below scenarios:
Steps:
1. memory backing 2M guest vm start -> stop libvirt -> mount 1G path -> start libvirt -> hotplug 1G dimm -> restart vm -> restart libvirtd -> hotplug 1G dimm
2. mount 1G path -> memory backing 2M guest vm start -> restart libvirtd -> hogplug 1G dimm -> restart libvirtd -> restart vm ->hogplug 1G dimm

Tested with these settings：remember_owner=1 or 0, memfd memory backing, default memory backing, 1G hugepage memory backing, 1G hugepage path as /mnt/hugepages1G

Comment 15 errata-xmlrpc 2023-01-12 09:18:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0099

Note You need to log in before you can comment on or make changes to this bug.