2152083 – libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted [rhel-9.1.0.z]

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2152083 - libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted [rhel-9.1.0.z]

Summary: libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	9.2
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Michal Privoznik
QA Contact:	liang cong
Docs Contact:
URL:
Whiteboard:
Depends On:	2151869
Blocks:
TreeView+	depends on / blocked

Reported:	2022-12-09 07:04 UTC by RHEL Program Management Team
Modified:	2023-01-23 15:21 UTC (History)
CC List:	14 users (show)
Fixed In Version:	libvirt-8.5.0-7.2.el9_1
Doc Type:	Bug Fix
Doc Text:	Cause: When libvirt is restarted after a hugetlbfs was mounted and a guest is running, libvirt tries to create guest specific path in the new hugetlbfs mount point. Because of a bug in namespace code this fails which results in the guest being killed by libvirt. Consequence: Guest is killed on libvirtd restart. Fix: Twofold. Firstly, the namespace code was fixed so that creating this guest specific path now succeeds. Secondly, the creation is postponed until really needed (memory hotplug). Result: Guests can now survive libvirtd restart.
Clone Of:	2151869
Environment:
Last Closed:	2023-01-23 15:18:08 UTC
Type:	---
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-141751	0	None	None	None	2022-12-09 07:24:15 UTC
Red Hat Product Errata	RHBA-2023:0311	0	None	None	None	2023-01-23 15:18:17 UTC

Comment 2 liang cong 2022-12-15 09:28:28 UTC

Preverified on scratch build:
# rpm -q libvirt
libvirt-8.5.0-7.2.el9_1_rc.c5ca229de8.x86_64


Verify steps:
1. Prepare huge page memory:
# echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# echo 2048 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

2. Define a guest with below memorybacking xml.
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>

3. Start the VM and stop virtqemud
# virsh start vm1 && systemctl stop virtqemud
Domain 'vm1' started

Warning: Stopping virtqemud.service, but it can still be activated by:
  virtqemud-ro.socket
  virtqemud-admin.socket
  virtqemud.socket


4. Mount 1G hugepage path
# mkdir /dev/hugepages1G
# mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G


5. Do virsh list and guest still in running state.

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

6. Prepare memory device hotplug xml like below:
# cat dimm1G.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>1048576</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>


7. Hotplug dimm memory device:
# virsh attach-device vm1 dimm1G.xml 
Device attached successfully

8. Prepare memory device with 2M hugepage source hotplug xml like below:
# cat dimm2M.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>2048</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>

9. Hotplug dimm memory device:
# virsh attach-device vm1 dimm2M.xml 
Device attached successfully


10. Shutoff vm
# virsh destroy vm1
Domain vm1 destroyed


11. Restart virtqemud
# systemctl restart virtqemud

12. Start vm
# virsh start vm1
Domain 'vm1' started


Also check the below scenarios:
Steps:
1. memory backing 2M guest vm start -> stop virtqemud -> mount 1G path -> start virtqemud -> hotplug 1G dimm -> restart vm -> restart virtqemud -> hotplug 1G dimm
2. mount 1G path -> memory backing 2M guest vm start -> restart virtqemud -> hogplug 1G dimm -> restart virtqemud -> restart vm ->hogplug 1G dimm

Tested with these settings：remember_owner=1 or 0, memfd memory backing, default memory backing, 1G hugepage memory backing, 1G hugepage path as /mnt/hugepages1G

Comment 3 liang cong 2022-12-27 12:05:14 UTC

Verified on build:
# rpm -q libvirt qemu-kvm
libvirt-8.5.0-7.2.el9_1.x86_64
qemu-kvm-7.0.0-13.el9.x86_64


Verify steps:
1. Prepare huge page memory:
# echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# echo 3072 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

2. Define a guest with below memorybacking xml.
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>

3. Start the VM and stop virtqemud
# virsh start vm1 && systemctl stop virtqemud
Domain 'vm1' started

Warning: Stopping virtqemud.service, but it can still be activated by:
  virtqemud-ro.socket
  virtqemud-admin.socket
  virtqemud.socket


4. Mount 1G hugepage path
# mkdir /dev/hugepages1G
# mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G


5. Do virsh list and guest still in running state.

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

# virsh -r list --all
 Id   Name   State
----------------------
 1    vm1    running

6. Prepare memory device hotplug xml like below:
# cat dimm1G.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>1048576</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>


7. Hotplug dimm memory device:
# virsh attach-device vm1 dimm1G.xml 
Device attached successfully

8. Prepare memory device with 2M hugepage source hotplug xml like below:
# cat dimm2M.xml 
<memory model='dimm'>
    <source>
      <pagesize unit='KiB'>2048</pagesize>
      <nodemask>0-1</nodemask>
    </source>
    <target>
      <size unit='KiB'>1048576</size>
      <node>0</node>
    </target>
  </memory>

9. Hotplug dimm memory device:
# virsh attach-device vm1 dimm2M.xml 
Device attached successfully


10. Shutoff vm
# virsh destroy vm1
Domain vm1 destroyed


11. Restart virtqemud
# systemctl restart virtqemud

12. Start vm
# virsh start vm1
Domain 'vm1' started


Also check the below scenarios:
Steps:
1. memory backing 2M guest vm start -> stop virtqemud -> mount 1G path -> start virtqemud -> hotplug 1G dimm -> restart vm -> restart virtqemud -> hotplug 1G dimm
2. mount 1G path -> memory backing 2M guest vm start -> restart virtqemud -> hogplug 1G dimm -> restart virtqemud -> restart vm ->hogplug 1G dimm

Tested with these settings：remember_owner=1 or 0, memfd memory backing, default memory backing, 1G hugepage memory backing, 1G hugepage path as /mnt/hugepages1G

Comment 6 liang cong 2023-01-05 05:12:01 UTC

Mark it verified as comment#3

Comment 10 errata-xmlrpc 2023-01-23 15:18:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0311

Note You need to log in before you can comment on or make changes to this bug.