Bug 2152083
| Summary: | libvirt kills virtual machine on restart when 2M and 1G hugepages are mounted [rhel-9.1.0.z] | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | RHEL Program Management Team <pgm-rhel-tools> |
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> |
| libvirt sub component: | General | QA Contact: | liang cong <lcong> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | ailan, duclee, dzheng, gveitmic, haizhao, jdenemar, jsuchane, lcong, lmen, mprivozn, virt-maint, yafu, yalzhang, ymankad |
| Version: | 9.2 | Keywords: | Triaged, Upstream, ZStream |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-8.5.0-7.2.el9_1 | Doc Type: | Bug Fix |
| Doc Text: |
Cause:
When libvirt is restarted after a hugetlbfs was mounted and a guest is running, libvirt tries to create guest specific path in the new hugetlbfs mount point. Because of a bug in namespace code this fails which results in the guest being killed by libvirt.
Consequence:
Guest is killed on libvirtd restart.
Fix:
Twofold. Firstly, the namespace code was fixed so that creating this guest specific path now succeeds. Secondly, the creation is postponed until really needed (memory hotplug).
Result:
Guests can now survive libvirtd restart.
|
Story Points: | --- |
| Clone Of: | 2151869 | Environment: | |
| Last Closed: | 2023-01-23 15:18:08 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2151869 | ||
| Bug Blocks: | |||
Verified on build:
# rpm -q libvirt qemu-kvm
libvirt-8.5.0-7.2.el9_1.x86_64
qemu-kvm-7.0.0-13.el9.x86_64
Verify steps:
1. Prepare huge page memory:
# echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# echo 3072 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
2. Define a guest with below memorybacking xml.
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB'/>
</hugepages>
</memoryBacking>
3. Start the VM and stop virtqemud
# virsh start vm1 && systemctl stop virtqemud
Domain 'vm1' started
Warning: Stopping virtqemud.service, but it can still be activated by:
virtqemud-ro.socket
virtqemud-admin.socket
virtqemud.socket
4. Mount 1G hugepage path
# mkdir /dev/hugepages1G
# mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G
5. Do virsh list and guest still in running state.
# virsh -r list --all
Id Name State
----------------------
1 vm1 running
# virsh -r list --all
Id Name State
----------------------
1 vm1 running
6. Prepare memory device hotplug xml like below:
# cat dimm1G.xml
<memory model='dimm'>
<source>
<pagesize unit='KiB'>1048576</pagesize>
<nodemask>0-1</nodemask>
</source>
<target>
<size unit='KiB'>1048576</size>
<node>0</node>
</target>
</memory>
7. Hotplug dimm memory device:
# virsh attach-device vm1 dimm1G.xml
Device attached successfully
8. Prepare memory device with 2M hugepage source hotplug xml like below:
# cat dimm2M.xml
<memory model='dimm'>
<source>
<pagesize unit='KiB'>2048</pagesize>
<nodemask>0-1</nodemask>
</source>
<target>
<size unit='KiB'>1048576</size>
<node>0</node>
</target>
</memory>
9. Hotplug dimm memory device:
# virsh attach-device vm1 dimm2M.xml
Device attached successfully
10. Shutoff vm
# virsh destroy vm1
Domain vm1 destroyed
11. Restart virtqemud
# systemctl restart virtqemud
12. Start vm
# virsh start vm1
Domain 'vm1' started
Also check the below scenarios:
Steps:
1. memory backing 2M guest vm start -> stop virtqemud -> mount 1G path -> start virtqemud -> hotplug 1G dimm -> restart vm -> restart virtqemud -> hotplug 1G dimm
2. mount 1G path -> memory backing 2M guest vm start -> restart virtqemud -> hogplug 1G dimm -> restart virtqemud -> restart vm ->hogplug 1G dimm
Tested with these settings：remember_owner=1 or 0, memfd memory backing, default memory backing, 1G hugepage memory backing, 1G hugepage path as /mnt/hugepages1G
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:0311 |
Preverified on scratch build: # rpm -q libvirt libvirt-8.5.0-7.2.el9_1_rc.c5ca229de8.x86_64 Verify steps: 1. Prepare huge page memory: # echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages # echo 2048 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 2. Define a guest with below memorybacking xml. <memoryBacking> <hugepages> <page size='2048' unit='KiB'/> </hugepages> </memoryBacking> 3. Start the VM and stop virtqemud # virsh start vm1 && systemctl stop virtqemud Domain 'vm1' started Warning: Stopping virtqemud.service, but it can still be activated by: virtqemud-ro.socket virtqemud-admin.socket virtqemud.socket 4. Mount 1G hugepage path # mkdir /dev/hugepages1G # mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages1G 5. Do virsh list and guest still in running state. # virsh -r list --all Id Name State ---------------------- 1 vm1 running # virsh -r list --all Id Name State ---------------------- 1 vm1 running 6. Prepare memory device hotplug xml like below: # cat dimm1G.xml <memory model='dimm'> <source> <pagesize unit='KiB'>1048576</pagesize> <nodemask>0-1</nodemask> </source> <target> <size unit='KiB'>1048576</size> <node>0</node> </target> </memory> 7. Hotplug dimm memory device: # virsh attach-device vm1 dimm1G.xml Device attached successfully 8. Prepare memory device with 2M hugepage source hotplug xml like below: # cat dimm2M.xml <memory model='dimm'> <source> <pagesize unit='KiB'>2048</pagesize> <nodemask>0-1</nodemask> </source> <target> <size unit='KiB'>1048576</size> <node>0</node> </target> </memory> 9. Hotplug dimm memory device: # virsh attach-device vm1 dimm2M.xml Device attached successfully 10. Shutoff vm # virsh destroy vm1 Domain vm1 destroyed 11. Restart virtqemud # systemctl restart virtqemud 12. Start vm # virsh start vm1 Domain 'vm1' started Also check the below scenarios: Steps: 1. memory backing 2M guest vm start -> stop virtqemud -> mount 1G path -> start virtqemud -> hotplug 1G dimm -> restart vm -> restart virtqemud -> hotplug 1G dimm 2. mount 1G path -> memory backing 2M guest vm start -> restart virtqemud -> hogplug 1G dimm -> restart virtqemud -> restart vm ->hogplug 1G dimm Tested with these settings：remember_owner=1 or 0, memfd memory backing, default memory backing, 1G hugepage memory backing, 1G hugepage path as /mnt/hugepages1G