Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Due to an unhandled error, the qemu-kvm process exited when trying to hot-plug memory devices on guests using 1GB hugepages. With this update, the error is handled within qemu-kvm, and as a result, the process does not crash.
Reproduce:
qemu-kvm-rhev-2.3.0-31.el7
kernel-3.10.0-373.el7.x86_64
Steps:
1. prepare hugepages env:
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-373.el7.x86_64 root=/dev/mapper/rhel_dell--per715--04-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per715-04/root rd.lvm.lv=rhel_dell-per715-04/swap rhgb quiet LANG=en_US.UTF-8 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0
# mount
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)
# cat /proc/meminfo | grep Huge
AnonHugePages: 8192 kB
HugePages_Total: 4
HugePages_Free: 4
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
2. start a guest with 1G hugepage:
# /usr/libexec/qemu-kvm -m 1G,slots=4,maxmem=32G -smp 4 \
-mem-prealloc -mem-path /dev/hugepages1G -numa node \
-drive file=/home/guest/RHEL-Server-7.3-64-virtio.qcow2,id=drive-virtio-disk1,media=disk,cache=none,snapshot=off,format=qcow2,aio=native,if=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,bootindex=0\
-netdev tap,id=idw97X5F -device virtio-net-pci,netdev=idw97X5F,mac='9a:37:37:37:37:8e',bus=pci.0,addr=0x5,id='idMqFTXq' -vga cirrus -monitor stdio -vnc :0
3. hotplug a memory device with size=512M
(qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G
memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
And the qemu process exit.
So the bug is reproduced.
Verify:
qemu-kvm-rhev-2.6.0-1.el7
kernel-3.10.0-373.el7.x86_64
with same steps as above, after step3, HMP prints "memory size 0x20000000 must be equal to or larger than page size 0x40000000" , and guest works well.
So the bug is fixed.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2016-2673.html
Description of problem: qemu process exit after hot-plug a memory device which guest use 1G hugepages Version-Release number of selected component (if applicable): libvirt-1.2.17-13.el7.x86_64 qemu-kvm-rhev-2.3.0-29.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. prepare hugepages env: # cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10.0-316.el7.x86_64 root=UUID=15d52150-0581-4d91-bc9b-6909fb677212 ro rd.md=0 rd.dm=0 console=tty0 rd.luks=0 processor.max_cstate=1 biosdevname=0 crashkernel=256M rd.lvm=0 reboot=pci console=ttyS0,115200 vconsole.keymap=us vconsole.font=latarcyrheb-sun16 nmi_watchdog=0 rhgb quiet intel_iommu=on isolcpus=14,15 rcu_nocbs=14,15 nohz_full=14,15 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0 intel_pstate=disable # mount ... hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G) hugetlbfs on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M) # /etc/libvirt/qemu.conf hugetlbfs_mount = ["/dev/hugepages1G", "/dev/hugepages2M"] # virsh freepages --all Node 0: 4KiB: 9176414 2048KiB: 3600 1048576KiB: 15 Node 1: 4KiB: 9810379 2048KiB: 3600 1048576KiB: 15 2. start a guest which use 1G hugepages # virsh dumpxml RHEL71-lhuang <domain type='kvm'> <name>RHEL71-lhuang</name> <uuid>e7041d2f-4811-4fae-93f8-20e029817e63</uuid> <maxMemory slots='16' unit='KiB'>555000000</maxMemory> <memory unit='KiB'>14680064</memory> <currentMemory unit='KiB'>14680064</currentMemory> <memtune> <hard_limit unit='KiB'>111111111111111</hard_limit> </memtune> <memoryBacking> <hugepages/> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static' current='4'>16</vcpu> <iothreads>4</iothreads> <numatune> <memory mode='strict' nodeset='0-1'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <pmu state='off'/> </features> <cpu mode='host-passthrough'> <feature policy='require' name='tsc-deadline'/> <numa> <cell id='0' cpus='0-7' memory='2097152' unit='KiB'/> <cell id='1' cpus='8-14' memory='1048576' unit='KiB'/> <cell id='2' cpus='15' memory='1048576' unit='KiB'/> </numa> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='threads'/> <source file='/nfs/lhuang/r7_ext4.raw'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:cd:d5:02'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </memballoon> <memory model='dimm'> <source> <nodemask>0-1</nodemask> <pagesize unit='KiB'>1048576</pagesize> </source> <target> <size unit='KiB'>10485760</size> <node>0</node> </target> </memory> </devices> </domain> # virsh start RHEL71-lhuang Domain RHEL71-lhuang started # ps aux|grep qemu qemu 25079 99.5 0.1 15172296 263076 ? SLl 13:49 0:13 /usr/libexec/qemu-kvm -name RHEL71-lhuang -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,mem-merge=off -cpu host,+tsc-deadline,pmu=off -m size=4194304k,slots=16,maxmem=555000832k -realtime mlock=on -smp 4,maxcpus=16,sockets=16,cores=1,threads=1 -object iothread,id=iothread1 -object iothread,id=iothread2 -object iothread,id=iothread3 -object iothread,id=iothread4 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu -numa node,nodeid=0,cpus=0-7,mem=2048 -numa node,nodeid=1,cpus=8-14,mem=1024 -numa node,nodeid=2,cpus=15,mem=1024 -object memory-backend-file,id=memdimm0,prealloc=yes,mem-path=/dev/hugepages1G/libvirt/qemu,size=10737418240,host-nodes=0-1,policy=bind -device pc-dimm,node=0,memdev=memdimm0,id=dimm0 -uuid e7041d2f-4811-4fae-93f8-20e029817e63 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-RHEL71-lhuang/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/nfs/lhuang/r7_ext4.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:cd:d5:02,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/r6.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 -msg timestamp=on 3. hotplug a memory device which memory size < 1G: # cat memdev3.xml <memory model='dimm'> <target> <size unit='m'>512</size> <node>1</node> </target> </memory> # virsh attach-device RHEL71-lhuang memdev3.xml error: Failed to attach device from memdev3.xml error: Unable to read from monitor: Connection reset by peer 4. check the guest log: .... 2015-10-09T05:52:55.699057Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000 2015-10-09 05:52:55.793+0000: shutting down .... Actual results: qemu process exit after hot-plug a memory device which memory size < 1G and guest use 1G hugepages Expected results: guest not exit and output error or libvirt catch this more early. Additional info: