Red Hat Bugzilla – Bug 1270144
qemu process exit after hot-plug a memory device which guest use 1G hugepages
Last modified: 2016-11-07 15:43:23 EST
Description of problem: qemu process exit after hot-plug a memory device which guest use 1G hugepages Version-Release number of selected component (if applicable): libvirt-1.2.17-13.el7.x86_64 qemu-kvm-rhev-2.3.0-29.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. prepare hugepages env: # cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10.0-316.el7.x86_64 root=UUID=15d52150-0581-4d91-bc9b-6909fb677212 ro rd.md=0 rd.dm=0 console=tty0 rd.luks=0 processor.max_cstate=1 biosdevname=0 crashkernel=256M rd.lvm=0 reboot=pci console=ttyS0,115200 vconsole.keymap=us vconsole.font=latarcyrheb-sun16 nmi_watchdog=0 rhgb quiet intel_iommu=on isolcpus=14,15 rcu_nocbs=14,15 nohz_full=14,15 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0 intel_pstate=disable # mount ... hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G) hugetlbfs on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M) # /etc/libvirt/qemu.conf hugetlbfs_mount = ["/dev/hugepages1G", "/dev/hugepages2M"] # virsh freepages --all Node 0: 4KiB: 9176414 2048KiB: 3600 1048576KiB: 15 Node 1: 4KiB: 9810379 2048KiB: 3600 1048576KiB: 15 2. start a guest which use 1G hugepages # virsh dumpxml RHEL71-lhuang <domain type='kvm'> <name>RHEL71-lhuang</name> <uuid>e7041d2f-4811-4fae-93f8-20e029817e63</uuid> <maxMemory slots='16' unit='KiB'>555000000</maxMemory> <memory unit='KiB'>14680064</memory> <currentMemory unit='KiB'>14680064</currentMemory> <memtune> <hard_limit unit='KiB'>111111111111111</hard_limit> </memtune> <memoryBacking> <hugepages/> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static' current='4'>16</vcpu> <iothreads>4</iothreads> <numatune> <memory mode='strict' nodeset='0-1'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <pmu state='off'/> </features> <cpu mode='host-passthrough'> <feature policy='require' name='tsc-deadline'/> <numa> <cell id='0' cpus='0-7' memory='2097152' unit='KiB'/> <cell id='1' cpus='8-14' memory='1048576' unit='KiB'/> <cell id='2' cpus='15' memory='1048576' unit='KiB'/> </numa> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='threads'/> <source file='/nfs/lhuang/r7_ext4.raw'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:cd:d5:02'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </memballoon> <memory model='dimm'> <source> <nodemask>0-1</nodemask> <pagesize unit='KiB'>1048576</pagesize> </source> <target> <size unit='KiB'>10485760</size> <node>0</node> </target> </memory> </devices> </domain> # virsh start RHEL71-lhuang Domain RHEL71-lhuang started # ps aux|grep qemu qemu 25079 99.5 0.1 15172296 263076 ? SLl 13:49 0:13 /usr/libexec/qemu-kvm -name RHEL71-lhuang -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,mem-merge=off -cpu host,+tsc-deadline,pmu=off -m size=4194304k,slots=16,maxmem=555000832k -realtime mlock=on -smp 4,maxcpus=16,sockets=16,cores=1,threads=1 -object iothread,id=iothread1 -object iothread,id=iothread2 -object iothread,id=iothread3 -object iothread,id=iothread4 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu -numa node,nodeid=0,cpus=0-7,mem=2048 -numa node,nodeid=1,cpus=8-14,mem=1024 -numa node,nodeid=2,cpus=15,mem=1024 -object memory-backend-file,id=memdimm0,prealloc=yes,mem-path=/dev/hugepages1G/libvirt/qemu,size=10737418240,host-nodes=0-1,policy=bind -device pc-dimm,node=0,memdev=memdimm0,id=dimm0 -uuid e7041d2f-4811-4fae-93f8-20e029817e63 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-RHEL71-lhuang/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/nfs/lhuang/r7_ext4.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:cd:d5:02,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/r6.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 -msg timestamp=on 3. hotplug a memory device which memory size < 1G: # cat memdev3.xml <memory model='dimm'> <target> <size unit='m'>512</size> <node>1</node> </target> </memory> # virsh attach-device RHEL71-lhuang memdev3.xml error: Failed to attach device from memdev3.xml error: Unable to read from monitor: Connection reset by peer 4. check the guest log: .... 2015-10-09T05:52:55.699057Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000 2015-10-09 05:52:55.793+0000: shutting down .... Actual results: qemu process exit after hot-plug a memory device which memory size < 1G and guest use 1G hugepages Expected results: guest not exit and output error or libvirt catch this more early. Additional info:
Reassigning to qemu. qemu shouldn't just abort during hotplug but rather report an error.
7.2.z backport http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-November/msg00028.html
Reproduce: kernel: 3.10.0-329.el7.x86_64 qemu-kvm: qemu-kvm-rhev-2.3.0-31.el7.x86_64 steps: 1. prepare hugepages env: #cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-329.el7.x86_64 root=/dev/mapper/rhel_dhcp--10--61-root ro console=tty0 console=ttyS0,115200 reboot=pci crashkernel=auto rd.lvm.lv=rhel_dhcp-10-61/root rd.lvm.lv=rhel_dhcp-10-61/swap biosdevname=0 rhgb intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0 #mount ... hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,pagesize=1G) ... # cat /proc/meminfo | grep Huge AnonHugePages: 24576 kB HugePages_Total: 8 HugePages_Free: 8 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB 2. start a guest with 1G hugepage: /usr/libexec/qemu-kvm -name rhel72-sn5 -m 4G,slots=8,maxmem=40G -mem-prealloc -mem-path /dev/hugepages1G -numa node \ -realtime mlock=on \ -smp 16,sockets=16,cores=1,threads=1 \ -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -boot menu=on,strict=on \ -object iothread,id=iothread0 -drive file=/home/guest/rhel72-sn5.img,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-data-disk,id=data-disk,iothread=iothread0,bus=pci.0,addr=0x7 \ -netdev tap,id=hostnet1,vhost=on -device e1000,netdev=hostnet1,id=net2,mac=10:1a:4a:42:0a:0b,bus=pci.0,addr=0xa \ -usb -device usb-tablet,id=input0 -vga qxl \ -spice port=5901,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on -monitor stdio \ -msg timestamp=on 3. hotplug a memory device with size=512M (qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G 2015-11-13T07:41:46.486441Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000 And the qemu process exit. So, the bug is reproduced. Verify: kernel: 3.10.0-329.el7.x86_64 qemu-kvm: qemu-kvm-rhev-2.3.0-31.el7.next.candidate.x86_64 With same steps as above. The guest doesn't quit, and HMP prints "memory size 0x20000000 must be equal to or larger than huge page size 0x40000000". So, the bug is fixed.
Fix included in qemu-kvm-rhev-2.3.0-31.el7_2.2
Reproduce: qemu-kvm-rhev-2.3.0-31.el7 kernel-3.10.0-373.el7.x86_64 Steps: 1. prepare hugepages env: # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-373.el7.x86_64 root=/dev/mapper/rhel_dell--per715--04-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per715-04/root rd.lvm.lv=rhel_dell-per715-04/swap rhgb quiet LANG=en_US.UTF-8 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0 # mount hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G) # cat /proc/meminfo | grep Huge AnonHugePages: 8192 kB HugePages_Total: 4 HugePages_Free: 4 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB 2. start a guest with 1G hugepage: # /usr/libexec/qemu-kvm -m 1G,slots=4,maxmem=32G -smp 4 \ -mem-prealloc -mem-path /dev/hugepages1G -numa node \ -drive file=/home/guest/RHEL-Server-7.3-64-virtio.qcow2,id=drive-virtio-disk1,media=disk,cache=none,snapshot=off,format=qcow2,aio=native,if=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,bootindex=0\ -netdev tap,id=idw97X5F -device virtio-net-pci,netdev=idw97X5F,mac='9a:37:37:37:37:8e',bus=pci.0,addr=0x5,id='idMqFTXq' -vga cirrus -monitor stdio -vnc :0 3. hotplug a memory device with size=512M (qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G memory size 0x20000000 must be equal to or larger than huge page size 0x40000000 And the qemu process exit. So the bug is reproduced. Verify: qemu-kvm-rhev-2.6.0-1.el7 kernel-3.10.0-373.el7.x86_64 with same steps as above, after step3, HMP prints "memory size 0x20000000 must be equal to or larger than page size 0x40000000" , and guest works well. So the bug is fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2673.html