Bug 1270144 - qemu process exit after hot-plug a memory device which guest use 1G hugepages
qemu process exit after hot-plug a memory device which guest use 1G hugepages
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 Linux
high Severity medium
: rc
: ---
Assigned To: Igor Mammedov
Virtualization Bugs
: ZStream
Depends On:
Blocks: 1305606 1283079 1288337
  Show dependency treegraph
 
Reported: 2015-10-09 01:58 EDT by Luyao Huang
Modified: 2016-11-07 15:43 EST (History)
17 users (show)

See Also:
Fixed In Version: qemu-kvm-rhev-2.5.0-1.el7
Doc Type: Bug Fix
Doc Text:
Due to an unhandled error, the qemu-kvm process exited when trying to hot-plug memory devices on guests using 1GB hugepages. With this update, the error is handled within qemu-kvm, and as a result, the process does not crash.
Story Points: ---
Clone Of:
: 1283079 (view as bug list)
Environment:
Last Closed: 2016-11-07 15:43:23 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Luyao Huang 2015-10-09 01:58:49 EDT
Description of problem:
qemu process exit after hot-plug a memory device which guest use 1G hugepages

Version-Release number of selected component (if applicable):
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-29.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare hugepages env:

# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-316.el7.x86_64 root=UUID=15d52150-0581-4d91-bc9b-6909fb677212 ro rd.md=0 rd.dm=0 console=tty0 rd.luks=0 processor.max_cstate=1 biosdevname=0 crashkernel=256M rd.lvm=0 reboot=pci console=ttyS0,115200 vconsole.keymap=us vconsole.font=latarcyrheb-sun16 nmi_watchdog=0 rhgb quiet intel_iommu=on isolcpus=14,15 rcu_nocbs=14,15 nohz_full=14,15 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0 intel_pstate=disable

# mount
...
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)
hugetlbfs on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M)

# /etc/libvirt/qemu.conf

hugetlbfs_mount = ["/dev/hugepages1G", "/dev/hugepages2M"]

# virsh freepages --all
Node 0:
4KiB: 9176414
2048KiB: 3600
1048576KiB: 15

Node 1:
4KiB: 9810379
2048KiB: 3600
1048576KiB: 15

2. start a guest which use 1G hugepages

# virsh dumpxml RHEL71-lhuang
<domain type='kvm'>
  <name>RHEL71-lhuang</name>
  <uuid>e7041d2f-4811-4fae-93f8-20e029817e63</uuid>
  <maxMemory slots='16' unit='KiB'>555000000</maxMemory>
  <memory unit='KiB'>14680064</memory>
  <currentMemory unit='KiB'>14680064</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>111111111111111</hard_limit>
  </memtune>
  <memoryBacking>
    <hugepages/>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static' current='4'>16</vcpu>
  <iothreads>4</iothreads>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <pmu state='off'/>
  </features>
  <cpu mode='host-passthrough'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-7' memory='2097152' unit='KiB'/>
      <cell id='1' cpus='8-14' memory='1048576' unit='KiB'/>
      <cell id='2' cpus='15' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='threads'/>
      <source file='/nfs/lhuang/r7_ext4.raw'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:cd:d5:02'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </memballoon>
    <memory model='dimm'>
      <source>
        <nodemask>0-1</nodemask>
        <pagesize unit='KiB'>1048576</pagesize>
      </source>
      <target>
        <size unit='KiB'>10485760</size>
        <node>0</node>
      </target>
    </memory>
  </devices>
</domain>


# virsh start RHEL71-lhuang
Domain RHEL71-lhuang started


# ps aux|grep qemu
qemu     25079 99.5  0.1 15172296 263076 ?     SLl  13:49   0:13 /usr/libexec/qemu-kvm -name RHEL71-lhuang -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,mem-merge=off -cpu host,+tsc-deadline,pmu=off -m size=4194304k,slots=16,maxmem=555000832k -realtime mlock=on -smp 4,maxcpus=16,sockets=16,cores=1,threads=1 -object iothread,id=iothread1 -object iothread,id=iothread2 -object iothread,id=iothread3 -object iothread,id=iothread4 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu -numa node,nodeid=0,cpus=0-7,mem=2048 -numa node,nodeid=1,cpus=8-14,mem=1024 -numa node,nodeid=2,cpus=15,mem=1024 -object memory-backend-file,id=memdimm0,prealloc=yes,mem-path=/dev/hugepages1G/libvirt/qemu,size=10737418240,host-nodes=0-1,policy=bind -device pc-dimm,node=0,memdev=memdimm0,id=dimm0 -uuid e7041d2f-4811-4fae-93f8-20e029817e63 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-RHEL71-lhuang/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/nfs/lhuang/r7_ext4.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:cd:d5:02,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/r6.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 -msg timestamp=on


3. hotplug a memory device which memory size < 1G:

# cat memdev3.xml 
    <memory model='dimm'>
      <target>
        <size unit='m'>512</size>
        <node>1</node>
      </target>
    </memory>

# virsh attach-device RHEL71-lhuang memdev3.xml
error: Failed to attach device from memdev3.xml
error: Unable to read from monitor: Connection reset by peer

4. check the guest log:
....
2015-10-09T05:52:55.699057Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
2015-10-09 05:52:55.793+0000: shutting down
....

Actual results:
qemu process exit after hot-plug a memory device which memory size < 1G and guest use 1G hugepages

Expected results:

guest not exit and output error or libvirt catch this more early.

Additional info:
Comment 1 Peter Krempa 2015-10-16 06:41:31 EDT
Reassigning to qemu. qemu shouldn't just abort during hotplug but rather report an error.
Comment 4 Yumei Huang 2015-11-13 03:04:27 EST
Reproduce:
kernel:  3.10.0-329.el7.x86_64
qemu-kvm: qemu-kvm-rhev-2.3.0-31.el7.x86_64

steps:
1. prepare hugepages env:
#cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-329.el7.x86_64 root=/dev/mapper/rhel_dhcp--10--61-root ro console=tty0 console=ttyS0,115200 reboot=pci crashkernel=auto rd.lvm.lv=rhel_dhcp-10-61/root rd.lvm.lv=rhel_dhcp-10-61/swap biosdevname=0 rhgb intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0

#mount
...
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,pagesize=1G)
...

# cat /proc/meminfo | grep Huge
AnonHugePages:     24576 kB
HugePages_Total:       8
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

2. start a guest with 1G hugepage:
/usr/libexec/qemu-kvm -name rhel72-sn5 -m 4G,slots=8,maxmem=40G -mem-prealloc -mem-path /dev/hugepages1G -numa node  \ 
-realtime mlock=on  \
 -smp 16,sockets=16,cores=1,threads=1 \
-no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -boot menu=on,strict=on \
-object iothread,id=iothread0 -drive file=/home/guest/rhel72-sn5.img,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop  -device virtio-blk-pci,drive=drive-data-disk,id=data-disk,iothread=iothread0,bus=pci.0,addr=0x7 \
-netdev tap,id=hostnet1,vhost=on -device e1000,netdev=hostnet1,id=net2,mac=10:1a:4a:42:0a:0b,bus=pci.0,addr=0xa \
-usb -device usb-tablet,id=input0 -vga qxl \
-spice port=5901,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on -monitor stdio  \
-msg timestamp=on

3. hotplug a memory device with size=512M
(qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G
2015-11-13T07:41:46.486441Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
And the qemu process exit.

So, the bug is reproduced.


Verify:
kernel:  3.10.0-329.el7.x86_64
qemu-kvm: qemu-kvm-rhev-2.3.0-31.el7.next.candidate.x86_64

With same steps as above. The guest doesn't quit, and HMP prints "memory size 0x20000000 must be equal to or larger than huge page size 0x40000000".
So, the bug is fixed.
Comment 6 Miroslav Rezanina 2015-11-18 05:07:05 EST
Fix included in qemu-kvm-rhev-2.3.0-31.el7_2.2
Comment 9 Yumei Huang 2016-05-16 04:50:59 EDT
Reproduce:
qemu-kvm-rhev-2.3.0-31.el7
kernel-3.10.0-373.el7.x86_64

Steps:
1. prepare hugepages env:
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-373.el7.x86_64 root=/dev/mapper/rhel_dell--per715--04-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per715-04/root rd.lvm.lv=rhel_dell-per715-04/swap rhgb quiet LANG=en_US.UTF-8 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0

# mount
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)

# cat /proc/meminfo | grep Huge
AnonHugePages:      8192 kB
HugePages_Total:       4
HugePages_Free:        4
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

2. start a guest with 1G hugepage:
# /usr/libexec/qemu-kvm  -m 1G,slots=4,maxmem=32G -smp 4   \

-mem-prealloc -mem-path /dev/hugepages1G  -numa node \

-drive file=/home/guest/RHEL-Server-7.3-64-virtio.qcow2,id=drive-virtio-disk1,media=disk,cache=none,snapshot=off,format=qcow2,aio=native,if=none  -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,bootindex=0\

-netdev tap,id=idw97X5F  -device virtio-net-pci,netdev=idw97X5F,mac='9a:37:37:37:37:8e',bus=pci.0,addr=0x5,id='idMqFTXq' -vga cirrus -monitor stdio -vnc :0     

3. hotplug a memory device with size=512M
(qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G
memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
And the qemu process exit.

So the bug is reproduced.

Verify:
qemu-kvm-rhev-2.6.0-1.el7
kernel-3.10.0-373.el7.x86_64

with same steps as above, after step3, HMP prints "memory size 0x20000000 must be equal to or larger than page size 0x40000000" , and guest works well. 

So the bug is fixed.
Comment 12 errata-xmlrpc 2016-11-07 15:43:23 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html

Note You need to log in before you can comment on or make changes to this bug.