RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1270144 - qemu process exit after hot-plug a memory device which guest use 1G hugepages
Summary: qemu process exit after hot-plug a memory device which guest use 1G hugepages
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.2
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Igor Mammedov
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1283079 1288337 1305606
TreeView+ depends on / blocked
 
Reported: 2015-10-09 05:58 UTC by Luyao Huang
Modified: 2016-11-07 20:43 UTC (History)
17 users (show)

Fixed In Version: qemu-kvm-rhev-2.5.0-1.el7
Doc Type: Bug Fix
Doc Text:
Due to an unhandled error, the qemu-kvm process exited when trying to hot-plug memory devices on guests using 1GB hugepages. With this update, the error is handled within qemu-kvm, and as a result, the process does not crash.
Clone Of:
: 1283079 (view as bug list)
Environment:
Last Closed: 2016-11-07 20:43:23 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2673 0 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2016-11-08 01:06:13 UTC

Description Luyao Huang 2015-10-09 05:58:49 UTC 
Description of problem:
qemu process exit after hot-plug a memory device which guest use 1G hugepages

Version-Release number of selected component (if applicable):
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-29.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare hugepages env:

# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-316.el7.x86_64 root=UUID=15d52150-0581-4d91-bc9b-6909fb677212 ro rd.md=0 rd.dm=0 console=tty0 rd.luks=0 processor.max_cstate=1 biosdevname=0 crashkernel=256M rd.lvm=0 reboot=pci console=ttyS0,115200 vconsole.keymap=us vconsole.font=latarcyrheb-sun16 nmi_watchdog=0 rhgb quiet intel_iommu=on isolcpus=14,15 rcu_nocbs=14,15 nohz_full=14,15 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0 intel_pstate=disable

# mount
...
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)
hugetlbfs on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M)

# /etc/libvirt/qemu.conf

hugetlbfs_mount = ["/dev/hugepages1G", "/dev/hugepages2M"]

# virsh freepages --all
Node 0:
4KiB: 9176414
2048KiB: 3600
1048576KiB: 15

Node 1:
4KiB: 9810379
2048KiB: 3600
1048576KiB: 15

2. start a guest which use 1G hugepages

# virsh dumpxml RHEL71-lhuang
<domain type='kvm'>
  <name>RHEL71-lhuang</name>
  <uuid>e7041d2f-4811-4fae-93f8-20e029817e63</uuid>
  <maxMemory slots='16' unit='KiB'>555000000</maxMemory>
  <memory unit='KiB'>14680064</memory>
  <currentMemory unit='KiB'>14680064</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>111111111111111</hard_limit>
  </memtune>
  <memoryBacking>
    <hugepages/>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static' current='4'>16</vcpu>
  <iothreads>4</iothreads>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <pmu state='off'/>
  </features>
  <cpu mode='host-passthrough'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-7' memory='2097152' unit='KiB'/>
      <cell id='1' cpus='8-14' memory='1048576' unit='KiB'/>
      <cell id='2' cpus='15' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='threads'/>
      <source file='/nfs/lhuang/r7_ext4.raw'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:cd:d5:02'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/r6.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </memballoon>
    <memory model='dimm'>
      <source>
        <nodemask>0-1</nodemask>
        <pagesize unit='KiB'>1048576</pagesize>
      </source>
      <target>
        <size unit='KiB'>10485760</size>
        <node>0</node>
      </target>
    </memory>
  </devices>
</domain>


# virsh start RHEL71-lhuang
Domain RHEL71-lhuang started


# ps aux|grep qemu
qemu     25079 99.5  0.1 15172296 263076 ?     SLl  13:49   0:13 /usr/libexec/qemu-kvm -name RHEL71-lhuang -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,mem-merge=off -cpu host,+tsc-deadline,pmu=off -m size=4194304k,slots=16,maxmem=555000832k -realtime mlock=on -smp 4,maxcpus=16,sockets=16,cores=1,threads=1 -object iothread,id=iothread1 -object iothread,id=iothread2 -object iothread,id=iothread3 -object iothread,id=iothread4 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu -numa node,nodeid=0,cpus=0-7,mem=2048 -numa node,nodeid=1,cpus=8-14,mem=1024 -numa node,nodeid=2,cpus=15,mem=1024 -object memory-backend-file,id=memdimm0,prealloc=yes,mem-path=/dev/hugepages1G/libvirt/qemu,size=10737418240,host-nodes=0-1,policy=bind -device pc-dimm,node=0,memdev=memdimm0,id=dimm0 -uuid e7041d2f-4811-4fae-93f8-20e029817e63 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-RHEL71-lhuang/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/nfs/lhuang/r7_ext4.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:cd:d5:02,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/r6.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 -msg timestamp=on


3. hotplug a memory device which memory size < 1G:

# cat memdev3.xml 
    <memory model='dimm'>
      <target>
        <size unit='m'>512</size>
        <node>1</node>
      </target>
    </memory>

# virsh attach-device RHEL71-lhuang memdev3.xml
error: Failed to attach device from memdev3.xml
error: Unable to read from monitor: Connection reset by peer

4. check the guest log:
....
2015-10-09T05:52:55.699057Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
2015-10-09 05:52:55.793+0000: shutting down
....

Actual results:
qemu process exit after hot-plug a memory device which memory size < 1G and guest use 1G hugepages

Expected results:

guest not exit and output error or libvirt catch this more early.

Additional info:
Comment 1 Peter Krempa 2015-10-16 10:41:31 UTC 
Reassigning to qemu. qemu shouldn't just abort during hotplug but rather report an error.
Comment 2 Igor Mammedov 2015-11-06 16:15:08 UTC 
7.2.z backport
http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-November/msg00028.html
Comment 4 Yumei Huang 2015-11-13 08:04:27 UTC 
Reproduce:
kernel:  3.10.0-329.el7.x86_64
qemu-kvm: qemu-kvm-rhev-2.3.0-31.el7.x86_64

steps:
1. prepare hugepages env:
#cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-329.el7.x86_64 root=/dev/mapper/rhel_dhcp--10--61-root ro console=tty0 console=ttyS0,115200 reboot=pci crashkernel=auto rd.lvm.lv=rhel_dhcp-10-61/root rd.lvm.lv=rhel_dhcp-10-61/swap biosdevname=0 rhgb intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0

#mount
...
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,pagesize=1G)
...

# cat /proc/meminfo | grep Huge
AnonHugePages:     24576 kB
HugePages_Total:       8
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

2. start a guest with 1G hugepage:
/usr/libexec/qemu-kvm -name rhel72-sn5 -m 4G,slots=8,maxmem=40G -mem-prealloc -mem-path /dev/hugepages1G -numa node  \ 
-realtime mlock=on  \
 -smp 16,sockets=16,cores=1,threads=1 \
-no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -boot menu=on,strict=on \
-object iothread,id=iothread0 -drive file=/home/guest/rhel72-sn5.img,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop  -device virtio-blk-pci,drive=drive-data-disk,id=data-disk,iothread=iothread0,bus=pci.0,addr=0x7 \
-netdev tap,id=hostnet1,vhost=on -device e1000,netdev=hostnet1,id=net2,mac=10:1a:4a:42:0a:0b,bus=pci.0,addr=0xa \
-usb -device usb-tablet,id=input0 -vga qxl \
-spice port=5901,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on -monitor stdio  \
-msg timestamp=on

3. hotplug a memory device with size=512M
(qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G
2015-11-13T07:41:46.486441Z memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
And the qemu process exit.

So, the bug is reproduced.


Verify:
kernel:  3.10.0-329.el7.x86_64
qemu-kvm: qemu-kvm-rhev-2.3.0-31.el7.next.candidate.x86_64

With same steps as above. The guest doesn't quit, and HMP prints "memory size 0x20000000 must be equal to or larger than huge page size 0x40000000".
So, the bug is fixed.
Comment 6 Miroslav Rezanina 2015-11-18 10:07:05 UTC 
Fix included in qemu-kvm-rhev-2.3.0-31.el7_2.2
Comment 9 Yumei Huang 2016-05-16 08:50:59 UTC 
Reproduce:
qemu-kvm-rhev-2.3.0-31.el7
kernel-3.10.0-373.el7.x86_64

Steps:
1. prepare hugepages env:
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-373.el7.x86_64 root=/dev/mapper/rhel_dell--per715--04-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per715-04/root rd.lvm.lv=rhel_dell-per715-04/swap rhgb quiet LANG=en_US.UTF-8 default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=0

# mount
hugetlbfs on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)

# cat /proc/meminfo | grep Huge
AnonHugePages:      8192 kB
HugePages_Total:       4
HugePages_Free:        4
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

2. start a guest with 1G hugepage:
# /usr/libexec/qemu-kvm  -m 1G,slots=4,maxmem=32G -smp 4   \

-mem-prealloc -mem-path /dev/hugepages1G  -numa node \

-drive file=/home/guest/RHEL-Server-7.3-64-virtio.qcow2,id=drive-virtio-disk1,media=disk,cache=none,snapshot=off,format=qcow2,aio=native,if=none  -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,bootindex=0\

-netdev tap,id=idw97X5F  -device virtio-net-pci,netdev=idw97X5F,mac='9a:37:37:37:37:8e',bus=pci.0,addr=0x5,id='idMqFTXq' -vga cirrus -monitor stdio -vnc :0     

3. hotplug a memory device with size=512M
(qemu) object_add memory-backend-file,id=mem0,size=512M,mem-path=/dev/hugepages1G
memory size 0x20000000 must be equal to or larger than huge page size 0x40000000
And the qemu process exit.

So the bug is reproduced.

Verify:
qemu-kvm-rhev-2.6.0-1.el7
kernel-3.10.0-373.el7.x86_64

with same steps as above, after step3, HMP prints "memory size 0x20000000 must be equal to or larger than page size 0x40000000" , and guest works well. 

So the bug is fixed.
Comment 12 errata-xmlrpc 2016-11-07 20:43:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html
Note You need to log in before you can comment on or make changes to this bug.