Bug 1329086

Summary: Qemu should prevent guest booting when no enough hugepage in binded node
Product: Red Hat Enterprise Linux 7 Reporter: Yumei Huang <yuhuang>
Component: qemu-kvm-rhevAssignee: Luiz Capitulino <lcapitulino>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.3CC: chayang, hhuang, imammedo, juzhang, mdeng, michen, qzhang, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-21 12:51:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Yumei Huang 2016-04-21 07:27:45 UTC 
Description of problem:
Boot one guest with hugepage binding to one node which doesn't have enough hugepage, when run dd cmd in guest, guest hangs, and hmp prints "(qemu) error: kvm run failed Bad address".  QE think qemu should prevent guest booting when no enough hugepages in the binded node. 

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.5.0-4.el7
kernel-3.10.0-373.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. In host(4 nodes), allocate 512M hugepage on each node. 
# cat /proc/meminfo | grep -i huge
AnonHugePages:      6144 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

# numastat -m | grep -i huge
AnonHugePages               0.00            0.00            6.00            0.00            6.00
HugePages_Total           512.00          512.00          512.00          512.00         2048.00
HugePages_Free            512.00          512.00          512.00          512.00         2048.00
HugePages_Surp              0.00            0.00            0.00            0.00            0.00

2. boot guest with 1G hugepage binding to node0
/usr/libexec/qemu-kvm  -m 1G,slots=4,maxmem=32G -smp 4 \

-object memory-backend-file,policy=bind,mem-path=/mnt/kvm_hugepage,host-nodes=0,size=1G,id=mem-mem1   -device pc-dimm,id=dimm-mem1,memdev=mem-mem1 \

-drive file=/home/guest/RHEL-Server-7.3-64-virtio.qcow2,id=drive-virtio-disk1,media=disk,cache=none,snapshot=off,format=qcow2,aio=native,if=none  -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,bootindex=0  -monitor stdio -vnc :0

3. run dd cmd in guest to allocate memory
# mount -t tmpfs none /mnt
# dd if=/dev/zero of=/mnt/test bs=1M count=1000

Actual results:
After step2, guest can boot successfully.
After step3, guest hangs, and hmp prints:
"
(qemu) error: kvm run failed Bad address
RAX=0000000000000000 RBX=ffff88003e001b00 RCX=0000000000000000 RDX=0000000000000040
RSI=ffff880122800000 RDI=ffff88003e001b00 RBP=ffff88012feb3a60 RSP=ffff88012feb3a38
R8 =0000000000000004 R9 =0000000000000000 R10=0000000000000003 R11=ffffffffffffffe2
R12=ffff880122800000 R13=ffffea00048a0000 R14=ffff880122800000 R15=ffff880122800000
RIP=ffffffff811be8d5 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00800000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0000 0000000000000000 ffffffff 00800000
FS =0000 00007f9ae55c8740 ffffffff 00800000
GS =0000 ffff88003fd80000 ffffffff 00800000
LDT=0000 0000000000000000 0000ffff 00000000
TR =0040 ffff88003fd91b40 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff88003fd89000 0000007f
IDT=     ffffffffff529000 00000fff
CR0=8005003b CR2=000000000bcf8003 CR3=000000003af47000 CR4=000007e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=89 e7 49 89 cc 4c 89 fe 48 89 df e8 5f fc ff ff 48 63 43 20 <4d> 89 24 07 41 0f b7 55 1a 8b 43 18 81 e2 ff 7f 00 00 48 63 c8 0f af d0 4c 01 e1 48 63 d2
"

Expected results:
After step2, qemu should quit and print "unable to map backing store for hugepages: Cannot allocate memory"

Additional info:
Comment 2 Yumei Huang 2016-04-21 07:50:35 UTC 
Correct comment0 step1, add mount step:
# mount -t hugetlbfs none /mnt/kvm_hugepage
Comment 3 Luiz Capitulino 2016-04-21 12:51:23 UTC 
Please, use the -mem-prealloc option. QEMU's default behavior by design is to continue when hugepage allocation fails.
Comment 4 Yumei Huang 2016-04-22 02:06:52 UTC 
(In reply to Luiz Capitulino from comment #3)
> Please, use the -mem-prealloc option. QEMU's default behavior by design is
> to continue when hugepage allocation fails.

Actually, when memory-backend-file(mem-mem1)'s size is 2G and without -mem-prealloc option, qemu quits directly and prints "unable to map backing store for hugepages: Cannot allocate memory".

Should I file a new bug for this ?
Comment 5 Luiz Capitulino 2016-04-22 17:57:43 UTC 
I guess this may have something to do with the numa memory policy you're using, but it's better to check with the memory backend device maintainer.

Igor?
Comment 6 Igor Mammedov 2016-05-02 13:35:58 UTC 
(In reply to Yumei Huang from comment #4)
> (In reply to Luiz Capitulino from comment #3)
> > Please, use the -mem-prealloc option. QEMU's default behavior by design is
> > to continue when hugepage allocation fails.
> 
> Actually, when memory-backend-file(mem-mem1)'s size is 2G and without
> -mem-prealloc option, qemu quits directly and prints "unable to map backing
> store for hugepages: Cannot allocate memory".
> 
> Should I file a new bug for this ?

memory is allocated before 'bind' policy comes in to effect,
so it's the same for 1Gb or 2Gb case.
Allocation may fail at early depending on available resources or
may fail later at runtime when accessing not yet allocated page.
I'd say what you see here is normal.

To ensure that guest won't hang/crash for not enough memory reason,
one needs to use mem-prealloc option.
Comment 7 Yumei Huang 2017-05-27 05:33:24 UTC 
*** Bug 1455450 has been marked as a duplicate of this bug. ***