Bug 1250378 - when boot multiple guests in one host,booting of the second guest will cause the first guest be killed as out of memory
when boot multiple guests in one host,booting of the second guest will cause ...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 Windows
medium Severity medium
: rc
: ---
Assigned To: Radim Krčmář
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-05 05:12 EDT by Yanan Fu
Modified: 2016-03-28 05:09 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-16 10:01:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
error log when the bug duplicate (15.89 KB, text/plain)
2015-08-05 21:36 EDT, Yanan Fu
no flags Details

  None (edit)
Description Yanan Fu 2015-08-05 05:12:20 EDT
Description of problem:
My host memory is 4G, boot one guest with "-m 4G", it works well.
then boot the second guest with "-m 4G" too, it can boot successfully, but the first guest will be killed.
If i change the CLI to "-m 2G", the two guest are ok.

Version-Release number of selected component (if applicable):
qemu-kvm: 2.3.0-13.el7
kernel:3.10.0-300.el7.x86_64
guest: win 8.1 (64)
virtio-win:virtio-win-1.7.4-1.el7.noarch

How reproducible:

Steps to Reproduce:
1.boot a win8.1 guest with "-m 4G", can boot successfully.
2.boot the second win8.1 guest with "-m 4G" too, boot successfully
3.check the first guest, it has been killed.

Actual results:
the first guest be killed.

Expected results:
the second guest should failed to boot, and show useful info to inform.

Additional info:
CLI
error log:
#Aug  5 15:04:09 localhost kernel: Out of memory: Kill process 12577 (qemu-kvm) score 270 or sacrifice child
#Aug  5 15:04:09 localhost kernel: Killed process 12577 (qemu-kvm) total-vm:4684256kB, anon-rss:4414688kB, file-rss:42204kB

normally,showed in the second qemu monitor:
(qemu) qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory Aborted
Comment 3 Yanan Fu 2015-08-05 21:36:30 EDT
Created attachment 1059690 [details]
error log when the bug duplicate
Comment 5 Radim Krčmář 2015-08-11 16:43:38 EDT
Are you using '-realtime mlock=on' QEMU parameter?

The host has 8G of memory so two 4G guests will fit only with the help of swap (or KSM).  The log shows that QEMU has no pages in the almost empty 8G swap, which points to the mlock parameter that disables swap for QEMU and allocates all memory on startup.

The memory is allocated and locked in two phases.
The allocation phase works fine because we allow memory overcommit (no pages are allocated at that point), but in the locking phase, there is not enough RAM to fit the second guest and the mlockall sysctl cannot fail because of insufficient memory (Linux is tragic in low-memory situations), so a oom killer pops up and something must die.

QEMU used to say "qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory" in RHEL 7.0?
Comment 6 Yanan Fu 2015-08-12 22:41:14 EDT
(In reply to Radim Krčmář from comment #5)
> Are you using '-realtime mlock=on' QEMU parameter?
> 
> The host has 8G of memory so two 4G guests will fit only with the help of
> swap (or KSM).  The log shows that QEMU has no pages in the almost empty 8G
> swap, which points to the mlock parameter that disables swap for QEMU and
> allocates all memory on startup.
> 
> The memory is allocated and locked in two phases.
> The allocation phase works fine because we allow memory overcommit (no pages
> are allocated at that point), but in the locking phase, there is not enough
> RAM to fit the second guest and the mlockall sysctl cannot fail because of
> insufficient memory (Linux is tragic in low-memory situations), so a oom
> killer pops up and something must die.
> 
> QEMU used to say "qemu-kvm: cannot set up guest memory 'pc.ram': Cannot
> allocate memory" in RHEL 7.0?

yes, i add '-realtime mlock=on' QEMU parameter. without this parameter,two VMs can start successfully.

Now, i have test many times in my side, but can not trigger:"qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory" now.
And i find, if i echo the score in "/pro/[qemu pid]/oom_adj",the order of the "kill" will be changed. (the one with higher score will be killed first).

command line:
/usr/libexec/qemu-kvm -name test -machine pc,accel=kvm,usb=off,dump-guest-core=on -m 6G -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1 -no-user-config -nodefaults -monitor stdio -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,strict=on -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x5 -device ich9-usb-ehci1,id=usb,bus=bridge1,addr=0x2.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=bridge1,multifunction=on,addr=0x2.0x0 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=bridge1,addr=0x2.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=bridge1,addr=0x2.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=bridge1,addr=0x4 -chardev socket,path=/tmp/yfu,server,nowait,id=yfu0 -netdev tap,id=hostnet,vhost=on -device virtio-net-pci,netdev=hostnet,id=net,mac=78:1a:4a:d6:b8:97,bus=bridge1,addr=0x6,bootindex=2 -vnc :1 -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -msg timestamp=on -monitor unix:/home/qmp,server,nowait -serial unix:/tmp/ttyS0,server,nowait -qmp tcp:0:4444,server,nowait -drive file=/root/RHEL-7.2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 -S -realtime mlock=on
Comment 7 Radim Krčmář 2015-09-16 10:01:23 EDT
The memory overcommit behaviour with '-realtime mlock=on' is not perfect, but expected and requires custom configuration, so the admin shouldn't let it happen.  Closing because improving the error case is not worth the effort.

Note You need to log in before you can comment on or make changes to this bug.