Bug 419771 - Creating full virt domain fails showing 'cannot allocate memory'
Creating full virt domain fails showing 'cannot allocate memory'
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
All Linux
high Severity medium
: ---
: ---
Assigned To: Michal Novotny
Virtualization Bugs
: Reopened
: 251295 (view as bug list)
Depends On:
Blocks: 445799 477162 524624
  Show dependency treegraph
Reported: 2007-12-11 09:09 EST by Sachin Prabhu
Modified: 2014-02-02 17:36 EST (History)
13 users (show)

See Also:
Fixed In Version: xen-3.0.3-85.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 524624 (view as bug list)
Last Closed: 2009-09-02 06:06:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
information from i386 machine (7.88 KB, application/x-compressed-tar)
2007-12-11 09:14 EST, Sachin Prabhu
no flags Details
information from x86_64 machine (2.71 KB, application/x-compressed-tar)
2007-12-11 09:15 EST, Sachin Prabhu
no flags Details
Domain creation 4M memory allocation patch (1.03 KB, patch)
2009-03-26 05:59 EDT, Michal Novotny
no flags Details | Diff

  None (edit)
Description Sachin Prabhu 2007-12-11 09:09:23 EST
Description of problem:

When creating a domain, xend balloons down dom0 memory for memory space of the
guest. This mechanism usually works fine, but sometimes creating a domain fails
showing "cannot allocate memory", even if dom0 seems to have enough memory for
ballooning down. Once this problem happens, I can't create the domain until I
balloon down dom0 memory by "xm mem-set" manually.

Version-Release number of selected component:

How reproducible:
Rarely. However it seems that the probability is higher if dom0 itself is under
high load.

Steps to Reproduce:
- boot dom0.
- create some domains.
- reboot dom0 until this problem happens.

Actual results:
Xen fails in creating the domain.

Expected results:
The domain will be created properly.

Hardware info:
 NEC Express5800/140Rd-4
 NEC Express5800/120Rg-1

Business impact:
Users may fail in creating domains, and have to run "xm mem-set" manually.

Additional info:
- We have seen this problem happen only when creating full virtualized guests. 
Even when this problem happened, I could create a para virtualized guest with
the same memory size.

- dom0 boot option

title Red Hat Enterprise Linux Server (2.6.18-52.el5xen) x86_64
root (hd0,5)
kernel /boot/xen.gz-2.6.18-52.el5 crashkernel=128M@32M
module /boot/vmlinuz-2.6.18-52.el5xen ro root=LABEL=/1 rhgb quiet
module /boot/initrd-2.6.18-52.el5xen.img

- When this problem happnes, xend.log looks like:

[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:200) XendDomainInfo.create(['vm', ['name',
'RC_x86_64_full_disk'], ['memory', '2048'], ['maxmem', '2048'],
['vcpus', '2'], ['uuid', '90038a46b51e9a9b7d12ad42eced2a77'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
'restart'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'],
['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot',
'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'],
['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']]], ['device', ['vbd',
['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]],
['device', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'],
['type', 'ioemu']]]])
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:306) parseConfig: config is ['vm', ['name',
'RC_x86_64_full_disk'], ['memory', '2048'], ['maxmem', '2048'],
['vcpus', '2'], ['uuid', '90038a46b51e9a9b7d12ad42eced2a77'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
'restart'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'],
['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot',
'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'],
['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']]], ['device', ['vbd',
['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]],
['device', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'],
['type', 'ioemu']]]]
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:411) parseConfig: result is {'shadow_memory': None,
'start_time': None, 'uuid': '90038a46-b51e-9a9b-7d12-ad42eced2a77',
'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None,
'image': ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'],
['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot',
'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'],
['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']], 'on_poweroff':
'destroy', 'bootloader_args': None, 'cpus': None, 'name':
'RC_x86_64_full_disk', 'backend': [], 'vcpus': 2, 'cpu_weight': None,
'features': None, 'vcpu_avail': None, 'memory': 2048, 'device': [('vbd',
['vbd', ['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode',
'w']]), ('vif', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge',
'virbr0'], ['type', 'ioemu']])], 'bootloader': None, 'cpu': None,
'maxmem': 2048}
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:1349) XendDomainInfo.construct: None
[2007-10-25 11:02:29 xend 3807] DEBUG (balloon:133) Balloon: 640 KiB
free; 0 to scrub; need 2048; retries: 20.
[2007-10-25 11:02:29 xend 3807] DEBUG (balloon:148) Balloon: setting
dom0 target to 3807 MiB.
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:1078) Setting memory target of domain Domain-0 (0) to
3807 MiB.
[2007-10-25 11:02:29 xend 3807] DEBUG (balloon:127) Balloon: 2688 KiB
free; need 2048; done.
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] ERROR
(XendDomainInfo:212) Domain construction failed
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 204, in create
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 1369, in construct
hvm = hvm)
Error: (12, 'Cannot allocate memory')
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:1557) XendDomainInfo.destroy: domid=None
[2007-10-25 11:02:29 xend 3807] ERROR (SrvBase:88) Request create failed.
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/web/SrvBase.py", line 85,
in perform
return op_method(op, req)
line 82, in op_create
raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (12, 'Cannot allocate memory')
Comment 1 Sachin Prabhu 2007-12-11 09:14:02 EST
Created attachment 284061 [details]
information from i386 machine

balloon.txt  xend.log  xm-info.txt  xm-list.txt taken after a failed attempt 
on i386 machine
Comment 2 Sachin Prabhu 2007-12-11 09:15:08 EST
Created attachment 284071 [details]
information from x86_64 machine

balloon.txt  xend.log  xm-info.txt  xm-list.txt
taken after failed attempt on x86_64 machine
Comment 5 RHEL Product and Program Management 2008-06-02 16:26:37 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
Comment 6 Steven Scoleri 2008-07-24 16:41:20 EDT
Is there a update (even prerelease) for this yet?  Has anyone come up with a
work around other than rebooting the HV?

Only effects me on fully virtuals as the original comment.

Anyway to detect when I'm close to running out of this type of memory since xm
list isn't reporting it correctly?
Comment 10 Michal Novotny 2009-03-05 11:22:32 EST
*** Bug 251295 has been marked as a duplicate of this bug. ***
Comment 11 Michal Novotny 2009-03-06 10:57:31 EST
Well, I was unable to duplicate this problem using Chris' virttest11xen kernel. I have tried to figure it out for some time and this is what happened sometimes when starting up new domains and mixing PV and FV ones too. I finally found them in the kernel-xen package connected to memory allocation but later I got advised to use virttest11xen kernel so I did and since then I was unable to replicate it. Unfortunatelly I don't know what solved this issue...
Comment 13 Michal Novotny 2009-03-09 04:51:19 EDT
Hi Masaki,
please try Chris' xen kernel. You can download it from:

I tested it using this kernel and I was unable to replicate the problem anymore and if you find any problem please tell me steps to replicate the problem on this kernel.

Comment 18 Michal Novotny 2009-03-26 05:59:15 EDT
Created attachment 336771 [details]
Domain creation 4M memory allocation patch

This is a patch for domain memory creation. There was an error about domain memory creation because there was not enough memory allocated it. This patch fixes this issue. It's been tested using this scenario on 8G physical memory machine:

1. xm create RHEL53 maxmem=2500 memory=2500
2. xm create RHEL5FV maxmem=2500 memory=2500
3. virt-install -l http://path/to/install/ -r 2500 -v --nodisks -n RHELtest

It was working correctly and before my patch it was not working at all and it returned error: "Cannot allocate memory". Later I tried it using virt-manager and to allocate 2500M of memory and it was working as well. Please try it using my patch...
Comment 23 Michal Novotny 2009-04-21 05:27:12 EDT
has any triage been done using my RPMs?

Comment 24 Jiri Denemark 2009-05-11 09:40:06 EDT
Fix built into xen-3.0.3-85.el5
Comment 26 Michal Novotny 2009-05-14 03:17:29 EDT
Here it is: http://people.redhat.com/jdenemar/xen.el5/ . Could you provide me /var/log/xen/xend.log if it fails again? Anyway could you give me exact steps to reproduce it and provide some information about platform (i386, x86_64, ia64) etc.?
I was unable to reproduce it using my patch on x86_64 one.

Comment 28 Chris Ward 2009-07-03 13:59:22 EDT
~~ Attention - RHEL 5.4 Beta Released! ~~

RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner!

If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity.

Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value.

Questions can be posted to this bug or your customer or partner representative.
Comment 31 Edward Wang 2009-07-26 22:27:52 EDT
This bug has been verified with xen 3.0.3-91.el5 on RHEL-5.4. Already fixed, set status to VERIFIED.
Comment 33 Edward Wang 2009-07-27 06:06:11 EDT

Sorry for the late response and followings are my verification steps. If you think anything not appropriate, please do not hesitate correting me. Thanks.

Suppose there is totally 4GB memory in the testing host:
1. balloon up dom0 memory to 3.5GB by "xm mem-set 0 3584";
2. create one pv domain rhel5u4pv by "xm create rhel5u4pv maxmem=3584 memory=3584", note that the unit is MB and 3584MB equals to 3.5GB;
3. destroy domain created in step 2;
4. repeat step 1 ~3 by picking up other different values for "maxmem" & "memory" above.

In step 2, during dom rhel5u4pv creation, because system free memory(equas to 4GB minus dom0 memory approximately) can not satisfy the requirement, dom0 is forced to balloon out some of its memory, it is expected and just the result of this bug fixing. Based on this(dom0 can balloon out part of its memory in a rational scope), I tell that this bug is fixed.

It is must to reserve certain minimal memory for dom0 which is defined in xend config file "/etc/xen/xend-config.sxp".

Thanks and hope it is what you needed
Edward Wang
Comment 36 errata-xmlrpc 2009-09-02 06:06:26 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.