Description of problem:

When creating a domain, xend balloons down dom0 memory for memory space of the
guest. This mechanism usually works fine, but sometimes creating a domain fails
showing "cannot allocate memory", even if dom0 seems to have enough memory for
ballooning down. Once this problem happens, I can't create the domain until I
balloon down dom0 memory by "xm mem-set" manually.

Version-Release number of selected component:
RHEL5.1GA-RC
xen-3.0.3-41.el5

How reproducible:
Rarely. However it seems that the probability is higher if dom0 itself is under
high load.

Steps to Reproduce:
- boot dom0.
- create some domains.
- reboot dom0 until this problem happens.

Actual results:
Xen fails in creating the domain.

Expected results:
The domain will be created properly.

Hardware info:
x86_64:
 NEC Express5800/140Rd-4
i386:
 NEC Express5800/120Rg-1

Business impact:
Users may fail in creating domains, and have to run "xm mem-set" manually.

Additional info:
- We have seen this problem happen only when creating full virtualized guests. 
Even when this problem happened, I could create a para virtualized guest with
the same memory size.

- dom0 boot option

title Red Hat Enterprise Linux Server (2.6.18-52.el5xen) x86_64
root (hd0,5)
kernel /boot/xen.gz-2.6.18-52.el5 crashkernel=128M@32M
module /boot/vmlinuz-2.6.18-52.el5xen ro root=LABEL=/1 rhgb quiet
module /boot/initrd-2.6.18-52.el5xen.img

- When this problem happnes, xend.log looks like:

[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:200) XendDomainInfo.create(['vm', ['name',
'RC_x86_64_full_disk'], ['memory', '2048'], ['maxmem', '2048'],
['vcpus', '2'], ['uuid', '90038a46b51e9a9b7d12ad42eced2a77'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
'restart'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'],
['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot',
'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'],
['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']]], ['device', ['vbd',
['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]],
['device', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'],
['type', 'ioemu']]]])
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:306) parseConfig: config is ['vm', ['name',
'RC_x86_64_full_disk'], ['memory', '2048'], ['maxmem', '2048'],
['vcpus', '2'], ['uuid', '90038a46b51e9a9b7d12ad42eced2a77'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
'restart'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'],
['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot',
'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'],
['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']]], ['device', ['vbd',
['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]],
['device', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'],
['type', 'ioemu']]]]
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:411) parseConfig: result is {'shadow_memory': None,
'start_time': None, 'uuid': '90038a46-b51e-9a9b-7d12-ad42eced2a77',
'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None,
'image': ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'],
['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot',
'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'],
['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']], 'on_poweroff':
'destroy', 'bootloader_args': None, 'cpus': None, 'name':
'RC_x86_64_full_disk', 'backend': [], 'vcpus': 2, 'cpu_weight': None,
'features': None, 'vcpu_avail': None, 'memory': 2048, 'device': [('vbd',
['vbd', ['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode',
'w']]), ('vif', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge',
'virbr0'], ['type', 'ioemu']])], 'bootloader': None, 'cpu': None,
'maxmem': 2048}
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:1349) XendDomainInfo.construct: None
[2007-10-25 11:02:29 xend 3807] DEBUG (balloon:133) Balloon: 640 KiB
free; 0 to scrub; need 2048; retries: 20.
[2007-10-25 11:02:29 xend 3807] DEBUG (balloon:148) Balloon: setting
dom0 target to 3807 MiB.
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:1078) Setting memory target of domain Domain-0 (0) to
3807 MiB.
[2007-10-25 11:02:29 xend 3807] DEBUG (balloon:127) Balloon: 2688 KiB
free; need 2048; done.
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] ERROR
(XendDomainInfo:212) Domain construction failed
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 204, in create
vm.construct()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 1369, in construct
hvm = hvm)
Error: (12, 'Cannot allocate memory')
[2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG
(XendDomainInfo:1557) XendDomainInfo.destroy: domid=None
[2007-10-25 11:02:29 xend 3807] ERROR (SrvBase:88) Request create failed.
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/web/SrvBase.py", line 85,
in perform
return op_method(op, req)
File
"/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 82, in op_create
raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (12, 'Cannot allocate memory')