Bug 419771
| Summary: | Creating full virt domain fails showing 'cannot allocate memory' | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Sachin Prabhu <sprabhu> | ||||||||
| Component: | xen | Assignee: | Michal Novotny <minovotn> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 5.1 | CC: | areis, astokes, batkisso, clasohm, cward, gozen, llim, madko, mfuruta, scoleris, tao, xen-maint, yshao | ||||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||||
| Target Release: | --- | ||||||||||
| Hardware: | All | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | xen-3.0.3-85.el5 | Doc Type: | Bug Fix | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | |||||||||||
| : | 524624 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2009-09-02 10:06:26 UTC | Type: | --- | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 445799, 477162, 524624 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 284061 [details]
information from i386 machine
balloon.txt xend.log xm-info.txt xm-list.txt taken after a failed attempt
on i386 machine
Created attachment 284071 [details]
information from x86_64 machine
balloon.txt xend.log xm-info.txt xm-list.txt
taken after failed attempt on x86_64 machine
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Is there a update (even prerelease) for this yet? Has anyone come up with a work around other than rebooting the HV? Only effects me on fully virtuals as the original comment. Anyway to detect when I'm close to running out of this type of memory since xm list isn't reporting it correctly? *** Bug 251295 has been marked as a duplicate of this bug. *** Well, I was unable to duplicate this problem using Chris' virttest11xen kernel. I have tried to figure it out for some time and this is what happened sometimes when starting up new domains and mixing PV and FV ones too. I finally found them in the kernel-xen package connected to memory allocation but later I got advised to use virttest11xen kernel so I did and since then I was unable to replicate it. Unfortunatelly I don't know what solved this issue... Hi Masaki, please try Chris' xen kernel. You can download it from: http://people.redhat.com/clalance/virttest/ I tested it using this kernel and I was unable to replicate the problem anymore and if you find any problem please tell me steps to replicate the problem on this kernel. Regards, Michal Created attachment 336771 [details] Domain creation 4M memory allocation patch This is a patch for domain memory creation. There was an error about domain memory creation because there was not enough memory allocated it. This patch fixes this issue. It's been tested using this scenario on 8G physical memory machine: 1. xm create RHEL53 maxmem=2500 memory=2500 2. xm create RHEL5FV maxmem=2500 memory=2500 3. virt-install -l http://path/to/install/ -r 2500 -v --nodisks -n RHELtest It was working correctly and before my patch it was not working at all and it returned error: "Cannot allocate memory". Later I tried it using virt-manager and to allocate 2500M of memory and it was working as well. Please try it using my patch... Hi, has any triage been done using my RPMs? Thanks, Michal Fix built into xen-3.0.3-85.el5 Here it is: http://people.redhat.com/jdenemar/xen.el5/ . Could you provide me /var/log/xen/xend.log if it fails again? Anyway could you give me exact steps to reproduce it and provide some information about platform (i386, x86_64, ia64) etc.? I was unable to reproduce it using my patch on x86_64 one. Michal ~~ Attention - RHEL 5.4 Beta Released! ~~ RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner! If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity. Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value. Questions can be posted to this bug or your customer or partner representative. This bug has been verified with xen 3.0.3-91.el5 on RHEL-5.4. Already fixed, set status to VERIFIED. Masaki, Sorry for the late response and followings are my verification steps. If you think anything not appropriate, please do not hesitate correting me. Thanks. Suppose there is totally 4GB memory in the testing host: Steps: 1. balloon up dom0 memory to 3.5GB by "xm mem-set 0 3584"; 2. create one pv domain rhel5u4pv by "xm create rhel5u4pv maxmem=3584 memory=3584", note that the unit is MB and 3584MB equals to 3.5GB; 3. destroy domain created in step 2; 4. repeat step 1 ~3 by picking up other different values for "maxmem" & "memory" above. Result: In step 2, during dom rhel5u4pv creation, because system free memory(equas to 4GB minus dom0 memory approximately) can not satisfy the requirement, dom0 is forced to balloon out some of its memory, it is expected and just the result of this bug fixing. Based on this(dom0 can balloon out part of its memory in a rational scope), I tell that this bug is fixed. Note: It is must to reserve certain minimal memory for dom0 which is defined in xend config file "/etc/xen/xend-config.sxp". Thanks and hope it is what you needed Edward Wang An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-1328.html |
Description of problem: When creating a domain, xend balloons down dom0 memory for memory space of the guest. This mechanism usually works fine, but sometimes creating a domain fails showing "cannot allocate memory", even if dom0 seems to have enough memory for ballooning down. Once this problem happens, I can't create the domain until I balloon down dom0 memory by "xm mem-set" manually. Version-Release number of selected component: RHEL5.1GA-RC xen-3.0.3-41.el5 How reproducible: Rarely. However it seems that the probability is higher if dom0 itself is under high load. Steps to Reproduce: - boot dom0. - create some domains. - reboot dom0 until this problem happens. Actual results: Xen fails in creating the domain. Expected results: The domain will be created properly. Hardware info: x86_64: NEC Express5800/140Rd-4 i386: NEC Express5800/120Rg-1 Business impact: Users may fail in creating domains, and have to run "xm mem-set" manually. Additional info: - We have seen this problem happen only when creating full virtualized guests. Even when this problem happened, I could create a para virtualized guest with the same memory size. - dom0 boot option title Red Hat Enterprise Linux Server (2.6.18-52.el5xen) x86_64 root (hd0,5) kernel /boot/xen.gz-2.6.18-52.el5 crashkernel=128M@32M module /boot/vmlinuz-2.6.18-52.el5xen ro root=LABEL=/1 rhgb quiet module /boot/initrd-2.6.18-52.el5xen.img - When this problem happnes, xend.log looks like: [2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', 'RC_x86_64_full_disk'], ['memory', '2048'], ['maxmem', '2048'], ['vcpus', '2'], ['uuid', '90038a46b51e9a9b7d12ad42eced2a77'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot', 'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']]], ['device', ['vbd', ['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]], ['device', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'], ['type', 'ioemu']]]]) [2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG (XendDomainInfo:306) parseConfig: config is ['vm', ['name', 'RC_x86_64_full_disk'], ['memory', '2048'], ['maxmem', '2048'], ['vcpus', '2'], ['uuid', '90038a46b51e9a9b7d12ad42eced2a77'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot', 'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']]], ['device', ['vbd', ['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]], ['device', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'], ['type', 'ioemu']]]] [2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None, 'start_time': None, 'uuid': '90038a46-b51e-9a9b-7d12-ad42eced2a77', 'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None, 'image': ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['device_model', '/usr/lib64/xen/bin/qemu-dm'], ['vcpus', '2'], ['boot', 'c'], ['acpi', '1'], ['apic', '1'], ['pae', '1'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1'], ['keymap', 'ja']], 'on_poweroff': 'destroy', 'bootloader_args': None, 'cpus': None, 'name': 'RC_x86_64_full_disk', 'backend': [], 'vcpus': 2, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 2048, 'device': [('vbd', ['vbd', ['dev', 'hda:disk'], ['uname', 'phy:/dev/sdb8'], ['mode', 'w']]), ('vif', ['vif', ['mac', '00:16:3e:15:7f:e1'], ['bridge', 'virbr0'], ['type', 'ioemu']])], 'bootloader': None, 'cpu': None, 'maxmem': 2048} [2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG (XendDomainInfo:1349) XendDomainInfo.construct: None [2007-10-25 11:02:29 xend 3807] DEBUG (balloon:133) Balloon: 640 KiB free; 0 to scrub; need 2048; retries: 20. [2007-10-25 11:02:29 xend 3807] DEBUG (balloon:148) Balloon: setting dom0 target to 3807 MiB. [2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG (XendDomainInfo:1078) Setting memory target of domain Domain-0 (0) to 3807 MiB. [2007-10-25 11:02:29 xend 3807] DEBUG (balloon:127) Balloon: 2688 KiB free; need 2048; done. [2007-10-25 11:02:29 xend.XendDomainInfo 3807] ERROR (XendDomainInfo:212) Domain construction failed Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 204, in create vm.construct() File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1369, in construct hvm = hvm) Error: (12, 'Cannot allocate memory') [2007-10-25 11:02:29 xend.XendDomainInfo 3807] DEBUG (XendDomainInfo:1557) XendDomainInfo.destroy: domid=None [2007-10-25 11:02:29 xend 3807] ERROR (SrvBase:88) Request create failed. Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/xen/web/SrvBase.py", line 85, in perform return op_method(op, req) File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 82, in op_create raise XendError("Error creating domain: " + str(ex)) XendError: Error creating domain: (12, 'Cannot allocate memory')