Bug 251295
| Summary: | Having trouble with running multiple guests at once | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Gurhan Ozen <gozen> | ||||
| Component: | xen | Assignee: | Michal Novotny <minovotn> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 5.0 | CC: | areis, gren, jburke, qcai, xen-maint | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2009-03-05 16:22:32 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 160885 [details]
an example of free output and xend.log when this happens.
This is bizarre - the logs show its failing in domain_create() call to the HV with ENOMEM. It has even tried to allocate the main RAM for the guest - its merely do the small allocs() for the domain data structures. We explicitly balloon 2 MB of slack to let this work - upstream said 1 MB was worst case need so we should be comfortably OK. Perhaps someplace in domain creation needs RAM from a special memory zone & that's unavailable ? What architecture is host ? (In reply to comment #2) > This is bizarre - the logs show its failing in domain_create() call to the HV > with ENOMEM. It has even tried to allocate the main RAM for the guest - its > merely do the small allocs() for the domain data structures. We explicitly > balloon 2 MB of slack to let this work - upstream said 1 MB was worst case need > so we should be comfortably OK. Perhaps someplace in domain creation needs RAM > from a special memory zone & that's unavailable ? What architecture is host ? > This is an AMD x86_64 host: processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 65 model name : Dual-Core AMD Opteron(tm) Processor 8218 stepping : 2 cpu MHz : 2593.586 cache size : 1024 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy bogomips : 6485.77 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp tm stc This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. This request was previously evaluated by Red Hat Product Management for inclusion in the current Red Hat Enterprise Linux release, but Red Hat was unable to resolve it in time. This request will be reviewed for a future Red Hat Enterprise Linux release. Sounds like a duplicate to me. According to your log you are trying to run a HVM machine and I have found out it happens sometimes for HVM guest when some guests are already running (no matter whether PV or FV ones). Anyway I did some testing using this configuration (all with the same memory and maxmem settings): 1) I ran Fedora 8 PV guest 2) ... and Fedora 8 FV guest 3) Then I tried another FV guest and it raised an error "Cannot allocate memory" but I still had over 4G of free RAM available 4) I tried running some other FV guest - still same error 5) I tried mem-setting (xm mem-set) it to some other amount of memory (decreased by some megs) and tried FV guest again - still no luck 6) I tried PV guest - it was starting and running fine 7) I tried FV guest then - voila, finally the FV guest started up 8) I tried another FV guest - it was starting fine too. Anyway if you have some other experiences just let me know but beware this error doesn't occur every time and sometimes it's very hard to reproduce. *** This bug has been marked as a duplicate of bug 419771 *** |
Description of problem: This is very inconsistent bug, but i ran into it few times. I am intermittently having problem with running multiple guests (regardless of they are paravirt or hvm or mix) at once . Here is some breakdown: [root@amd-tyan-10 ~]# free total used free shared buffers cached Mem: 8121708 4081616 4040092 0 194696 3202796 -/+ buffers/cache: 684124 7437584 Swap: 2096440 0 2096440 [root@amd-tyan-10 ~]# virsh start rhel4u5_pv Domain rhel4u5_pv started [root@amd-tyan-10 ~]# virsh start rhel51_i386_pv Domain rhel51_i386_pv started [root@amd-tyan-10 ~]# virsh list Id Name State ---------------------------------- 0 Domain-0 running 31 rhel4u5_pv blocked 32 rhel51_i386_pv running [root@amd-tyan-10 ~]# free total used free shared buffers cached Mem: 7201792 4092136 3109656 0 196844 3202792 -/+ buffers/cache: 692500 6509292 Swap: 2096440 0 2096440 Got 2 guests running and 3 gigs of memory free. So far so good, but then when I try to start a 3rd guest: [root@amd-tyan-10 ~]# virsh start rhel5_1 libvir: Xen Daemon error : POST operation failed: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") error: Failed to start domain rhel5_1 Gives error about not being able to allocate memory even though there is enough memory. Then, after few minutes, without any further changes in the system: [root@amd-tyan-10 ~]# virsh list Id Name State ---------------------------------- 0 Domain-0 running 31 rhel4u5_pv blocked 32 rhel51_i386_pv blocked [root@amd-tyan-10 ~]# virsh start rhel5_1 Domain rhel5_1 started [root@amd-tyan-10 ~]# virsh start rhel3u9 Domain rhel3u9 started [root@amd-tyan-10 ~]# free total used free shared buffers cached Mem: 6126592 4120932 2005660 0 203948 3202964 -/+ buffers/cache: 714020 5412572 Swap: 2096440 0 2096440 Things are back to normal again. I am stomped as to why this could be happening. It has happened as early as 2nd guest. I will attach xend.log along with the free output to this bug retrieved when I was having an issue. Version-Release number of selected component (if applicable): # rpm -q xen kernel-xen libvirt xen-3.0.3-33.el5 kernel-xen-2.6.18-37.el5 libvirt-0.2.3-5.el5 libvirt-0.2.3-5.el5 How reproducible: This does happen pretty frequently but there is no consistency as to when it could happen. Steps to Reproduce: 1. make sure that you have enough memory available and keep on starting different guests. 2. 3. Actual results: Expected results: Additional info: