Bug 251295 - Having trouble with running multiple guests at once
Having trouble with running multiple guests at once
Status: CLOSED DUPLICATE of bug 419771
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.0
All Linux
low Severity low
: ---
: ---
Assigned To: Michal Novotny
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-08-08 03:41 EDT by Gurhan Ozen
Modified: 2014-02-02 17:36 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-03-05 11:22:32 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
an example of free output and xend.log when this happens. (3.95 KB, text/plain)
2007-08-08 03:41 EDT, Gurhan Ozen
no flags Details

  None (edit)
Description Gurhan Ozen 2007-08-08 03:41:41 EDT
Description of problem:
  This is very inconsistent bug, but i ran into it few times. I am
intermittently having problem with running multiple guests (regardless of they
are paravirt or hvm or mix) at once . Here is some breakdown:



[root@amd-tyan-10 ~]# free
             total       used       free     shared    buffers     cached
Mem:       8121708    4081616    4040092          0     194696    3202796
-/+ buffers/cache:     684124    7437584
Swap:      2096440          0    2096440
[root@amd-tyan-10 ~]# virsh start rhel4u5_pv
Domain rhel4u5_pv started

[root@amd-tyan-10 ~]# virsh start rhel51_i386_pv
Domain rhel51_i386_pv started

[root@amd-tyan-10 ~]# virsh list
 Id Name                 State
----------------------------------
  0 Domain-0             running
 31 rhel4u5_pv           blocked
 32 rhel51_i386_pv       running

[root@amd-tyan-10 ~]# free
             total       used       free     shared    buffers     cached
Mem:       7201792    4092136    3109656          0     196844    3202792
-/+ buffers/cache:     692500    6509292
Swap:      2096440          0    2096440


  Got 2 guests running and 3 gigs of memory free. So far so good, but then when
I try to start a 3rd guest:

[root@amd-tyan-10 ~]# virsh start rhel5_1
libvir: Xen Daemon error : POST operation failed: (xend.err "Error creating
domain: (12, 'Cannot allocate memory')")
error: Failed to start domain rhel5_1


  Gives error about not being able to allocate memory even though there is
enough memory. Then, after few minutes, without any further changes in the system:


[root@amd-tyan-10 ~]# virsh list
 Id Name                 State
----------------------------------
  0 Domain-0             running
 31 rhel4u5_pv           blocked
 32 rhel51_i386_pv       blocked

[root@amd-tyan-10 ~]# virsh start rhel5_1
Domain rhel5_1 started

[root@amd-tyan-10 ~]# virsh start rhel3u9
Domain rhel3u9 started

[root@amd-tyan-10 ~]# free
             total       used       free     shared    buffers     cached
Mem:       6126592    4120932    2005660          0     203948    3202964
-/+ buffers/cache:     714020    5412572
Swap:      2096440          0    2096440


  Things are back to normal again. I am stomped as to why this could be
happening. It has happened as early as 2nd guest. I will attach xend.log along
with the free output to this bug retrieved when I was having an issue.

Version-Release number of selected component (if applicable):

# rpm -q xen kernel-xen libvirt
xen-3.0.3-33.el5
kernel-xen-2.6.18-37.el5
libvirt-0.2.3-5.el5
libvirt-0.2.3-5.el5

How reproducible:
This does happen pretty frequently but there is no consistency as to when it
could happen.

Steps to Reproduce:
1. make sure that you have enough memory available and keep on starting
different guests.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Gurhan Ozen 2007-08-08 03:41:41 EDT
Created attachment 160885 [details]
an example of free output and xend.log when this happens.
Comment 2 Daniel Berrange 2007-08-08 13:54:48 EDT
This is bizarre - the logs show its failing in domain_create() call to the HV
with ENOMEM. It has even tried to allocate the main RAM for the guest - its
merely do the small allocs() for the domain data structures. We explicitly
balloon 2 MB of slack to let this work - upstream said 1 MB was worst case need
so we should be comfortably OK. Perhaps someplace in domain creation needs RAM
from a special memory zone & that's unavailable ? What architecture is host ?
Comment 3 Gurhan Ozen 2007-08-14 14:06:58 EDT
(In reply to comment #2)
> This is bizarre - the logs show its failing in domain_create() call to the HV
> with ENOMEM. It has even tried to allocate the main RAM for the guest - its
> merely do the small allocs() for the domain data structures. We explicitly
> balloon 2 MB of slack to let this work - upstream said 1 MB was worst case need
> so we should be comfortably OK. Perhaps someplace in domain creation needs RAM
> from a special memory zone & that's unavailable ? What architecture is host ?
> 

  This is an AMD x86_64 host:
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 65
model name      : Dual-Core AMD Opteron(tm) Processor 8218
stepping        : 2
cpu MHz         : 2593.586
cache size      : 1024 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush
mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cx16
lahf_lm cmp_legacy svm cr8_legacy
bogomips        : 6485.77
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
Comment 5 RHEL Product and Program Management 2007-10-15 23:50:06 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 6 RHEL Product and Program Management 2008-03-11 15:43:20 EDT
This request was previously evaluated by Red Hat Product Management
for inclusion in the current Red Hat Enterprise Linux release, but
Red Hat was unable to resolve it in time.  This request will be
reviewed for a future Red Hat Enterprise Linux release.
Comment 8 Michal Novotny 2009-03-05 11:22:32 EST
Sounds like a duplicate to me. According to your log you are trying to run a HVM machine and I have found out it happens sometimes for HVM guest when some guests are already running (no matter whether PV or FV ones). Anyway I did some  testing using this configuration (all with the same memory and maxmem settings):

1) I ran Fedora 8 PV guest
2) ... and Fedora 8 FV guest
3) Then I tried another FV guest and it raised an error "Cannot allocate memory" but I still had over 4G of free RAM available
4) I tried running some other FV guest - still same error
5) I tried mem-setting (xm mem-set) it to some other amount of memory (decreased by some megs) and tried FV guest again - still no luck
6) I tried PV guest - it was starting and running fine
7) I tried FV guest then - voila, finally the FV guest started up
8) I tried another FV guest - it was starting fine too.

Anyway if you have some other experiences just let me know but beware this error doesn't occur every time and sometimes it's very hard to reproduce.

*** This bug has been marked as a duplicate of bug 419771 ***

Note You need to log in before you can comment on or make changes to this bug.