Bug 452281

Summary: Unable to decrease memory allocation
Product: Red Hat Satellite 5 Reporter: Clifford Perry <cperry>
Component: VirtualizationAssignee: Devan Goodwin <dgoodwin>
Status: CLOSED CURRENTRELEASE QA Contact: wes hayutin <whayutin>
Severity: low Docs Contact:
Priority: low    
Version: 510   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: sat511 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-08-13 19:04:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 429319    

Description Clifford Perry 2008-06-20 17:38:03 UTC
Description of problem:
During Summit labs I hit an issue where trying to decrease the amount of RAM for
a guest failed, due to the host system having only 400MB available. 

From what I was guessing, it seemed that I had:
- 2 gig physical host
- host instance was using ~400mb
- guest 1 was assigned 512MB
- guest 2 was assigned 512MB
- leaving ~600MB used by no one

Try in UI to decrease allocation from 512 to 256 for guest 1 - this throws an
error in UI basically saying not enough memory - I think it said, so Host has
400MB, with guest 1 and 2, I have 1024 MB used, I do not have enough free memory
allocated to change anything, not even zero. 

Sooooo.... maybe the issue is that libvirt API call we used in RHEL 5.0 changed
and so our logic now does not work (since pretty sure it did work last yr when I
did it). 

Thanks,
Cliff. 



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Devan Goodwin 2008-06-24 12:54:02 UTC
I think I've found the problem here, a little tricky but situation is as follows:

Code was written with a bad assumption, that the host's total memory in our
database was static and could be used to determine what settings are valid for
guests. If we try to change memory settings we add up all the *new* guest values
and see if it exceeds what the host has total - about 256mb for the host itself.

This is flawed because I just realized today, allocating guest memory actually
takes away from the host's total memory on the actual box. (i.e. it doesn't
behave like a regular application where there's just less free memory, the
actual total changes) Was mistaken how this behaved.

This wasn't caught in testing because normally when we registered a system it
had no guests running, Satellite got the correct total value and used it
repeatedly. I assume in this scenario the box might have been registered with
guests already running, or perhaps a hardware refresh was scheduled after the
guests were up.

I am not yet sure what to do. 

I do not see any way we can get an accurate count of how much memory a system
has at the outset, once guests have been allocated memory every indication on
the actual host is that it's total memory is now decreased.

Without knowing whether or not a hardware refresh has taken place, it's going to
be very hard to try to guess whether the guest memory settings we're attempting
to apply exceed what is available. We could assume a hardware refresh every time
we modify memory, but if increasing guest memory the guests actually have to be
restarted, in which case this wouldn't even work. (and we probably can't assume
a restart)

It may be best to start automatically allowing decreases in memory to get us
part way there. (but this still is just a workaround)

Still pondering what to do...

Comment 2 Devan Goodwin 2008-06-24 13:43:44 UTC
The options I've been able to come up with:

1. Make hardware refreshes store the hosts memory as what Linux reports plus the
memory allocated to all running guests.

GOOD: Logic remains the same in most all places.
BAD: We'd now be showing system as having X amount of memory when it really has
X-Y available to the operating system.

2. Change guest memory logic and assume a hardware refresh any time we change
virt guest memory.

BAD: Takes time before hardware refresh even happens without osad, rapid memory
commands would be no better. If guest memory is increasing past what it was
booted with, the guest must be rebooted before the setting even takes effect, in
which case the hardware refresh wouldn't even get the right number.

3. Change guest memory logic, and error message to just a warning and link to
hardware refresh, but allow the settings change to proceed.

BAD: If a hardware refresh wasn't performed after a previous guest creation or
memory increase, the warning wouldn't even trigger and yet the request would go
through and have no effect leading to a confused user. We may as well not be
checking at all, leading me to the fourth possibility.

4. Stop checking memory and just warn the user that they must verify the system
has an appropriate amount of memory free.

I am not seeing a clean way out of this, please let me know if anyone has
suggestions. I am leaning towards option #4, as there doesn't seem to be a way
to reliably check this, rather than hack around it and potentially display the
wrong information, it may be better to just remove it.

Comment 4 Devan Goodwin 2008-06-24 16:01:08 UTC
#4 looks like the way to go. Whenever you adjust memory settings you will be
prompted with the following two warnings:

"YOURGUEST must be shutdown for max memory settings to take effect." - This is
changed from "rebooted" as this does not affect max memory settings. Note that
current memory allocation can be decreased without a reboot.

"Ensure the host system has sufficient free memory if increasing guest memory
allocation. Otherwise the guest may refuse to boot the next time it is started."
- Tested this, if you assign an absurdly large amount of memory to your guest
and attempt to start it the start will fail (and take a bit of time realizing it
must fail), during which virsh and xm commands act a little strangely, but the
start does eventually error out, we get a failed action in Satellite, guest is
not running, and the system returns to functioning as normal. The memory setting
can be decreased from the satellite UI and the start attempted again.

Steps to reproduce initial issue (note I don't recommend using these to verify
the bug as we've removed this check now):

1. Create or allocate a couple guests totalling over half the system's memory.
2. Schedule a hardware refresh for the host.
3. Attempt to decrease the memory allocation for either of the guests. (but the
new total guest allocation must be more than the host currently has free (+256
reserved mb approximately)

Steps to test this fix:

1. Change a guests memory allocation.
2. Ensure the two warning messages above appear.

If you then shutdown the guest and start it up again (note: NOT reboot) virsh
dominfo should return the proper max memory setting.

If you specify too much memory, the guest should refuse to start and an error
action should appear in the event history for the host.



Comment 5 Devan Goodwin 2008-06-24 17:02:16 UTC
Modified in 5.1:

Sending       
code/src/com/redhat/rhn/frontend/strings/java/StringResource_en_US.xml

Sending        code/src/com/redhat/rhn/manager/system/SystemManager.java

Sending        code/src/com/redhat/rhn/manager/system/test/SystemManagerTest.java

Transmitting file data ...

Committed revision 174405.


Ported to git in commit 7ff741aee3c8f2cf2b323650bd3d74048114f849.


Comment 6 Jeff Browning 2008-07-02 21:19:59 UTC
Verified. Sat no longer checks host memory. Message stating that the user must
make sure that the setting is workable is displayed as stated in the ticket above.

Comment 7 Steve Salevan 2008-07-21 21:29:10 UTC
Moving to RELEASE_PENDING.

Comment 8 Brandon Perkins 2008-08-13 19:04:20 UTC
5.1.1 Satellite is now GA, bugs Closed for Current Release.

Comment 9 Brandon Perkins 2008-08-13 19:09:21 UTC
5.1.1 Satellite is now GA, bugs Closed for Current Release.