Bug 967573 - PRD33 - alert on VMs not respecting balloon
Summary: PRD33 - alert on VMs not respecting balloon
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: RFEs
Version: 3.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.3.0
Assignee: Doron Fediuck
QA Contact: Lukas Svaty
URL:
Whiteboard: sla
Depends On:
Blocks: 975626 978136 978137 978139 1019470
TreeView+ depends on / blocked
 
Reported: 2013-05-27 13:31 UTC by Itamar Heim
Modified: 2016-02-10 20:15 UTC (History)
11 users (show)

Fixed In Version: IS9
Doc Type: Release Note
Doc Text:
When ballooning is enabled for a cluster, appropriate messages now appear in the Events tab of the administration portal when the following errors occur: * The balloon is requested but the balloon driver on the virtual machine is not responding. * The balloon is inflated, but the Memory Overcommit Manager (MOM) has lost control over the balloon device for various reasons such as failure of the guest agent.
Clone Of:
: 975626 (view as bug list)
Environment:
Last Closed: 2014-01-21 17:23:46 UTC
oVirt Team: SLA
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2014:0038 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Virtualization Manager 3.3.0 update 2014-01-21 22:03:06 UTC
oVirt gerrit 17486 0 None None None Never

Description Itamar Heim 2013-05-27 13:31:05 UTC
related to bug 967572 for balloon support.
this is for engine to report on guests not respecting ballooning requests (which once cgroups for memory are supported, will get swapped out for their unreserved memory)

Comment 2 Doron Fediuck 2013-07-01 16:15:18 UTC
For this BZ the engine will produce an event in the event log.
Going forward (in next versions) we'll provide additional information on memory usage and consumption of hosts and VMs, so balloon usage will be available there as well.
For now, you should look for an event on a VM which does not cooperate with the ballooning requests.

Comment 4 Doron Fediuck 2013-08-04 09:10:23 UTC
Deatils on how to test;

Setting up
============

Engine:
Enable ballooning in cluster optimization

Host:
--
up-to-date vdsm and mom
swapoff -a
set rpc-port: 8080 in /etc/vdsm/mom.conf
restart vdsm

monitor the host (and its VMs) using:
/var/log/vdsm/mom.log
vdsCommand -s 0 getAllVmStats
python -c "import xmlrpclib; from pprint import pprint; mom = xmlrpclib.ServerProxy('http://localhost:8080'); pprint(mom.getStatistics())"

VMs (you need two):
--
RAM: Host's RAM / 2
Resource allocation/Guaranteed memory: 512MB (or some other small number)

yum install ovirt-guest-agent
systemctl start ovirt-guest-agent
systemctl enable ovirt-guest-agent
mount -t tmpfs -o size=<almost all free RAM in Bytes> none /mnt
swapoff -a

Testing:
============
Once you have two VMs ready issue the next line in both:

cat /dev/urandom >/mnt/data

Then you can watch the host and once the the host's free memory drops bellow about 20% you should see ballooning commands like "mom.Controllers.Balloon - INFO - Ballooning guest:half1 from 2096400 to 1991580" being logged to mom.log.

If the agent is down you will see "mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-guest-agent is not active".
If everything is ok on the VMs side you should see "GuestMonitor-<VM name> is ready". 

At that time you should stop the cat commands and start playing with stopping the guest-agent or rmmod-ing virtio_balloon module.

Be aware of the fact you are filling the guest's memory and it has no swap. When the kernel needs memory and there is no free space in it, it first kills the ovirt-guest-agent and then panics... That is a feature though :)

Comment 6 Lukas Svaty 2013-09-03 12:05:00 UTC
verified on is12
info and warning available in /var/log/vdsm/mom.log

after moment when there is no free memory for kernel
guest-agent is killled afterwards kernel panics (as specified in comment#4):
#service ovirt-guest-agent status
ovirt-guest-agent dead but pid file exists
	
appropriate message apears in webadmin portal in events tab:
"The Balloon driver on VM vm1 on host 10.34.62.203 is requested but unavailable."

Maybe this would be better to move in the future to Alerts tab to warn user to enable balloon in VM.

Comment 8 errata-xmlrpc 2014-01-21 17:23:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html


Note You need to log in before you can comment on or make changes to this bug.