Bug 967573 - PRD33 - alert on VMs not respecting balloon
PRD33 - alert on VMs not respecting balloon
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: RFEs (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.3.0
Assigned To: Doron Fediuck
Lukas Svaty
: FutureFeature
Depends On:
Blocks: 975626 978136 978137 978139 1019470
  Show dependency treegraph
Reported: 2013-05-27 09:31 EDT by Itamar Heim
Modified: 2016-02-10 15:15 EST (History)
11 users (show)

See Also:
Fixed In Version: IS9
Doc Type: Release Note
Doc Text:
When ballooning is enabled for a cluster, appropriate messages now appear in the Events tab of the administration portal when the following errors occur: * The balloon is requested but the balloon driver on the virtual machine is not responding. * The balloon is inflated, but the Memory Overcommit Manager (MOM) has lost control over the balloon device for various reasons such as failure of the guest agent.
Story Points: ---
Clone Of:
: 975626 (view as bug list)
Last Closed: 2014-01-21 12:23:46 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: SLA
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 17486 None None None Never

  None (edit)
Description Itamar Heim 2013-05-27 09:31:05 EDT
related to bug 967572 for balloon support.
this is for engine to report on guests not respecting ballooning requests (which once cgroups for memory are supported, will get swapped out for their unreserved memory)
Comment 2 Doron Fediuck 2013-07-01 12:15:18 EDT
For this BZ the engine will produce an event in the event log.
Going forward (in next versions) we'll provide additional information on memory usage and consumption of hosts and VMs, so balloon usage will be available there as well.
For now, you should look for an event on a VM which does not cooperate with the ballooning requests.
Comment 4 Doron Fediuck 2013-08-04 05:10:23 EDT
Deatils on how to test;

Setting up

Enable ballooning in cluster optimization

up-to-date vdsm and mom
swapoff -a
set rpc-port: 8080 in /etc/vdsm/mom.conf
restart vdsm

monitor the host (and its VMs) using:
vdsCommand -s 0 getAllVmStats
python -c "import xmlrpclib; from pprint import pprint; mom = xmlrpclib.ServerProxy('http://localhost:8080'); pprint(mom.getStatistics())"

VMs (you need two):
RAM: Host's RAM / 2
Resource allocation/Guaranteed memory: 512MB (or some other small number)

yum install ovirt-guest-agent
systemctl start ovirt-guest-agent
systemctl enable ovirt-guest-agent
mount -t tmpfs -o size=<almost all free RAM in Bytes> none /mnt
swapoff -a

Once you have two VMs ready issue the next line in both:

cat /dev/urandom >/mnt/data

Then you can watch the host and once the the host's free memory drops bellow about 20% you should see ballooning commands like "mom.Controllers.Balloon - INFO - Ballooning guest:half1 from 2096400 to 1991580" being logged to mom.log.

If the agent is down you will see "mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-guest-agent is not active".
If everything is ok on the VMs side you should see "GuestMonitor-<VM name> is ready". 

At that time you should stop the cat commands and start playing with stopping the guest-agent or rmmod-ing virtio_balloon module.

Be aware of the fact you are filling the guest's memory and it has no swap. When the kernel needs memory and there is no free space in it, it first kills the ovirt-guest-agent and then panics... That is a feature though :)
Comment 6 Lukas Svaty 2013-09-03 08:05:00 EDT
verified on is12
info and warning available in /var/log/vdsm/mom.log

after moment when there is no free memory for kernel
guest-agent is killled afterwards kernel panics (as specified in comment#4):
#service ovirt-guest-agent status
ovirt-guest-agent dead but pid file exists
appropriate message apears in webadmin portal in events tab:
"The Balloon driver on VM vm1 on host is requested but unavailable."

Maybe this would be better to move in the future to Alerts tab to warn user to enable balloon in VM.
Comment 8 errata-xmlrpc 2014-01-21 12:23:46 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.