Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1100152

Summary: [scale] Host memory usage is not updated in RHEVM UI after removing VMs
Product: Red Hat Enterprise Virtualization Manager Reporter: Yuri Obshansky <yobshans>
Component: ovirt-engineAssignee: Roy Golan <rgolan>
Status: CLOSED NOTABUG QA Contact: Yuri Obshansky <yobshans>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: dfediuck, gklein, iheim, lpeer, michal.skrivanek, rbalakri, rgolan, Rhev-m-bugs, sherold, yeylon, yobshans
Target Milestone: ---   
Target Release: 3.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: sla
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-30 11:00:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
RHEVM UI snapshot
none
engine.log
none
vdsm.log
none
Engine UI screenshot
none
top screenshot
none
screenshot memory usage 98%
none
screenshot memory usage 3% none

Description Yuri Obshansky 2014-05-22 06:42:36 UTC
Description of problem:
Host memory usage is not updated in RHEVM UI after removing VMs
Show 99% usage when it should be lower (something about 3%)

Version-Release number of selected component (if applicable):
Build - 3.4.0-0.16.rc.el6ev
OS Version: RHEL - 6Server - 6.5.0.1.el6
Kernel Version: 2.6.32 - 431.17.1.el6.x86_64
KVM Version: 0.12.1.2 - 2.415.el6_5.9
LIBVIRT Version: libvirt-0.10.2-29.el6_5.7
VDSM Version: vdsm-4.14.7-0.2.rc.el6ev

How reproducible:
100 % reproducible
My RHEVM has: 1 Data Center, 4 Hosts

Steps to Reproduce:
1. Populated 200 VMs in DC with 4 hosts 
2. Removed 200 VMs
3. Look at RHEVM UI 
one or several hosts memory usage still 99 % 
4. Connect to host using ssh 
5. Run top command
Memory usage is about 3 %  

Actual results:
99% Memory usage
Physical Memory: 64379 MB total, 63735 MB used, 644 MB free

Expected results:
3% Memory usage
Should be such as top shows
Mem:  65924860k total,  2130144k used, 63794716k free,

Additional info:

Comment 1 Yuri Obshansky 2014-05-22 06:44:29 UTC
Created attachment 898223 [details]
RHEVM UI snapshot

Comment 2 Doron Fediuck 2014-06-19 11:43:07 UTC
Can we get the engine and specific vdsm log to see what is being reported?

Comment 3 Yuri Obshansky 2014-06-19 14:36:57 UTC
It's a little bit complicated right now. Environment is under load tests. 
I hope, I'll be able reproduce it and collect log files on next week.

Comment 4 Martin Sivák 2014-07-23 09:21:23 UTC
Yuri, any update?

Comment 5 Yuri Obshansky 2014-08-03 13:36:30 UTC
I'll upload log files during this week.

Comment 6 Yuri Obshansky 2014-09-01 10:48:31 UTC
Created attachment 933300 [details]
engine.log

Comment 7 Yuri Obshansky 2014-09-01 10:52:23 UTC
Created attachment 933302 [details]
vdsm.log

Comment 8 Yuri Obshansky 2014-09-01 10:58:21 UTC
I've reproduced that bug on another environment.
Unfortunately, I cannot overload host with large number of VMs (due the limited number of MAC addresses in our environmnet) to reach 99% CPU.
But, you can see host CPU Utilization is 66% while no one VMs is running on it
and top command shows 0.1 %.
See attached engine.log, vdsm.log, RHEVM UI screenshot, top screeshot.

Comment 9 Yuri Obshansky 2014-09-01 10:59:25 UTC
Created attachment 933303 [details]
Engine UI screenshot

Comment 10 Yuri Obshansky 2014-09-01 10:59:51 UTC
Created attachment 933304 [details]
top screenshot

Comment 11 Roy Golan 2014-11-09 09:19:54 UTC
Yuri pls include the output of

# vdsClient 0 getVdsStats | egrep 'cpuSys|cpuUser'


the sum of both numbers is what reported in the UI.

if the figures don't match we should see what's in the db for vds_statistics column

select cpu_sys,cpu_user from vds where vds_name = 'YOURHOST';

Comment 12 Yuri Obshansky 2014-11-10 07:06:51 UTC
Roy, what build you prefer?
3.4.* ор 3.5.0-0.17.beta.el6ev

Comment 13 Roy Golan 2014-11-10 08:43:59 UTC
(In reply to Yuri Obshansky from comment #12)
> Roy, what build you prefer?
> 3.4.* ор 3.5.0-0.17.beta.el6ev

3.5 unless you got a 3.4 setup and you can quickly get it.

Comment 14 Yuri Obshansky 2014-11-17 16:34:13 UTC
I tried reproduce bug on RHEVM 3.5.0-0.18.beta.el6ev and vdsm-4.16.7.2-1.el6ev
It didn't reproduce. 
I created 90 VMs on host and it's memory usage was 98%
(see uploaded screenshot ui_mem_3.5_99_percent_before.jpg)
VDSM client shows the same number:
[root@host05-rack06 ~]# vdsClient -s 0 getVdsStats | egrep 'memUsed'
	memUsed = '98'
And engine database has the same number also:
engine=# select usage_mem_percent from vds where vds_name = 'HOST-05';
 usage_mem_percent 
-------------------
                98
I removed all VMs and host memory usage was reduced to 3% as expected.
(see uploaded screenshot ui_mem_3.5_99_percent_after.jpg)
VDSM client shows the same number:
[root@host05-rack06 ~]# vdsClient -s 0 getVdsStats | egrep 'memUsed'
	memUsed = '3'
And engine database has the same number also:
engine=# select usage_mem_percent from vds where vds_name = 'HOST-05';
 usage_mem_percent 
-------------------
                 3
So, bug could be closed.

Comment 15 Yuri Obshansky 2014-11-17 16:34:56 UTC
Created attachment 958286 [details]
screenshot memory usage 98%

Comment 16 Yuri Obshansky 2014-11-17 16:35:28 UTC
Created attachment 958287 [details]
screenshot memory usage 3%