Bug 604043 - python script PES results in memory consumption
Summary: python script PES results in memory consumption
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
(Show other bugs)
Version: 6.0
Hardware: x86_64 Linux
high
urgent
Target Milestone: rc
: ---
Assignee: Red Hat Kernel Manager
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-06-15 09:21 UTC by Ludek Dolihal
Modified: 2010-08-31 07:10 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-31 07:10:58 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
memory graph from munin (32.54 KB, image/png)
2010-06-15 09:21 UTC, Ludek Dolihal
no flags Details

Description Ludek Dolihal 2010-06-15 09:21:47 UTC
Created attachment 424085 [details]
memory graph from munin

Description of problem:
Since we updated to rhel6 snap6 pes(simulation of production environment) started to consume memory. Seems like some bug in python (memory leak??). This bug is independent on type of test we run in pes. Machines which run on rhel5.5 or rhel6 snap 4 runned without problems for months in case of snap4 weeks (after that we updated on snap6) but snap6 machine consumes memory within one day. We are not sure what causes this memory consumption. 

Version-Release number of selected component (if applicable):
rhel6 snap6
python 2.6.2
actual version of pes in repository


How reproducible:
on snap 6 allways, install pes and run tests and wait

Steps to Reproduce:
1.install rhel6 snap6
2.install pes
3.run tests in pes and wait
  
Actual results:
memory runs out

Expected results:
no memory leaks

Additional info:
memory usage from munin

Comment 2 RHEL Product and Program Management 2010-06-15 09:33:14 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 5 Ludek Dolihal 2010-06-23 09:47:55 UTC
The reason why we thing it is a python bug is simple. Pes is written in python and it tests serveral services including scp, apache, ldap, etc. When the pes is stopped memory is not leaking. I tried to switch off some test to find out if the leak is in tests or not. But it seems to me that memory is leaking independently on tests. So from my point of view the problem should be in python. But if according to you there are nearly no changes in the python at all we will have to look elsewhere.

Comment 6 Dave Malcolm 2010-06-23 13:14:11 UTC
(In reply to comment #5)
> The reason why we thing it is a python bug is simple. Pes is written in python
> and it tests serveral services including scp, apache, ldap, etc. When the pes
> is stopped memory is not leaking. I tried to switch off some test to find out
> if the leak is in tests or not. But it seems to me that memory is leaking
> independently on tests. [snip]
What happens if you switch off all of the tests?  Is it possible to isolate a test that causes the memory behavior you describe?

Can you identify which processes are leaking memory?

> [snip] So from my point of view the problem should be in 
> python. But if according to you there are nearly no changes in the python at
> all we will have to look elsewhere.    

I'm don't see anything here to suggest a memory leak in Python.  Do you have a breakdown of memory usage by process (e.g. "top")?  Does a python process appear to have excessive memory usage?  If so, we can look into that, but as it stands, this bug needs much more information before I can meaningfully investigate it.

Comment 8 Ludek Dolihal 2010-06-23 15:29:04 UTC
I thought of kernel problem as well but python was my first idea. Memory leaks are independent of test that are on. If whole pes is switched off the "leaks" disappear but that is logical because the whole machine does nothing basically. If I use htop thenthe result doesn't show memory consumation caused by user processes so the problem will probably be in the kernel.

Comment 9 Dave Malcolm 2010-06-23 18:11:43 UTC
(In reply to comment #8)
[snip]
> If I use htop thenthe result doesn't show memory consumation caused by user
> processes so the problem will probably be in the kernel.    
If I'm reading the above correctly, it sounds like user-space processes aren't showing memory leaks, hence this doesn't sound like a python problem (as a user-space process).

Reassigning to "kernel".

Comment 10 Ludek Dolihal 2010-06-24 06:34:42 UTC
Yes you are right. Reassign to kernel.

Comment 11 Ludek Dolihal 2010-06-29 10:09:50 UTC
After the upgrade to rhel6.0 -beta2-5.0 everything seems to be ok. So it was probably some kind of kernel bug.

Comment 13 RHEL Product and Program Management 2010-07-15 15:17:20 UTC
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **


Note You need to log in before you can comment on or make changes to this bug.