Created attachment 424085 [details]
memory graph from munin
Description of problem:
Since we updated to rhel6 snap6 pes(simulation of production environment) started to consume memory. Seems like some bug in python (memory leak??). This bug is independent on type of test we run in pes. Machines which run on rhel5.5 or rhel6 snap 4 runned without problems for months in case of snap4 weeks (after that we updated on snap6) but snap6 machine consumes memory within one day. We are not sure what causes this memory consumption.
Version-Release number of selected component (if applicable):
actual version of pes in repository
on snap 6 allways, install pes and run tests and wait
Steps to Reproduce:
1.install rhel6 snap6
3.run tests in pes and wait
memory runs out
no memory leaks
memory usage from munin
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release. Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release. This request is not yet committed for
The reason why we thing it is a python bug is simple. Pes is written in python and it tests serveral services including scp, apache, ldap, etc. When the pes is stopped memory is not leaking. I tried to switch off some test to find out if the leak is in tests or not. But it seems to me that memory is leaking independently on tests. So from my point of view the problem should be in python. But if according to you there are nearly no changes in the python at all we will have to look elsewhere.
(In reply to comment #5)
> The reason why we thing it is a python bug is simple. Pes is written in python
> and it tests serveral services including scp, apache, ldap, etc. When the pes
> is stopped memory is not leaking. I tried to switch off some test to find out
> if the leak is in tests or not. But it seems to me that memory is leaking
> independently on tests. [snip]
What happens if you switch off all of the tests? Is it possible to isolate a test that causes the memory behavior you describe?
Can you identify which processes are leaking memory?
> [snip] So from my point of view the problem should be in
> python. But if according to you there are nearly no changes in the python at
> all we will have to look elsewhere.
I'm don't see anything here to suggest a memory leak in Python. Do you have a breakdown of memory usage by process (e.g. "top")? Does a python process appear to have excessive memory usage? If so, we can look into that, but as it stands, this bug needs much more information before I can meaningfully investigate it.
I thought of kernel problem as well but python was my first idea. Memory leaks are independent of test that are on. If whole pes is switched off the "leaks" disappear but that is logical because the whole machine does nothing basically. If I use htop thenthe result doesn't show memory consumation caused by user processes so the problem will probably be in the kernel.
(In reply to comment #8)
> If I use htop thenthe result doesn't show memory consumation caused by user
> processes so the problem will probably be in the kernel.
If I'm reading the above correctly, it sounds like user-space processes aren't showing memory leaks, hence this doesn't sound like a python problem (as a user-space process).
Reassigning to "kernel".
Yes you are right. Reassign to kernel.
After the upgrade to rhel6.0 -beta2-5.0 everything seems to be ok. So it was probably some kind of kernel bug.
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.
** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **