Red Hat Bugzilla – Bug 213696
perfromance issue with python+threads+popen
Last modified: 2008-05-03 14:30:43 EDT
Description of problem:
Using the combination of threads+popen from python results in code that
is solwer [by a factor of 10 or more]
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. run the attached python code on fc6
[balay@localhost junk]$ python th.py
thread overhead 0.79657793045
combo overhead 43.1583888531
pipe overhead 1.39669799805
All 3 timings above to be similar [or closeby]
- The problem doesn't exist on FC3
- The problem is reproduealbe on Ubuntu 6.10
- The problem is reproduceable by installing native python [tried 2.2.3, 2.3.5,
2.4.4, 2.5 on FC6]
The above things makes me conclude its probaby a glibc problem - not python problem.
Note: if I switch the order of the tests in my code to do popen() tests before
popen()+threads, [combo] - then I don't notice performance degradation in the
[balay@localhost junk]$ python th.py
thread overhead 0.750590085983
pipe overhead 1.84653496742
combo overhead 1.59167599678
[The test code is striped from actual code with the problem - so the structure
might be funky - and perhaps not minimal to demonstrate this problem]
I'd apreciate any help in narrowing down the cause of problem.
Created attachment 140153 [details]
Sorry, can't reproduce this.
thread overhead 0.401163101196
combo overhead 1.78790402412
pipe overhead 1.42629718781
is what I'm consistently getting.
Anyway, reassigning to python, unless you come up with a non-python reproducer.
Its strange that you can't reproduce it. Are you using a multi-proc machine? [My
tests have been on 1-cpu box [p-m 1.6GHz laptop]
Anyway I've made a bit more progress. Looks like in the bad case - the select()
call is looping forever.. Perhaps its a code bug.. I'll keep digging.
thanks for checking on this..
Created attachment 140203 [details]
This modifcation to usage of select() within the spawned thread appears to
workarround this issue. Its not clear to me if this is a bug in the code or bad
behavior of select()
Should have mentioned [with the orignal buggy code] on FC3, select() loop gets
executes once [for all the cases I've tried] - but on FC6 - it gets executed
thousands of times [for eg: 42451, 31316 etc..] causing the bad performance.
Fedora apologizes that these issues have not been resolved yet. We're
sorry it's taken so long for your bug to be properly triaged and acted
on. We appreciate the time you took to report this issue and want to
make sure no important bugs slip through the cracks.
If you're currently running a version of Fedora Core between 1 and 6,
please note that Fedora no longer maintains these releases. We strongly
encourage you to upgrade to a current Fedora release. In order to
refocus our efforts as a project we are flagging all of the open bugs
for releases which are no longer maintained and closing them.
If this bug is still open against Fedora Core 1 through 6, thirty days
from now, it will be closed 'WONTFIX'. If you can reporduce this bug in
the latest Fedora version, please change to the respective version. If
you are unable to do this, please add a comment to this bug requesting
Thanks for your help, and we apologize again that we haven't handled
these issues to this point.
The process we are following is outlined here:
We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.
And if you'd like to join the bug triage team to help make things
better, check out http://fedoraproject.org/wiki/BugZappers
I've just tired the test script on FC8 [same hardware as the previous FC6 test],
and I can't reproduce this problem.
So I guess this issue is fixed in FC8 glibc2.7.2. [Both default python-2.5.1 &
python-2.2.3 give reasonable results.]
This issue can be closed.
asterix:/home/balay/download>rpm -q glibc python
thread overhead 0.70380282402
combo overhead 1.14291810989
pipe overhead 1.12582397461
thread overhead 0.485852956772
combo overhead 0.738920927048
pipe overhead 0.671370983124
(In reply to comment #7)
> This issue can be closed.
Thanks for reporting back. Closing this issue.