Red Hat Bugzilla – Bug 213696
perfromance issue with python+threads+popen
Last modified: 2008-05-03 14:30:43 EDT
Description of problem: Using the combination of threads+popen from python results in code that is solwer [by a factor of 10 or more] Version-Release number of selected component (if applicable): glibc-2.5-3 python-2.4.3-18.fc6 How reproducible: Always Steps to Reproduce: 1. run the attached python code on fc6 [balay@localhost junk]$ python th.py Actual results: thread overhead 0.79657793045 combo overhead 43.1583888531 pipe overhead 1.39669799805 Expected results: All 3 timings above to be similar [or closeby] Additional info: - The problem doesn't exist on FC3 - The problem is reproduealbe on Ubuntu 6.10 - The problem is reproduceable by installing native python [tried 2.2.3, 2.3.5, 2.4.4, 2.5 on FC6] The above things makes me conclude its probaby a glibc problem - not python problem. Note: if I switch the order of the tests in my code to do popen() tests before popen()+threads, [combo] - then I don't notice performance degradation in the test code. <modify code> [balay@localhost junk]$ python th.py thread overhead 0.750590085983 pipe overhead 1.84653496742 combo overhead 1.59167599678 [The test code is striped from actual code with the problem - so the structure might be funky - and perhaps not minimal to demonstrate this problem] I'd apreciate any help in narrowing down the cause of problem.
Created attachment 140153 [details] python-test-code
Sorry, can't reproduce this. python ./th.py thread overhead 0.401163101196 combo overhead 1.78790402412 pipe overhead 1.42629718781 is what I'm consistently getting. Anyway, reassigning to python, unless you come up with a non-python reproducer.
Its strange that you can't reproduce it. Are you using a multi-proc machine? [My tests have been on 1-cpu box [p-m 1.6GHz laptop] Anyway I've made a bit more progress. Looks like in the bad case - the select() call is looping forever.. Perhaps its a code bug.. I'll keep digging. thanks for checking on this..
Created attachment 140203 [details] work-arround-code This modifcation to usage of select() within the spawned thread appears to workarround this issue. Its not clear to me if this is a bug in the code or bad behavior of select()
Should have mentioned [with the orignal buggy code] on FC3, select() loop gets executes once [for all the cases I've tried] - but on FC6 - it gets executed thousands of times [for eg: 42451, 31316 etc..] causing the bad performance.
Fedora apologizes that these issues have not been resolved yet. We're sorry it's taken so long for your bug to be properly triaged and acted on. We appreciate the time you took to report this issue and want to make sure no important bugs slip through the cracks. If you're currently running a version of Fedora Core between 1 and 6, please note that Fedora no longer maintains these releases. We strongly encourage you to upgrade to a current Fedora release. In order to refocus our efforts as a project we are flagging all of the open bugs for releases which are no longer maintained and closing them. http://fedoraproject.org/wiki/LifeCycle/EOL If this bug is still open against Fedora Core 1 through 6, thirty days from now, it will be closed 'WONTFIX'. If you can reporduce this bug in the latest Fedora version, please change to the respective version. If you are unable to do this, please add a comment to this bug requesting the change. Thanks for your help, and we apologize again that we haven't handled these issues to this point. The process we are following is outlined here: http://fedoraproject.org/wiki/BugZappers/F9CleanUp We will be following the process here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this doesn't happen again. And if you'd like to join the bug triage team to help make things better, check out http://fedoraproject.org/wiki/BugZappers
I've just tired the test script on FC8 [same hardware as the previous FC6 test], and I can't reproduce this problem. So I guess this issue is fixed in FC8 glibc2.7.2. [Both default python-2.5.1 & python-2.2.3 give reasonable results.] This issue can be closed. asterix:/home/balay/download>rpm -q glibc python glibc-2.7-2 python-2.5.1-15.fc8 asterix:/home/balay/download>python th.py thread overhead 0.70380282402 combo overhead 1.14291810989 pipe overhead 1.12582397461 asterix:/home/balay/download>/home/balay/soft/python-2.2.3/bin/python th.py thread overhead 0.485852956772 combo overhead 0.738920927048 pipe overhead 0.671370983124 asterix:/home/balay/download>
(In reply to comment #7) > This issue can be closed. Thanks for reporting back. Closing this issue.