This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 213696 - perfromance issue with python+threads+popen
perfromance issue with python+threads+popen
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: James Antill
Brock Organ
Depends On:
  Show dependency treegraph
Reported: 2006-11-02 11:58 EST by Satish Balay
Modified: 2008-05-03 14:30 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-05-03 14:30:43 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
python-test-code (2.19 KB, text/x-python)
2006-11-02 11:58 EST, Satish Balay
no flags Details
work-arround-code (2.26 KB, application/x-python)
2006-11-02 19:37 EST, Satish Balay
no flags Details

  None (edit)
Description Satish Balay 2006-11-02 11:58:49 EST
Description of problem:
Using the combination of threads+popen from python results in code that
is solwer [by a factor of 10 or more]

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. run the attached python code on fc6

[balay@localhost junk]$ python
Actual results:

thread overhead 0.79657793045
combo overhead 43.1583888531
pipe overhead 1.39669799805

Expected results:

All 3 timings above to be similar [or closeby]

Additional info:

- The problem doesn't exist on FC3
- The problem is reproduealbe on Ubuntu 6.10
- The problem is reproduceable by installing native python [tried 2.2.3, 2.3.5,
2.4.4, 2.5 on FC6]

The above things makes me conclude its probaby a glibc problem - not python problem.

Note: if I switch the order of the tests in my code to do popen() tests before
popen()+threads, [combo] - then I don't notice performance degradation in  the
test code.

<modify code>
[balay@localhost junk]$ python
thread overhead 0.750590085983
pipe overhead 1.84653496742
combo overhead 1.59167599678

[The test code is striped from actual code with the problem - so the structure
might be funky - and perhaps not minimal to demonstrate this problem]

I'd apreciate any help in narrowing down the cause of problem.
Comment 1 Satish Balay 2006-11-02 11:58:49 EST
Created attachment 140153 [details]
Comment 2 Jakub Jelinek 2006-11-02 12:57:46 EST
Sorry, can't reproduce this.
python ./
thread overhead 0.401163101196
combo overhead 1.78790402412
pipe overhead 1.42629718781
is what I'm consistently getting.

Anyway, reassigning to python, unless you come up with a non-python reproducer.
Comment 3 Satish Balay 2006-11-02 13:41:51 EST
Its strange that you can't reproduce it. Are you using a multi-proc machine? [My
tests have been on 1-cpu box [p-m 1.6GHz laptop]

Anyway I've made a bit more progress. Looks like in the bad case - the select()
call is looping forever.. Perhaps its a code bug.. I'll keep digging.

thanks for checking on this..
Comment 4 Satish Balay 2006-11-02 19:37:50 EST
Created attachment 140203 [details]

This modifcation to usage of select() within the spawned thread appears to
workarround this issue. Its not clear to me if this is a bug in the code or bad
behavior of select()
Comment 5 Satish Balay 2006-11-02 19:44:14 EST
Should have mentioned  [with the orignal buggy code] on FC3, select() loop gets
executes once [for all the cases I've tried] - but on  FC6 - it gets executed
thousands of times [for eg: 42451, 31316 etc..] causing the bad performance.
Comment 6 Bug Zapper 2008-04-04 00:21:38 EDT
Fedora apologizes that these issues have not been resolved yet. We're
sorry it's taken so long for your bug to be properly triaged and acted
on. We appreciate the time you took to report this issue and want to
make sure no important bugs slip through the cracks.

If you're currently running a version of Fedora Core between 1 and 6,
please note that Fedora no longer maintains these releases. We strongly
encourage you to upgrade to a current Fedora release. In order to
refocus our efforts as a project we are flagging all of the open bugs
for releases which are no longer maintained and closing them.

If this bug is still open against Fedora Core 1 through 6, thirty days
from now, it will be closed 'WONTFIX'. If you can reporduce this bug in
the latest Fedora version, please change to the respective version. If
you are unable to do this, please add a comment to this bug requesting
the change.

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we are following is outlined here:

We will be following the process here: to ensure this
doesn't happen again.

And if you'd like to join the bug triage team to help make things
better, check out
Comment 7 Satish Balay 2008-04-04 00:57:33 EDT
I've just tired the test script on FC8 [same hardware as the previous FC6 test],
and I can't reproduce this problem.

So I guess this issue is fixed in FC8 glibc2.7.2. [Both default python-2.5.1 &
python-2.2.3 give reasonable results.]

This issue can be closed.

asterix:/home/balay/download>rpm -q glibc python
thread overhead 0.70380282402
combo overhead 1.14291810989
pipe overhead 1.12582397461
thread overhead 0.485852956772
combo overhead 0.738920927048
pipe overhead 0.671370983124
Comment 8 Johan Kok 2008-05-03 14:30:43 EDT
(In reply to comment #7)
> This issue can be closed.

Thanks for reporting back. Closing this issue.

Note You need to log in before you can comment on or make changes to this bug.