Bug 667431 - Python Popen.communicate is unable to collect stderr, stdout and returncode for a short lived processes
Python Popen.communicate is unable to collect stderr, stdout and returncode f...
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: python (Show other bugs)
Unspecified Unspecified
low Severity medium
: rc
: ---
Assigned To: Dave Malcolm
Petr Šplíchal
: Reopened
Depends On:
Blocks: 967590
  Show dependency treegraph
Reported: 2011-01-05 10:33 EST by Federico Simoncelli
Modified: 2016-05-31 21:40 EDT (History)
8 users (show)

See Also:
Fixed In Version: python-2.6.6-22.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 734841 967590 (view as bug list)
Last Closed: 2011-12-06 05:23:57 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
test_shortprocess.py (261 bytes, text/x-python)
2011-01-05 10:33 EST, Federico Simoncelli
no flags Details
subprocess-sigpipe-el6.patch (1020 bytes, patch)
2011-01-05 10:35 EST, Federico Simoncelli
no flags Details | Diff
subprocess-sigpipe-fc14.patch (1.64 KB, patch)
2011-01-05 10:36 EST, Federico Simoncelli
no flags Details | Diff
Backport of the upstream patch (9.75 KB, patch)
2011-04-07 15:47 EDT, Dave Malcolm
no flags Details | Diff

External Trackers
Tracker ID Priority Status Summary Last Updated
Python 10963 None None None Never

  None (edit)
Description Federico Simoncelli 2011-01-05 10:33:54 EST
Created attachment 471885 [details]

Description of problem:
Using Popen.communicate over a short-lived process results in a broken pipe exception (errno 32) which will prevent to collect the stderr, stdout and returncode of the dead process.

Version-Release number of selected component (if applicable):
# rpm -qf /usr/lib64/python2.6/subprocess.py

How reproducible:
$ python test_shortprocess.py

Actual results:
Traceback (most recent call last):
  File "test_shortprocess.py", line 7, in <module>
    out, err = p.communicate(2**20 * '\0')
  File "/usr/lib64/python2.7/subprocess.py", line 740, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1256, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1319, in _communicate_with_poll
    input_offset += os.write(fd, chunk)
OSError: [Errno 32] Broken pipe

Expected results:
$ python test_process-fix.py 
dd: unrecognized operand `option=crash'
Try `dd --help' for more information.
Comment 1 Federico Simoncelli 2011-01-05 10:35:17 EST
Created attachment 471886 [details]

Proposed patch for el6.
Comment 2 Federico Simoncelli 2011-01-05 10:36:06 EST
Created attachment 471887 [details]

Proposed patch for fc14.
Comment 4 Dave Malcolm 2011-01-20 15:30:21 EST
Thanks for filing this bug.

I've filed it within the upstream python tracker as:

The patch I've attached there slightly reworks the fix, and adds a unit test.

However, I'm not fully convinced that this is the correct approach, in that the change could mask stdin under-reads from poorly-written subprocesses that exit with failures but don't return with a non-zero exit code (alas, too many programs exit with 0 even when exiting with an error).
Comment 10 RHEL Product and Program Management 2011-02-22 16:35:00 EST
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.
Comment 16 Dave Malcolm 2011-04-07 15:47:54 EDT
Created attachment 490627 [details]
Backport of the upstream patch

The attached patch is a backport of the patch applied upstream to 2.7 to the somewhat modified 2.6.6 currently in RHEL 6.1

The upstream patch contained two new test cases; I've added some additional cases over and above those ones.  All of test/test_subprocess.py passes on my test build (x86_64; python-2.6.6-17 + this new patch)

I've also tested the patched version using attachment 471885 [details] (from comment #0), and it fixes the reported traceback, giving the expected results.
Comment 24 errata-xmlrpc 2011-12-06 05:23:57 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.