Bug 1397428

Summary: Redirection to file results in traceback
Product: [Fedora] Fedora Reporter: Jaroslav Škarvada <jskarvad>
Component: pythonAssignee: Charalampos Stratakis <cstratak>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 24CC: bkabrda, cstratak, dmalcolm, ivazqueznet, jonathansteffan, mhroncok, ncoghlan, pviktori, rkuska, tomspur, torsava
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-22 15:59:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jaroslav Škarvada 2016-11-22 13:55:54 UTC 
Description of problem:
The same code when redirected from STDOUT to file can result in traceback, which is strange.

Version-Release number of selected component (if applicable):
python-2.7.12-6.fc24.x86_64

How reproducible:
Always

Steps to Reproduce:
1.cat > test.py <<:EOF
#!/bin/python

url = 'https://github.com/EttusResearch/uhd/releases'
import requests
text = requests.get(url).text
print(text)
:EOF

2. python test.py
3. python test.py > log.txt

Actual results:
2. content
3. Traceback (most recent call last):
  File "./test0.py", line 10, in <module>
    print text
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 1016: ordinal not in range(128)

Expected results:
2. content
3. content

Additional info:
Redirection to file shouldn't result in different behavior, nor in traceback.
Comment 1 Miro Hrončok 2016-11-22 15:55:10 UTC 
Smaller reproducer:

1.cat > test.py <<:EOF
print(u'\xb7')
:EOF

2. python test.py
3. python test.py | head

Actual results:
2. ·
3. Traceback (most recent call last):
  File "test.py", line 1, in <module>
    print(u'\xb7')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 0: ordinal not in range(128)

Expected results:
2. ·
3. ·

But I'm afraid this is a "feature" of Python 2.

See http://stackoverflow.com/questions/19696652/piping-output-causes-python-program-to-fail
Comment 2 Dave Malcolm 2016-11-22 15:59:29 UTC 
The issue is that in CPython 2 the encoding of stdout changes from utf-8 to ascii if stdout is not a tty.

I agree that it's counter-intuitive.

FWIW, I proposed changing this in Fedora 13:

  https://fedoraproject.org/wiki/Features/PythonEncodingUsesSystemLocale

but I was asked to not change this by the upstream Python development community, to ensure consistent behavior with every other CPython 2 implementation.

See that link for more information, and some workarounds.

Closing this out as WONTFIX.