Bug 1751175

Summary: The fix CVE-2018-18074 leads to a regression
Product: Red Hat Enterprise Linux 7 Reporter: Christophe Besson <cbesson>
Component: python-requestsAssignee: Tomas Orsava <torsava>
Status: CLOSED DUPLICATE QA Contact: RHEL CS Apps Subsystem QE <rhel-cs-apps-subsystem-qe>
Severity: medium Docs Contact:
Priority: urgent    
Version: 7.7CC: cbesson, fkrska, jkejda, jomurphy, pviktori, torsava
Target Milestone: betaKeywords: Patch, Regression, Reproducer
Target Release: 7.7   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-23 14:09:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
proper handling for default ports in auth stripping none

Description Christophe Besson 2019-09-11 10:46:06 UTC 
Created attachment 1614031 [details]
proper handling for default ports in auth stripping

Description of problem:
Since updating to python-requests 2.6.0.5.el7 (from 2.6.0.1.el7) the customer can no longer succesfully monitor our EMC Unity array using the check_unity scripts, because python's requests library no longer succesfully completes authentication to the array.

Root cause:
The auth header is dropped after the redirection, whereas the scheme/hostname/port is the same, but the default port has been explicitly mentioned in the request (http/80 or https/443).

Version-Release number of selected component (if applicable):
2.6.0.5

How reproducible:
See below

Steps to Reproduce:
1/ Create 2 subdirs in Apache root directory, foo and bar, protected with a basic AuthType.

2/ Write a small php script which does a redirection (without re-using the full URL from the $_SERVER global var).
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<?php
header("Location: /bar", true, 301);
exit();
?>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

3/ Write a small python script to do a HTTP GET request with the default port explicitly mentioned.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
import requests
session = requests.Session()
session.auth = ('foo', 'foofoo')
response = session.get('http://192.168.122.12:80/foo')
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Actual results:
* Observe that it works with a previous version of python-requests (2.6.0-1 in this example)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ tshark port http
  4 0.000213603 192.168.122.1 -> 192.168.122.12 HTTP 299 GET /foo HTTP/1.1 
  6 0.000683891 192.168.122.12 -> 192.168.122.1 HTTP 591 HTTP/1.1 301 Moved Permanently  (text/html)
  8 0.001785669 192.168.122.1 -> 192.168.122.12 HTTP 300 GET /foo/ HTTP/1.1 
  9 0.002696987 192.168.122.12 -> 192.168.122.1 HTTP 341 HTTP/1.1 302 Found 
 10 0.003495810 192.168.122.1 -> 192.168.122.12 HTTP 299 GET /bar HTTP/1.1 
 11 0.003791207 192.168.122.12 -> 192.168.122.1 HTTP 590 HTTP/1.1 301 Moved Permanently  (text/html)
 12 0.004511989 192.168.122.1 -> 192.168.122.12 HTTP 300 GET /bar/ HTTP/1.1 
 13 0.005380148 192.168.122.12 -> 192.168.122.1 HTTP 342 HTTP/1.1 200 OK  (text/html)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

=> The authentication is still present after the HTTP 302 response:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hypertext Transfer Protocol
    GET /bar HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET /bar HTTP/1.1\r\n]
            [Message: GET /bar HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /bar
        Request Version: HTTP/1.1
    Host: 192.168.122.12\r\n
    Connection: keep-alive\r\n
    Accept: */*\r\n
    Accept-Encoding: gzip, deflate\r\n
    Authorization: Basic Zm9vOmZvb2Zvbw==\r\n                          <===========
        Credentials: foo:foofoo                                        <===========
    User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.21.3.el7.x86_64\r\n
    \r\n
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Observe that it does NOT work with python-requests-2.6.0-5:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  4 0.000106526 192.168.122.12 -> 192.168.122.12 HTTP 301 GET /foo HTTP/1.1 
  6 0.000980269 192.168.122.12 -> 192.168.122.12 HTTP 593 HTTP/1.1 301 Moved Permanently  (text/html)
  8 0.002424241 192.168.122.12 -> 192.168.122.12 HTTP 263 GET /foo/ HTTP/1.1 
  9 0.002551020 192.168.122.12 -> 192.168.122.12 HTTP 746 HTTP/1.1 401 Unauthorized  (text/html)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

=> In detail, the frames 8 without the auth header, which leads to the 401 response (frame 9).
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hypertext Transfer Protocol
    HTTP/1.1 301 Moved Permanently\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 301 Moved Permanently\r\n]
            [Message: HTTP/1.1 301 Moved Permanently\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Status Code: 301
        Response Phrase: Moved Permanently
    Date: Wed, 11 Sep 2019 08:55:41 GMT\r\n
    Server: Apache/2.4.6 (Red Hat Enterprise Linux) PHP/5.4.16\r\n
    Location: http://192.168.122.12/foo/\r\n
    Content-Length: 234\r\n
        [Content length: 234]
    Keep-Alive: timeout=5, max=100\r\n
    Connection: Keep-Alive\r\n
    Content-Type: text/html; charset=iso-8859-1\r\n
    \r\n
    [HTTP response 1/1]
    [Time since request: 0.000873743 seconds]
    [Request in frame: 4]
Line-based text data: text/html
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n
    <html><head>\n
    <title>301 Moved Permanently</title>\n
    </head><body>\n
    <h1>Moved Permanently</h1>\n
    <p>The document has moved <a href="http://192.168.122.12/foo/">here</a>.</p>\n
    </body></html>\n

Hypertext Transfer Protocol
    HTTP/1.1 401 Unauthorized\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 401 Unauthorized\r\n]
            [Message: HTTP/1.1 401 Unauthorized\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Status Code: 401
        Response Phrase: Unauthorized
    Date: Wed, 11 Sep 2019 08:55:41 GMT\r\n
    Server: Apache/2.4.6 (Red Hat Enterprise Linux) PHP/5.4.16\r\n
    WWW-Authenticate: Basic realm="Restricted Files"\r\n
    Content-Length: 381\r\n
        [Content length: 381]
    Keep-Alive: timeout=5, max=99\r\n
    Connection: Keep-Alive\r\n
    Content-Type: text/html; charset=iso-8859-1\r\n
    \r\n
    [HTTP response 2/2]
    [Time since request: 0.000126779 seconds]
    [Prev request in frame: 4]
    [Prev response in frame: 6]
    [Request in frame: 8]
Line-based text data: text/html
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n
    <html><head>\n
    <title>401 Unauthorized</title>\n
    </head><body>\n
    <h1>Unauthorized</h1>\n
    <p>This server could not verify that you\n
    are authorized to access the document\n
    requested.  Either you supplied the wrong\n
    credentials (e.g., bad password), or your\n
    browser doesn't understand how to supply\n
    the credentials required.</p>\n
    </body></html>\n
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expected results:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  4 0.000070794 192.168.122.12 -> 192.168.122.12 HTTP 301 GET /foo HTTP/1.1 
  6 0.001320044 192.168.122.12 -> 192.168.122.12 HTTP 593 HTTP/1.1 301 Moved Permanently  (text/html)
  8 0.003348937 192.168.122.12 -> 192.168.122.12 HTTP 302 GET /foo/ HTTP/1.1 
  9 0.004326371 192.168.122.12 -> 192.168.122.12 HTTP 355 HTTP/1.1 301 Moved Permanently 
 10 0.005436360 192.168.122.12 -> 192.168.122.12 HTTP 301 GET /bar HTTP/1.1 
 11 0.005847319 192.168.122.12 -> 192.168.122.12 HTTP 592 HTTP/1.1 301 Moved Permanently  (text/html)
 12 0.006739319 192.168.122.12 -> 192.168.122.12 HTTP 302 GET /bar/ HTTP/1.1 
 13 0.007998180 192.168.122.12 -> 192.168.122.12 HTTP 344 HTTP/1.1 200 OK  (text/html)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Additional info:
I provide the (modified) patch from the upstream which has been merged in the master branch as an attachment, it fixes the issue for me.

https://github.com/psf/requests/pull/4851

Please also note that the python3-requests coming with RHEL8.x is also impacted.
Comment 6 Tomas Orsava 2019-09-23 13:16:22 UTC 
Hi Filip,
I was about to add the patch when I noticed that it is already fixed in RHEL 7.8!

https://bugzilla.redhat.com/show_bug.cgi?id=1745417

Could you please test with the new version from that BZ, python-requests-2.6.0-6.el7?

I think we should close this issue as a duplicate and start the "Accelerated fix release (Z-Stream)" on the original BZ. Do you agree?
Comment 8 Tomas Orsava 2019-09-23 14:09:18 UTC 

*** This bug has been marked as a duplicate of bug 1745417 ***