Bug 1302894

Summary: [behind proxy] 'oc rsh' and 'oc exec' fail behind an authenticated proxy
Product: OpenShift Container Platform Reporter: Michael A. Cleverly <michael.cleverly>
Component: ocAssignee: Fabiano Franz <ffranz>
Status: CLOSED ERRATA QA Contact: Wei Sun <wsun>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.0CC: aos-bugs, Brandon.Richins, ederevea, erich, jokerman, michael.cleverly, mmccomas, tdawson, veer, xtian, xxia, yapei
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-12 16:27:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
localhost proxy to add missing Proxy-Authorization header
none
--v=9 level logging with workaround in place
none
--v=9 level logging WITHOUT workaround none

Description Michael A. Cleverly 2016-01-28 22:07:52 UTC
Description of problem:

The 'oc rsh' and 'oc exec' commands make two http requests.  While both go through the proxy (specified by the http_proxy & https_proxy environment variables) only the first of the two includes a Proxy-Authorization header with the necessary proxy credentials.

How reproducible:

Always.

Steps to Reproduce:

1. Be behind an authenticated proxy server

2. Set proper proxy environment variables:

    export http_proxy='http://user:pass@proxy:port'
    export https_proxy='http://user:pass@proxy:port'

3. Run 'oc rsh $POD'


Actual results:

Will fail with an error like: "error: error sending request: Post https://ose3-master.devday.osecloud.com:8443/api/v1/namespaces/mycliproject-imuser13/pods/time-1-py9ub/exec?command=%2Fbin%2Fbash&container=time&container=time&stderr=true&stdin=true&stdout=true&tty=true: EOF"

Expected results:

Open an ssh session on the pod.

Additional info:

The HTTP headers from the first connection to the proxy are:

    CONNECT ose3-master.devday.osecloud.com:8443 HTTP/1.1
    Host: ose3-master.devday.osecloud.com:8443
    User-Agent: Go 1.1 package http
    Proxy-Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

The HTTP headers from the second connection to the proxy are:

    CONNECT ose3-master.devday.osecloud.com:8443 HTTP/1.1
    Host: ose3-master.devday.osecloud.com:8443
    User-Agent: Go 1.1 package http

The only difference is that the second request lacked Proxy-Authorization and so the proxy server returns an HTTP 407 response:

    HTTP/1.1 407 Proxy Authentication Required
    Server: squid/3.3.13
    Mime-Version: 1.0
    Date: Thu, 28 Jan 2016 21:57:04 GMT
    Content-Type: text/html
    Content-Length: 340
    X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
    Vary: Accept-Language
    Content-Language: en
    Proxy-Authenticate: NTLM
    Proxy-Authenticate: Basic realm="Intermountain Health Care Proxy (Note: Password is case sensitive)"
    X-Cache: MISS from proxy.ihc.com
    Via: 1.1 proxy.ihc.com (squid/3.3.13)
    Connection: close

Comment 1 Fabiano Franz 2016-01-29 13:44:38 UTC
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1302422

Comment 2 Michael A. Cleverly 2016-01-29 18:46:22 UTC
Created attachment 1119496 [details]
localhost proxy to add missing Proxy-Authorization header

Usage: 

Modify lines 8-10 of bug-1302894-workaround.tcl with your authenticated proxy information, then run ./bug-1302894-workaround.tcl

In another Window:

export set http_proxy=http://localhost:8080
export set https_proxy=http://localhost:8080

Then run: oc rsh $POD 

This will take the credentials provided from oc's first CONNECT request and include them with the second allowing the 'oc rsh' (or 'oc exec') command to work.

Comment 3 Michael A. Cleverly 2016-01-29 19:03:58 UTC
Comment on attachment 1119496 [details]
localhost proxy to add missing Proxy-Authorization header

Usage: 

Edit lines 8-10 of bug-1302894-workaround.tcl to specify your authenticated proxy

Run ./bug-1302894-workaround.tcl

In another window, run:

export set http_proxy=http://user:pass@localhost:8080
export set https_proxy=http://user:pass@localhost:8080
oc rsh $POD

Comment 4 Michael A. Cleverly 2016-01-29 19:13:27 UTC
Created attachment 1119500 [details]
--v=9 level logging with workaround in place

Comment 5 Michael A. Cleverly 2016-01-29 19:14:36 UTC
Created attachment 1119501 [details]
--v=9 level logging WITHOUT workaround

Comment 6 Fabiano Franz 2016-02-20 00:12:47 UTC
Fixed in https://github.com/openshift/origin/pull/7362.

Comment 7 Xingxing Xia 2016-02-22 09:30:43 UTC
Verified against latest Origin, the bug is fixed.
The latest version of OSE has not yet merged the fix in. Will verify against OSE when it merged.

Comment 8 Xingxing Xia 2016-02-23 03:14:22 UTC
Verified against latest OSE, versions are:
oc v3.1.1.905
kubernetes v1.2.0-alpha.7-703-gbc4550d

Verification steps:
1. (If there is no existing one) prepare a proxy server that needs authentication
1> $ sudo yum install squid
2> $ sudo htpasswd -c /etc/squid/passwd xxia
3> $ sudo vi /etc/squid/squid.conf # Add the following lines in proper places:
auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid/passwd

acl SSL_ports port 8443

acl squid_user proxy_auth xxia

http_access allow squid_user
4> $ sudo service squid restart

2. oc login and create project

3. Create pod (from new app)
$ oc new-app -f origin/examples/sample-app/application-template-stibuild.json

4. Check `oc rsh`, `oc exec` behind authenticated proxy
1>
$ export http_proxy=xxia:<password>@<proxy server>:3128
$ export https_proxy=xxia:<password>@<proxy server>:3128
2>
$ oc rsh database-1-uqnvn
bash-4.2$
$ oc exec database-1-uqnvn ls /etc/hosts
/etc/hosts

Actual results:
4.2 Both `oc rsh`, `oc exec` commands succeed.

The bug is fixed, so move it to VERIFIED.

Comment 11 errata-xmlrpc 2016-05-12 16:27:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064