Bug 1430105

Summary:	[preview][prod]container do not get original remote client IP address, X-Forwarded-For is incorrect
Product:	OpenShift Online	Reporter:	Alex Syrnikov <san>
Component:	Routing	Assignee:	Stefanie Forrester <dakini>
Status:	CLOSED CURRENTRELEASE	QA Contact:	zhaliu
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.x	CC:	aos-bugs, bbennett, jawnsy, xtian, yufchang
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-05-10 16:54:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alex Syrnikov 2017-03-07 21:26:34 UTC

Description of problem:

I created docker container with http service, but in HTTP request my application did not get original remote client address in X-Forwarded-For header field. Actually container do not get remote address in any field.

Version-Release number of selected component (if applicable):

current developer preview for OpenShift v3 (doker based containers)

How reproducible:

1. create docker image with http service, which will print HTTP request headers on stdout,
2. store that image to hub.docker.com
3. create project and select "deploy image", chuse "Image Name".
4. finish creating project, create route
5. Make HTTP request to service and look for container\'s logs

Actual results:

My request headers
GET / HTTP/1.1
Host: openshift-masterspline-hello-openshift5.44fs.preview.openshiftapps.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,ru;q=0.6
X-Forwarded-Host: openshift-masterspline-hello-openshift5.44fs.preview.openshiftapps.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Forwarded: for=172.31.0.91;host=openshift-masterspline-hello-openshift5.44fs.preview.openshiftapps.com;proto=http
X-Forwarded-For: 172.31.0.91

Expected results:

The same headers, but with my real IP in X-Forwarded-For
X-Forwarded-For: 84.17.12.94

Comment 1 Jonathan Yu 2017-03-21 21:18:13 UTC

Hi Alex,

Thanks for the report!

It looks like the ELB needs to be configured with the PROXY protocol so that the original IP address is passed through. It seems that HAProxy is setting correct headers, but is unaware that there's a reverse proxy in front, or does not trust the inbound X-Forwarded-For headers due to configuration.

We will take a look and get back to you.

Comment 2 Stefanie Forrester 2017-03-22 14:03:43 UTC

Enabled proxy protocol on router:
oc env dc/router ROUTER_USE_PROXY_PROTOCOL=true

Verified ELBs are using TCP protocol:


$  aws elb describe-load-balancers --load-balancer-name "$lb" |      jq '[.LoadBalancerDescriptions[]|.ListenerDescriptions]'
[
  [
    {
      "Listener": {
        "InstancePort": 80,
        "LoadBalancerPort": 80,
        "Protocol": "TCP",
        "InstanceProtocol": "TCP"
      },
      "PolicyNames": []
    },
    {
      "Listener": {
        "InstancePort": 5000,
        "LoadBalancerPort": 5000,
        "Protocol": "TCP",
        "InstanceProtocol": "TCP"
      },
      "PolicyNames": []
    },
    {
      "Listener": {
        "InstancePort": 443,
        "LoadBalancerPort": 443,
        "Protocol": "TCP",
        "InstanceProtocol": "TCP"
      },
      "PolicyNames": []
    }
  ]
]

Comment 3 zhaliu 2017-03-23 08:06:49 UTC

The route can not be accessed now!

#oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/caddy-docker.json

#oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/edge/service_unsecure.json

#oc expose svc service-unsecure

#oc get route
NAME               HOST/PORT                                               PATH      SERVICES           PORT      TERMINATION
service-unsecure   service-unsecure-alpha.44fs.preview.openshiftapps.com             service-unsecure   http 

#curl http://service-unsecure-alpha.44fs.preview.openshiftapps.com
curl: (52) Empty reply from server

Comment 4 Ben Bennett 2017-03-24 17:47:42 UTC

I think we need to follow this guide to set up the ELB:
  http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html#enable-proxy-protocol-cli

Comment 5 Stefanie Forrester 2017-05-10 16:54:33 UTC

This issue has been fixed in the latest version of OpenShift Online, which is available here:

https://manage.openshift.com/