Bug 1464379 - [trello:TbZPhHKE] Route is not available when access from localhost after adding localhost ip in whitelist on OCP env based on AWS/GCE
[trello:TbZPhHKE] Route is not available when access from localhost after add...
Status: CLOSED NOTABUG
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Phil Cameron
zhaozhanqi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-23 05:46 EDT by Yan Du
Modified: 2017-06-28 16:14 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-28 16:14:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 2 Ben Bennett 2017-06-23 11:09:10 EDT
My hunch is that this is because the ELB in front of the router is confusing the source address.  @Phil, can you take a look at this to see if src honors the proxy protocol?  If so, we need to update the docs to tell people it will only work if proxy protocol is in use.
Comment 3 Weibin Liang 2017-06-27 15:52:00 EDT
@yandu, it works fine in my env with 3.9.121 image, curl from both master and node works.

[root@ip-172-18-8-94 ~]# oc annotate route route1 --overwrite haproxy.router.openshift.io/ip_whitelist='172.18.8.94 172.18.3.148'
route "route1" annotated
[root@ip-172-18-8-94 ~]# curl --resolve hello-openshift.com:80:172.18.3.148 http://hello-openshift.com
Hello OpenShift!
[root@ip-172-18-8-94 ~]# oc project default
Now using project "default" on server "https://ip-172-18-8-94.ec2.internal:8443".
[root@ip-172-18-8-94 ~]# oc rsh router-1-cqgf1
sh-4.2$ grep -B6 white haproxy.config 
# Plain http backend
backend be_http:p1:route1
  mode http
  option redispatch
  option forwardfor
  balance leastconn
  acl whitelist src 172.18.8.94 172.18.3.148
  tcp-request content reject if !whitelist 
sh-4.2$ exit
exit

Additional info:
When oc env dc/router ROUTER_USE_PROXY_PROTOCOL=false, both curl work
When oc env dc/router ROUTER_USE_PROXY_PROTOCOL=true, both curl fail
Comment 4 Eric Paris 2017-06-27 20:09:51 EDT
Does route1-d1.0623-8o4.qe.rhcloud.com point to the specific node running the router or is that a Google Load Balancer which points to the specific node running the router?

If route1-d1.0623-8o4.qe.rhcloud.com points directly to the router then I'm not sure why weibin works and yandu fails.

If route1-d1.0623-8o4.qe.rhcloud.com does go through a Google TCP Proxy Load Balancer than we can explain what happened. If it goes through the proxy then the Google Load Balancer needs to be configured with "proxy protocol".

https://cloud.google.com/compute/docs/load-balancing/tcp-ssl/tcp-proxy#proxy-protocol

And the router needs to run with ROUTER_USE_PROXY_PROTOCOL=true

Both sides must always match! Either proxy protocol must be on in the LB AND the Router or must be off in both. If the do not match NOTHING will work.


If there is no google load balancer involved can you give us login details for the reproducer? I wonder if some form of NAT could exist which might cause the issue you are describing. But I'm not certain how to prove that without trial and error.
Comment 5 Yan Du 2017-06-27 22:20:45 EDT
I think @weibin may misunderstand the bug.
I could also curl successfully from master/node. It is just not working when we curl from own local vm, most normal user may use their own vms to connect OCP and  access the route since they don't have permission to login to master/node. You could refer to detail info in step5 and step6 above.
Comment 6 Eric Paris 2017-06-27 23:00:08 EDT
@yandu   Ohhhh ok. So the problem is likely that the router is seeing the source as 127.0.01? Can you add 127.0.0.1 to the whitelist. If that works it would prove my hypothesis?
Comment 8 Weibin Liang 2017-06-28 14:34:16 EDT
@yandu, updated my testing env as you mentioned above, now I can not curl from local VM when add my VM ip/10.18.41.193 in whitelist.

In AWS router, tcpdump to captures the http packets when curl from local VM, and you will see the source IP in http packets is 66.187.233.206 (not 10.18.41.193), if you put 66.187.233.206 in whitelist, curl from local VM will work again.

The test case failed because of including the wrong ip address in whitelist.

Note You need to log in before you can comment on or make changes to this bug.