Bug 1380790

Summary:	Wrong route info shown under /var/lib/haproxy/conf/os_sni_passthrough.map
Product:	OpenShift Container Platform	Reporter:	Weibin Liang <weliang>
Component:	Networking	Assignee:	Jacob Tanenbaum <jtanenba>
Networking sub component:	router	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED NOTABUG	Docs Contact:
Severity:	low
Priority:	low	CC:	aloughla, aos-bugs, bbennett, bmeng, bperkins
Version:	3.3.0
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-15 14:53:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Weibin Liang 2016-09-30 14:35:35 UTC

Description of problem:
Create namespace router shards env, in p1 create p1-router and one route, in p2 create one reencrypte route, in p3 create one passthrough route, all p1,p2 and p3 have same namespace labels, passthrough route infor is wrong under /var/lib/haproxy/conf/os_sni_passthrough.map

Version-Release number of selected component (if applicable):
[root@dhcp-41-211 ~]# oc version
oc v3.3.0.32
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://dhcp-41-211.bos.redhat.com:8443
openshift v3.3.0.32
kubernetes v1.3.0+52492b4

How reproducible: Reproducible by following below steps


Steps to Reproduce:
[root@dhcp-41-211 ~]# oc new-project p1
Now using project "p1" on server "https://dhcp-41-211.bos.redhat.com:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
[root@dhcp-41-211 ~]# oc label namespace p1 "router=p2"
namespace "p1" labeled
[root@dhcp-41-211 ~]# oadm policy add-scc-to-user privileged -z user1
[root@dhcp-41-211 ~]# oadm router p1-router --replicas=0  --service-account=user1 -n p1 --host-network=true
info: password for stats user admin has been set to phf1Xni5RD
--> Creating router p1-router ...
    serviceaccount "user1" created
    error: rolebinding "router-p1-router-role" already exists
    deploymentconfig "p1-router" created
    service "p1-router" created
--> Failed
[root@dhcp-41-211 ~]# oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:p1:user1
[root@dhcp-41-211 ~]# oc env dc/p1-router NAMESPACE_LABELS="router=p2"
deploymentconfig "p1-router" updated
[root@dhcp-41-211 ~]# oc scale dc/p1-router --replicas=1
deploymentconfig "p1-router" scaled
[root@dhcp-41-211 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/OSE3.3/hello-openshift-twopods.json
route "hello-route" created
service "hello-service" created
pod "hello-pod-1" created
pod "hello-pod-2" created

[root@dhcp-41-211 ~]# 
[root@dhcp-41-211 ~]# oc new-project p2
Now using project "p2" on server "https://dhcp-41-211.bos.redhat.com:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
[root@dhcp-41-211 ~]# oc label namespace p2 "router=p2"
namespace "p2" labeled
[root@dhcp-41-211 ~]# wget https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/route.pem
--2016-09-30 10:17:17--  https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/route.pem
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.56.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.56.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1480 (1.4K) [text/plain]
Saving to: ‘route.pem.7’

100%[===================================================================================================================================>] 1,480       --.-K/s   in 0s      

2016-09-30 10:17:17 (476 MB/s) - ‘route.pem.7’ saved [1480/1480]

[root@dhcp-41-211 ~]# oc create -f  https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/reencrypt-route.json
route "route-reen" created
service "service-secure" created
pod "caddy-docker-1" created
pod "caddy-docker-2" created
[root@dhcp-41-211 ~]# 
[root@dhcp-41-211 ~]# 
[root@dhcp-41-211 ~]# oc new-project p3
Now using project "p3" on server "https://dhcp-41-211.bos.redhat.com:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
[root@dhcp-41-211 ~]# oc label namespace p3 "router=p2"
namespace "p3" labeled
[root@dhcp-41-211 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/passthrough-route.json
route "route-secure" created
service "hello-nginx-https" created
pod "hello-nginx-docker-1" created
pod "hello-nginx-docker-2" created
[root@dhcp-41-211 ~]# 
[root@dhcp-41-211 ~]# 
[root@dhcp-41-211 ~]# oc project p1
Now using project "p1" on server "https://dhcp-41-211.bos.redhat.com:8443".
[root@dhcp-41-211 ~]# oc scale dc/p1-router --replicas=0 && oc scale dc/p1-router --replicas=1
deploymentconfig "p1-router" scaled
deploymentconfig "p1-router" scaled
[root@dhcp-41-211 ~]# sleep 30
[root@dhcp-41-211 ~]# oc project p1
Already on project "p1" on server "https://dhcp-41-211.bos.redhat.com:8443".
[root@dhcp-41-211 ~]# oc get route
NAME          HOST/PORT             PATH      SERVICES        PORT      TERMINATION
hello-route   hello-openshift.com             hello-service   <all>     
[root@dhcp-41-211 ~]# oc project p2
Now using project "p2" on server "https://dhcp-41-211.bos.redhat.com:8443".
[root@dhcp-41-211 ~]# oc get route
NAME         HOST/PORT          PATH      SERVICES         PORT      TERMINATION
route-reen   reen.example.com             service-secure   https     reencrypt
[root@dhcp-41-211 ~]# oc project p3
Now using project "p3" on server "https://dhcp-41-211.bos.redhat.com:8443".
[root@dhcp-41-211 ~]# oc get route
NAME           HOST/PORT         PATH      SERVICES            PORT      TERMINATION
route-secure   www.example.com             hello-nginx-https   <all>     passthrough
[root@dhcp-41-211 ~]# curl --resolve hello-openshift.com:80:$IP http://hello-openshift.com
Hello OpenShift!
[root@dhcp-41-211 ~]# curl --resolve reen.example.com:443:$IP https://reen.example.com --cacert route.pem
Hello-OpenShift-1 https-8443
[root@dhcp-41-211 ~]# curl --resolve www.example.com:443:$IP  https://www.example.com --cacert route.pem
Hello World
[root@dhcp-41-211 ~]# 


From router node:
[root@dhcp-41-239 ~]# cid=$(docker ps | egrep "openshift3/(origin|ose)-haproxy-router" | awk '{print $1}')
[root@dhcp-41-239 ~]# sudo nsenter -m -u -n -i -p -t $(docker inspect --format "{{ .State.Pid }}" "$cid")
[root@dhcp-41-239 /]# cd /var/lib/haproxy/conf
[root@dhcp-41-239 conf]# more /var/lib/haproxy/conf/os_http_be.map
hello-openshift.com p1_hello-route
[root@dhcp-41-239 conf]# ls
default_pub_keys.pem  haproxy.config           os_edge_http_be.map      os_edge_http_redirect.map  os_reencrypt.map        os_tcp_be.map
error-page-503.http   haproxy-config.template  os_edge_http_expose.map  os_http_be.map             os_sni_passthrough.map
[root@dhcp-41-239 conf]# more os_reencrypt.map
reen.example.com p2_route-reen
[root@dhcp-41-239 conf]# more os_sni_passthrough.map
www.example.com 1


Actual results:
[root@dhcp-41-239 conf]# more os_sni_passthrough.map
www.example.com 1


Expected results:
[root@dhcp-41-239 conf]# more os_sni_passthrough.map
www.example.com p3-route-secure

Additional info:

Comment 1 Meng Bo 2016-10-09 08:32:00 UTC

First, this should not be a problem which related to the router sharding.
The contents of os_sni_passthrough.map file will be always like this.

From the haproxy router template:

{{/*
    os_sni_passthrough.map: contains a mapping of routes that expect to have an sni header and should be passed
                                        through to the host_be.  Driven by the termination type of the ServiceAliasConfigs
*/}}
{{ define "/var/lib/haproxy/conf/os_sni_passthrough.map" }}
{{     range $idx, $cfg := .State }}
{{       if and (eq $cfg.Path "") (eq $cfg.TLSTermination "passthrough") }}
{{$cfg.Host}} 1
{{       end }}
{{     end }}
{{ end }}{{/* end sni passthrough map template */}}

The "1" in the file looks like an indicator.

Comment 2 Weibin Liang 2016-10-10 15:13:49 UTC

From end user point of view, what does "1" mean? It make no sense to customers.

for consistency purpose, passthrough route in router sharding setup should display the same meaningful result as reencrypt and edge route did.

Comment 3 zhaozhanqi 2016-10-11 01:32:01 UTC

hi, Weibin

I'm not sure if you already found any issues about function of passthrough route unless the content of this file is not perfect.

for me, I don't think the customers I mean admin user here care about what the content is and also the normal user have not the right to view it. 
 
since the function of passthrough is working well. so I think this should not be an issue too.

Comment 4 Weibin Liang 2016-10-17 15:49:54 UTC

If there two services using two different passthrough routes in two different namespaces:

I hope the correct route entries should be like: 
www.example1.com 1 p1-route1
www.example2.com 1 p2-route2

The current route entries are like this:
www.example1.com 1
www.example2.com 1

I am not sure how www.example1.com and www.example2.com know which namespace and which route it should use.

I haven't setup to test above config, just curious

Comment 5 Ben Bennett 2016-10-31 17:38:48 UTC

Jake, can you please see why we are writing 1 here rather than {{$idx}} for the mapping file?  It seems to be the only one where it is different.

This is not a user-visible change, so I'm lowering the priority.

Comment 6 Jacob Tanenbaum 2017-02-15 14:53:03 UTC

The "1" can be anything, the haproxy template uses it as an if-exists check. Using 1 reduces the size of the map file, whether we care about the size of the map file could be a topic of discussion.