Bug 1268904 - HAProxy template has bind ports hardcoded
HAProxy template has bind ports hardcoded
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
3.0.0
Unspecified Unspecified
high Severity low
: ---
: ---
Assigned To: Ram Ranganathan
zhaozhanqi
:
Depends On:
Blocks: 1267746
  Show dependency treegraph
 
Reported: 2015-10-05 11:09 EDT by Miheer Salunke
Modified: 2017-03-08 13 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-09-27 05:30:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Origin (Github) 4907 None None None Never

  None (edit)
Description Miheer Salunke 2015-10-05 11:09:16 EDT
Description of problem:
The oadm router command supports binding to other ports, but it doesn't work because of the hardcoded values in the haproxy template (bind :443, bind :80)

Version-Release number of selected component (if applicable):
3.0

How reproducible:
Always

Steps to Reproduce:
1. Try oadm router command  with the --ports flag, but it doesn't work because of the hardcoded values in the haproxy template (bind :443, bind :80)
2.
3.

Actual results:
oadm router command  with the --ports flag doesn't work 

Expected results:
oadm router command  with the --ports flag should work 

Additional info:
The code: https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L36 & https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L53
Comment 3 Ram Ranganathan 2015-10-29 14:10:08 EDT
Was OOO - just got back today. Yeah, will have to address it in 3.2
Comment 5 Josep 'Pep' Turro Mauri 2016-01-11 13:50:54 EST
Just to clarify then:

(In reply to Miheer Salunke from comment #4)

> And it would be nice to have:
> 
>  ports:
> - containerPort: 8080
>   hostPort: 8080
>   protocol: TCP
>  - containerPort: 8443
>   hostPort: 8443
>   protocol: TCP
> 
> And using host network.

So this request is an RFE to have the router image's ports configurable somehow, right? Updating component accordingly
Comment 7 Ram Ranganathan 2016-02-29 20:17:59 EST
@Harald, was on vacation. Just got back so am not certain what the 3.2 release schedule or dates are but I created a PR for this. 
See: https://github.com/openshift/origin/pull/7704
Comment 8 Ram Ranganathan 2016-03-01 03:04:37 EST
Setting to ON_QA 
To test create and edit the router deployment config:
   $ oadm router --latest-images

   $ oc edit dc router
   
   #  and set the ROUTER_SERVICE_HTTP_PORT and ROUTER_SERVICE_HTTPS_PORT environment
   #  variables appropriately. And also set the ports (container and host ports) to
   #  the same port values (example 10080 and 10443). 


Or alternatively, you could use:
   oadm router --latest-images --ports=... 
   oc env dc/router ROUTER_SERVICE_HTTP[S]_PORT=...
commands to set the ports + env values. 

Also note, there could potentially be other code ala the console (UI) and CLI code and external integrations that may have 80/443 hardcoded to check if a service is up and running, so this fix is specific for this use case to allow custom port numbers and allow multiple routers to be run with host-networking (use different port numbers).
Comment 9 zhaozhanqi 2016-03-01 05:51:00 EST
Tested this issue on 
# oc version
oc v1.1.3-362-g7ba70f0
kubernetes v1.2.0-alpha.7-703-gbc4550d
openshift/origin-haproxy-router          latest              8391e00c75f7

scenario steps:
1. Create multi-router with different port with host-networking and container networking

# oc get endpoints first second third
NAME      ENDPOINTS                                                 AGE
first     172.18.14.20:10080,172.18.14.20:10443,172.18.14.20:1936   1h
second    172.18.14.20:80,172.18.14.20:443,172.18.14.20:1937        30m
third     172.17.0.4:1080,172.17.0.4:1938,172.17.0.4:1443           5m

2. create pod/service/route(unsecure,edge,passthrouth/reencryption)

the steps are work well

so this bug should has been fixed on origin.

since bug is reported on OSE and the code has not been merged to latest OSE. so in case it is forgot to be merged. I changed the status to 'assigned' for now. 
please help change back to 'ON_QA' once it was merged to OSE, I will verify it. thanks
Comment 10 zhaozhanqi 2016-03-01 06:06:36 EST
sorry,should be 'modified'
Comment 11 zhaozhanqi 2016-03-04 04:23:01 EST
hi, ram

I saw clayton stop the hostport port in https://github.com/openshift/origin/commit/241d02b8e89d492ad680559e56bb1549c9985c34

as my understanding, he want to use '--ports' to implement the mapping port between host and container when using container network mode.

and this https://github.com/openshift/origin/pull/7704 did not resolve this issue.

could you help confirm it, thanks.
Comment 12 Ram Ranganathan 2016-03-04 19:36:44 EST
@zhaozhanqi not sure I understand your test case. The default port numbers we
used in the past were always 80 and 443 irrespective of whether you run in
host-networking or container-networking mode.

Basically to test you'd run: 
oadm router --latest-images
oc env dc router ROUTER_SERVICE_HTTPS_PORT=5443 ROUTER_SERVICE_HTTP_PORT=5080

# cleanup router  
oadm router --latest-images --host-network=false
oc env dc router ROUTER_SERVICE_HTTPS_PORT=10443 ROUTER_SERVICE_HTTP_PORT=10080

The ports needing to be exposed in container networking mode are orthogonal to this PR. 
Meaning even without this PR, they need to be done.


And in the host networking case, these would match up automatically, example test case:
oadm router --latest-images
oc env dc router ROUTER_SERVICE_HTTPS_PORT=5443 ROUTER_SERVICE_HTTP_PORT=5080
curl -k -H "Host: allow-http.header.test" -H "now: $(date +%s)" http://127.0.0.1:5080
Comment 13 zhaozhanqi 2016-03-06 21:45:59 EST
@Ram

yes, before for container networking the router can export 'hostPort', thus container and host will have a mapping port,example:
# docker ps
CONTAINER ID        IMAGE                                                                                           COMMAND                  CREATED             STATUS              PORTS                                                              NAMES
b53732ae460b        brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-haproxy-router:v3.1.1.903   "/usr/bin/openshift-r"   2 days ago          Up 2 days                                                                              k8s_router.c86b40c7_router-1-723p8_default_c83ddf2f-e1d7-11e5-ac77-525400719108_6814bfb9
5dc1dcbc6c63        brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-pod:v3.1.1.903              "/pod"                   2 days ago          Up 2 days           0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:1936->1936/tcp   k8s_POD.9629577a_router-1-723p8_default_c83ddf2f-e1d7-11e5-ac77-525400719108_422aa0c2


you can see the mapping info: 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:1936->1936/tcp 

then we can also access the router using host port.

but now after the https://github.com/openshift/origin/commit/241d02b8e89d492ad680559e56bb1549c9985c34
rotuer.go #line 414

the 'hostPort' has been deleted. then the container are:

# docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
5017f2393100        openshift/origin-haproxy-router:latest   "/usr/bin/openshift-r"   8 minutes ago       Up 8 minutes                            k8s_router.706c7f85_router-1-7892t_default_20c1bb85-e408-11e5-8528-0e754fd826a7_4faa5e5a
bc6c09a5d360        openshift/origin-pod:v1.1.3              "/pod"                   9 minutes ago       Up 8 minutes                            k8s_POD.909543fa_router-1-7892t_default_20c1bb85-e408-11e5-8528-0e754fd826a7_72461cb1

you can see NO mapping port between host and container.
since router container ip can NOT be accessed out of openshift cluster

So I'm wondering how to resolve the router container ip from out of openshift cluster.
I thought this will be resolve using '--ports'.
Comment 14 zhaozhanqi 2016-03-07 02:44:39 EST
@Ram

Comment 13 issue can be resolved via setting 'externalIPs' in the router service. 
I will verified this bug once it's merged to OSE
Comment 15 Ram Ranganathan 2016-03-07 13:41:45 EST
@zhaozhanqi - thx for the update. Cool yeah this change was just to allow the port number to be configured.
 
The changes to container networking Clayton made were aside from this fix and now require the port to be exposed - example via a node-port service. Glad you figured that out.
Comment 16 zhaozhanqi 2016-03-22 04:26:25 EDT
@Ram Ranganathan

for hostnetwork mode router, if customer is using the specified port: eg 1081/10444.  if so, we need to tell user add iptables or Do we have another method?


steps:

1. Create router using --ports

 oadm router second --credentials=/etc/origin/master/openshift-router.kubeconfig --service-account=router --images='openshift3/ose-${component}:${version}' --stats-port=6701 --ports='1081:1081,10444:10444'

2. oc env dc second ROUTER_SERVICE_HTTP_PORT=1081
   oc env dc second ROUTER_SERVICE_HTTP_PORTS=10443

3. return no route when accessing the nodeport
   # curl 10.36.136.222:1081
curl: (7) Failed connect to 10.36.136.222:1081; Connection refused
Comment 17 Ram Ranganathan 2016-03-22 13:50:29 EDT
@zhaozhanqi - so this is assuming the nodes (where the router pods run) have those ports open - similar to the case for 80/443. Otherwise, no traffic is going to get in. 
I guess, we can probably add a section to the docs to mentioning to how to run routers on custom ports. Will send a PR to update the docs sometime later this week. Thx
Comment 18 Ram Ranganathan 2016-05-16 14:32:33 EDT
Added the docs changes earlier - forgot to update the bugz here. 
The docs PR was: https://github.com/openshift/openshift-docs/pull/1942 
and has been merged.

@zhaozhanqi FYI.
Comment 19 zhaozhanqi 2016-05-16 23:12:33 EDT
@Ram

In this line:
https://github.com/openshift/openshift-docs/pull/1942/files#diff-7c33c875858cf1c9cf05f070744861faR247

should add option 'ports' when creating router ,like 

oadm router --replicas=0 --ports='10080:10080,10443:10443' 

otherwise, 

the endpoints will still 80 and 443

# oc get endpoints router
router             10.66.136.222:80,10.66.136.222:443,10.66.136.222:1936   22m

that's mean the hostport and container port still 80/443. 

and I doubt this will make container network router can not work
Comment 20 Ram Ranganathan 2016-05-17 15:27:18 EDT
@zhaozhanqi, true that we can clarify it a bit more (that example is for host networking but point taken re: endpoints info mismatch).
Just updated the example. PR: https://github.com/openshift/openshift-docs/pull/2115
Comment 21 zhaozhanqi 2016-05-18 01:27:39 EDT
@Ram

thanks, it looks good to me.
Comment 24 Troy Dawson 2016-09-01 11:14:51 EDT
This has been merged into ose and is in OSE v3.3.0.28 or newer.
Comment 25 Meng Bo 2016-09-02 02:23:03 EDT
Checked on router image openshift3/ose-haproxy-router:v3.3.0.28 6ff97e2fd698

Issue has been fixed.

The ports can be modified via env ROUTER_SERVICE_HTTP_PORT ROUTER_SERVICE_HTTPS_PORT ROUTER_SERVICE_SNI_PORT ROUTER_SERVICE_NO_SNI_PORT.
Comment 27 errata-xmlrpc 2016-09-27 05:30:12 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933

Note You need to log in before you can comment on or make changes to this bug.