Bug 1268904 - HAProxy template has bind ports hardcoded
Summary: HAProxy template has bind ports hardcoded
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.0.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Ram Ranganathan
QA Contact: zhaozhanqi
Depends On:
Blocks: 1267746
TreeView+ depends on / blocked
Reported: 2015-10-05 15:09 UTC by Miheer Salunke
Modified: 2019-12-16 04:59 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-09-27 09:30:12 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Origin (Github) 4907 0 None None None Never
Red Hat Product Errata RHBA-2016:1933 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Description Miheer Salunke 2015-10-05 15:09:16 UTC
Description of problem:
The oadm router command supports binding to other ports, but it doesn't work because of the hardcoded values in the haproxy template (bind :443, bind :80)

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Try oadm router command  with the --ports flag, but it doesn't work because of the hardcoded values in the haproxy template (bind :443, bind :80)

Actual results:
oadm router command  with the --ports flag doesn't work 

Expected results:
oadm router command  with the --ports flag should work 

Additional info:
The code: https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L36 & https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L53

Comment 3 Ram Ranganathan 2015-10-29 18:10:08 UTC
Was OOO - just got back today. Yeah, will have to address it in 3.2

Comment 5 Josep 'Pep' Turro Mauri 2016-01-11 18:50:54 UTC
Just to clarify then:

(In reply to Miheer Salunke from comment #4)

> And it would be nice to have:
>  ports:
> - containerPort: 8080
>   hostPort: 8080
>   protocol: TCP
>  - containerPort: 8443
>   hostPort: 8443
>   protocol: TCP
> And using host network.

So this request is an RFE to have the router image's ports configurable somehow, right? Updating component accordingly

Comment 7 Ram Ranganathan 2016-03-01 01:17:59 UTC
@Harald, was on vacation. Just got back so am not certain what the 3.2 release schedule or dates are but I created a PR for this. 
See: https://github.com/openshift/origin/pull/7704

Comment 8 Ram Ranganathan 2016-03-01 08:04:37 UTC
Setting to ON_QA 
To test create and edit the router deployment config:
   $ oadm router --latest-images

   $ oc edit dc router
   #  variables appropriately. And also set the ports (container and host ports) to
   #  the same port values (example 10080 and 10443). 

Or alternatively, you could use:
   oadm router --latest-images --ports=... 
   oc env dc/router ROUTER_SERVICE_HTTP[S]_PORT=...
commands to set the ports + env values. 

Also note, there could potentially be other code ala the console (UI) and CLI code and external integrations that may have 80/443 hardcoded to check if a service is up and running, so this fix is specific for this use case to allow custom port numbers and allow multiple routers to be run with host-networking (use different port numbers).

Comment 9 zhaozhanqi 2016-03-01 10:51:00 UTC
Tested this issue on 
# oc version
oc v1.1.3-362-g7ba70f0
kubernetes v1.2.0-alpha.7-703-gbc4550d
openshift/origin-haproxy-router          latest              8391e00c75f7

scenario steps:
1. Create multi-router with different port with host-networking and container networking

# oc get endpoints first second third
NAME      ENDPOINTS                                                 AGE
first,,   1h
second,,        30m
third,,           5m

2. create pod/service/route(unsecure,edge,passthrouth/reencryption)

the steps are work well

so this bug should has been fixed on origin.

since bug is reported on OSE and the code has not been merged to latest OSE. so in case it is forgot to be merged. I changed the status to 'assigned' for now. 
please help change back to 'ON_QA' once it was merged to OSE, I will verify it. thanks

Comment 10 zhaozhanqi 2016-03-01 11:06:36 UTC
sorry,should be 'modified'

Comment 11 zhaozhanqi 2016-03-04 09:23:01 UTC
hi, ram

I saw clayton stop the hostport port in https://github.com/openshift/origin/commit/241d02b8e89d492ad680559e56bb1549c9985c34

as my understanding, he want to use '--ports' to implement the mapping port between host and container when using container network mode.

and this https://github.com/openshift/origin/pull/7704 did not resolve this issue.

could you help confirm it, thanks.

Comment 12 Ram Ranganathan 2016-03-05 00:36:44 UTC
@zhaozhanqi not sure I understand your test case. The default port numbers we
used in the past were always 80 and 443 irrespective of whether you run in
host-networking or container-networking mode.

Basically to test you'd run: 
oadm router --latest-images

# cleanup router  
oadm router --latest-images --host-network=false

The ports needing to be exposed in container networking mode are orthogonal to this PR. 
Meaning even without this PR, they need to be done.

And in the host networking case, these would match up automatically, example test case:
oadm router --latest-images
curl -k -H "Host: allow-http.header.test" -H "now: $(date +%s)"

Comment 13 zhaozhanqi 2016-03-07 02:45:59 UTC

yes, before for container networking the router can export 'hostPort', thus container and host will have a mapping port,example:
# docker ps
CONTAINER ID        IMAGE                                                                                           COMMAND                  CREATED             STATUS              PORTS                                                              NAMES
b53732ae460b        brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-haproxy-router:v3.1.1.903   "/usr/bin/openshift-r"   2 days ago          Up 2 days                                                                              k8s_router.c86b40c7_router-1-723p8_default_c83ddf2f-e1d7-11e5-ac77-525400719108_6814bfb9
5dc1dcbc6c63        brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-pod:v3.1.1.903              "/pod"                   2 days ago          Up 2 days >80/tcp,>443/tcp,>1936/tcp   k8s_POD.9629577a_router-1-723p8_default_c83ddf2f-e1d7-11e5-ac77-525400719108_422aa0c2

you can see the mapping info:>80/tcp,>443/tcp,>1936/tcp 

then we can also access the router using host port.

but now after the https://github.com/openshift/origin/commit/241d02b8e89d492ad680559e56bb1549c9985c34
rotuer.go #line 414

the 'hostPort' has been deleted. then the container are:

# docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
5017f2393100        openshift/origin-haproxy-router:latest   "/usr/bin/openshift-r"   8 minutes ago       Up 8 minutes                            k8s_router.706c7f85_router-1-7892t_default_20c1bb85-e408-11e5-8528-0e754fd826a7_4faa5e5a
bc6c09a5d360        openshift/origin-pod:v1.1.3              "/pod"                   9 minutes ago       Up 8 minutes                            k8s_POD.909543fa_router-1-7892t_default_20c1bb85-e408-11e5-8528-0e754fd826a7_72461cb1

you can see NO mapping port between host and container.
since router container ip can NOT be accessed out of openshift cluster

So I'm wondering how to resolve the router container ip from out of openshift cluster.
I thought this will be resolve using '--ports'.

Comment 14 zhaozhanqi 2016-03-07 07:44:39 UTC

Comment 13 issue can be resolved via setting 'externalIPs' in the router service. 
I will verified this bug once it's merged to OSE

Comment 15 Ram Ranganathan 2016-03-07 18:41:45 UTC
@zhaozhanqi - thx for the update. Cool yeah this change was just to allow the port number to be configured.
The changes to container networking Clayton made were aside from this fix and now require the port to be exposed - example via a node-port service. Glad you figured that out.

Comment 16 zhaozhanqi 2016-03-22 08:26:25 UTC
@Ram Ranganathan

for hostnetwork mode router, if customer is using the specified port: eg 1081/10444.  if so, we need to tell user add iptables or Do we have another method?


1. Create router using --ports

 oadm router second --credentials=/etc/origin/master/openshift-router.kubeconfig --service-account=router --images='openshift3/ose-${component}:${version}' --stats-port=6701 --ports='1081:1081,10444:10444'

2. oc env dc second ROUTER_SERVICE_HTTP_PORT=1081
   oc env dc second ROUTER_SERVICE_HTTP_PORTS=10443

3. return no route when accessing the nodeport
   # curl
curl: (7) Failed connect to; Connection refused

Comment 17 Ram Ranganathan 2016-03-22 17:50:29 UTC
@zhaozhanqi - so this is assuming the nodes (where the router pods run) have those ports open - similar to the case for 80/443. Otherwise, no traffic is going to get in. 
I guess, we can probably add a section to the docs to mentioning to how to run routers on custom ports. Will send a PR to update the docs sometime later this week. Thx

Comment 18 Ram Ranganathan 2016-05-16 18:32:33 UTC
Added the docs changes earlier - forgot to update the bugz here. 
The docs PR was: https://github.com/openshift/openshift-docs/pull/1942 
and has been merged.

@zhaozhanqi FYI.

Comment 19 zhaozhanqi 2016-05-17 03:12:33 UTC

In this line:

should add option 'ports' when creating router ,like 

oadm router --replicas=0 --ports='10080:10080,10443:10443' 


the endpoints will still 80 and 443

# oc get endpoints router
router   ,,   22m

that's mean the hostport and container port still 80/443. 

and I doubt this will make container network router can not work

Comment 20 Ram Ranganathan 2016-05-17 19:27:18 UTC
@zhaozhanqi, true that we can clarify it a bit more (that example is for host networking but point taken re: endpoints info mismatch).
Just updated the example. PR: https://github.com/openshift/openshift-docs/pull/2115

Comment 21 zhaozhanqi 2016-05-18 05:27:39 UTC

thanks, it looks good to me.

Comment 24 Troy Dawson 2016-09-01 15:14:51 UTC
This has been merged into ose and is in OSE v3.3.0.28 or newer.

Comment 25 Meng Bo 2016-09-02 06:23:03 UTC
Checked on router image openshift3/ose-haproxy-router:v3.3.0.28 6ff97e2fd698

Issue has been fixed.


Comment 27 errata-xmlrpc 2016-09-27 09:30:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.