Bug 1420543 - The --ports flag does not modify dc environment variables
Summary: The --ports flag does not modify dc environment variables
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.7.0
Assignee: Phil Cameron
QA Contact: zhaozhanqi
Depends On:
TreeView+ depends on / blocked
Reported: 2017-02-08 22:56 UTC by Steven Walter
Modified: 2022-08-04 22:20 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2017-11-28 21:52:23 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Steven Walter 2017-02-08 22:56:26 UTC
When creating router with `oadm router` and using the ports flag like --ports='900:900,999:999' the ROUTER_SERVICE_HTTP(S)_PORT is not set properly

#  oadm router example \
    --ports='49999:49999,50000:50000' \
    --stats-port=51936 \
    --replicas=2 \
    --selector="ha-router=primary" \
    --selector="region=infra" \
    --labels="example" \
    --default-cert=example.pem \

The ports are set properly in the haproxy.config and the service objects

[root@ose-test-master-01 ~]# oc get all
NAME                            REVISION        DESIRED       CURRENT   TRIGGERED BY
dc/example           1               2             2         config
NAME                            DESIRED         CURRENT       AGE
rc/example-1         2               2             15m
NAME                            CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
svc/example   <none>        49999/TCP,50000/TCP,51936/TCP   15m
NAME                            READY           STATUS        RESTARTS                        AGE
po/example-1-3ctx5   1/1             Running       0                               14m
po/example-1-w8plh   1/1             Running       0                               14m

However,  the env in the DC is partially incorrect:

[root@ose-test-master-01 ~]# oc get dc example -o json | jq -c '. | .spec.template.spec.containers[].env[] | select(.name | contains("PORT")) '

The haproxy template uses those values to bind ports like this:

frontend public
  bind :{{env "ROUTER_SERVICE_HTTP_PORT" "80"}}
frontend public_ssl
  bind :{{env "ROUTER_SERVICE_HTTPS_PORT" "443"}}

The to fix environment in the new router DC

[root@ose-test-master-01 ~]# oc set env dc/example  ROUTER_SERVICE_HTTP_PORT=49999
[root@ose-test-master-01 ~]# oc set env dc/example  ROUTER_SERVICE_HTTPS_PORT=50000

Docker inspect shows the container is on port 80 and 443. That is because of the hardcoded `EXPOSE` directive in the Dockerfile. Based on the ENV this seems to be a non-issue.

[root@ose-test-node-01 ~]# docker inspect --format='{{.Config.ExposedPorts}}'  dd02066d4845
map[443/tcp:{} 53/tcp:{} 80/tcp:{} 8443/tcp:{}]

4. Add label for sharding to the new router

[root@ose-test-master-01 ~]# oc set env dc/example  ROUTE_LABELS="router=example"

5. Add label to existing route for sharding.

[root@ose-test-master-01 ~]# oc label route v3simplebottle -n testing router=example

6. Profit!

[root@ose-test-master-01 ~]# curl http://v3simplebottle-test.os.example.com:49999
<h1> hello OpenShift </h1>
[root@ose-test-master-01 ~]#  curl -k https://v3simplebottle-test.os.pixar.com:50000
<h1> hello OpenShift </h1>

The following things `oadm router` does not do:

- properly set env "ROUTER_SERVICE_HTTPS_PORT"
- properly set env "ROUTER_SERVICE_HTTP_PORT"
- label the generated certs secret as it labels services, dc, and seemingly everything else it created

I have replicated this in my test cluster.

# openshift version
openshift v3.3.1.7
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

Comment 1 Ben Bennett 2017-02-09 13:50:13 UTC
I think this is "functions as designed ... but confusing".  The exposed ports are set up independently of the config for port binding.  In fact the exposed ports really aren't used for much since typically you are using host networking so can't remap the ports anyway.

I'm dropping te priority to "low" so we can clean up the command line help to make it clear that you need to set this and the environment variables.

Comment 2 dlbewley 2017-02-09 16:05:27 UTC
As you said, these are host ports, so I was expecting the `--ports` arg to make it all the way into the haproxy bind config by whatever means.

The goal was to run a second proxy (router) on a unique port while continuing to leverage the same ipfailover VIPs and DNS records.

Comment 3 Phil Cameron 2017-04-19 15:12:52 UTC
We have the following documentation on this topic. Please review it. I can make changes to clear up inaccurate or confusing discussions.



How to set it up:

# oadm router --help
      --ports='80:80,443:443': A comma delimited list of ports or port pairs to expose on the router pod. The default is
set for HAProxy. Port pairs are applied to the service and to host ports (if specified).

The above oadm help is confusing (if not wrong). I will change it.

Comment 4 Steven Walter 2017-05-08 14:02:55 UTC
Ok, the docs you link make sense. I think as long as we clarify the oadm help info, that would be sufficient from my pov.

Comment 5 Phil Cameron 2017-06-14 17:53:59 UTC
origin PR 13816

Comment 6 openshift-github-bot 2017-06-28 19:27:25 UTC
Commit pushed to master at https://github.com/openshift/origin

The --ports flag does not modify dc env variables

--ports is not intended to modify the env variables. That can
be done using oc env or by editing the router's dc. Rather it
sets the ports that are exposed in the router's service.

The help message is confusing and misleading.  This change
rewords the help text.

bug 1420543

Comment 9 zhaozhanqi 2017-10-09 02:22:45 UTC
verified this bug on v3.7.0-0.143.1

oadm router --help

      --ports='80:80,443:443': A comma delimited list of ports or port pairs that set the port in the router pod
containerPort and hostPort. It also sets service port and targetPort to expose on the router pod. This does not modify
the env variables. That can be done using oc env or by editing the router's dc. This is used when host-network=false.

Comment 13 errata-xmlrpc 2017-11-28 21:52:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.