Bug 1420543 - The --ports flag does not modify dc environment variables
The --ports flag does not modify dc environment variables
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
Unspecified Unspecified
low Severity low
: ---
: ---
Assigned To: Phil Cameron
Depends On:
  Show dependency treegraph
Reported: 2017-02-08 17:56 EST by Steven Walter
Modified: 2017-05-08 10:02 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Steven Walter 2017-02-08 17:56:26 EST
When creating router with `oadm router` and using the ports flag like --ports='900:900,999:999' the ROUTER_SERVICE_HTTP(S)_PORT is not set properly

#  oadm router example \
    --ports='49999:49999,50000:50000' \
    --stats-port=51936 \
    --replicas=2 \
    --selector="ha-router=primary" \
    --selector="region=infra" \
    --labels="example" \
    --default-cert=example.pem \

The ports are set properly in the haproxy.config and the service objects

[root@ose-test-master-01 ~]# oc get all
NAME                            REVISION        DESIRED       CURRENT   TRIGGERED BY
dc/example           1               2             2         config
NAME                            DESIRED         CURRENT       AGE
rc/example-1         2               2             15m
NAME                            CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
svc/example   <none>        49999/TCP,50000/TCP,51936/TCP   15m
NAME                            READY           STATUS        RESTARTS                        AGE
po/example-1-3ctx5   1/1             Running       0                               14m
po/example-1-w8plh   1/1             Running       0                               14m

However,  the env in the DC is partially incorrect:

[root@ose-test-master-01 ~]# oc get dc example -o json | jq -c '. | .spec.template.spec.containers[].env[] | select(.name | contains("PORT")) '

The haproxy template uses those values to bind ports like this:

frontend public
  bind :{{env "ROUTER_SERVICE_HTTP_PORT" "80"}}
frontend public_ssl
  bind :{{env "ROUTER_SERVICE_HTTPS_PORT" "443"}}

The to fix environment in the new router DC

[root@ose-test-master-01 ~]# oc set env dc/example  ROUTER_SERVICE_HTTP_PORT=49999
[root@ose-test-master-01 ~]# oc set env dc/example  ROUTER_SERVICE_HTTPS_PORT=50000

Docker inspect shows the container is on port 80 and 443. That is because of the hardcoded `EXPOSE` directive in the Dockerfile. Based on the ENV this seems to be a non-issue.

[root@ose-test-node-01 ~]# docker inspect --format='{{.Config.ExposedPorts}}'  dd02066d4845
map[443/tcp:{} 53/tcp:{} 80/tcp:{} 8443/tcp:{}]

4. Add label for sharding to the new router

[root@ose-test-master-01 ~]# oc set env dc/example  ROUTE_LABELS="router=example"

5. Add label to existing route for sharding.

[root@ose-test-master-01 ~]# oc label route v3simplebottle -n testing router=example

6. Profit!

[root@ose-test-master-01 ~]# curl http://v3simplebottle-test.os.example.com:49999
<h1> hello OpenShift </h1>
[root@ose-test-master-01 ~]#  curl -k https://v3simplebottle-test.os.pixar.com:50000
<h1> hello OpenShift </h1>

The following things `oadm router` does not do:

- properly set env "ROUTER_SERVICE_HTTPS_PORT"
- properly set env "ROUTER_SERVICE_HTTP_PORT"
- label the generated certs secret as it labels services, dc, and seemingly everything else it created

I have replicated this in my test cluster.

# openshift version
openshift v3.3.1.7
kubernetes v1.3.0+52492b4
etcd 2.3.0+git
Comment 1 Ben Bennett 2017-02-09 08:50:13 EST
I think this is "functions as designed ... but confusing".  The exposed ports are set up independently of the config for port binding.  In fact the exposed ports really aren't used for much since typically you are using host networking so can't remap the ports anyway.

I'm dropping te priority to "low" so we can clean up the command line help to make it clear that you need to set this and the environment variables.
Comment 2 dlbewley 2017-02-09 11:05:27 EST
As you said, these are host ports, so I was expecting the `--ports` arg to make it all the way into the haproxy bind config by whatever means.

The goal was to run a second proxy (router) on a unique port while continuing to leverage the same ipfailover VIPs and DNS records.
Comment 3 Phil Cameron 2017-04-19 11:12:52 EDT
We have the following documentation on this topic. Please review it. I can make changes to clear up inaccurate or confusing discussions.



How to set it up:

# oadm router --help
      --ports='80:80,443:443': A comma delimited list of ports or port pairs to expose on the router pod. The default is
set for HAProxy. Port pairs are applied to the service and to host ports (if specified).

The above oadm help is confusing (if not wrong). I will change it.
Comment 4 Steven Walter 2017-05-08 10:02:55 EDT
Ok, the docs you link make sense. I think as long as we clarify the oadm help info, that would be sufficient from my pov.

Note You need to log in before you can comment on or make changes to this bug.