When creating router with `oadm router` and using the ports flag like --ports='900:900,999:999' the ROUTER_SERVICE_HTTP(S)_PORT is not set properly # oadm router example \ --ports='49999:49999,50000:50000' \ --stats-port=51936 \ --replicas=2 \ --selector="ha-router=primary" \ --selector="region=infra" \ --labels="example" \ --default-cert=example.pem \ --service-account=router The ports are set properly in the haproxy.config and the service objects ``` [root@ose-test-master-01 ~]# oc get all NAME REVISION DESIRED CURRENT TRIGGERED BY dc/example 1 2 2 config NAME DESIRED CURRENT AGE rc/example-1 2 2 15m NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/example 172.30.207.88 <none> 49999/TCP,50000/TCP,51936/TCP 15m NAME READY STATUS RESTARTS AGE po/example-1-3ctx5 1/1 Running 0 14m po/example-1-w8plh 1/1 Running 0 14m ``` However, the env in the DC is partially incorrect: ``` [root@ose-test-master-01 ~]# oc get dc example -o json | jq -c '. | .spec.template.spec.containers[].env[] | select(.name | contains("PORT")) ' {"name":"ROUTER_SERVICE_HTTPS_PORT","value":"443"} {"name":"ROUTER_SERVICE_HTTP_PORT","value":"80"} {"name":"STATS_PORT","value":"51936"} ``` The haproxy template uses those values to bind ports like this: ``` frontend public bind :{{env "ROUTER_SERVICE_HTTP_PORT" "80"}} frontend public_ssl bind :{{env "ROUTER_SERVICE_HTTPS_PORT" "443"}} ``` The to fix environment in the new router DC ``` [root@ose-test-master-01 ~]# oc set env dc/example ROUTER_SERVICE_HTTP_PORT=49999 [root@ose-test-master-01 ~]# oc set env dc/example ROUTER_SERVICE_HTTPS_PORT=50000 ``` Docker inspect shows the container is on port 80 and 443. That is because of the hardcoded `EXPOSE` directive in the Dockerfile. Based on the ENV this seems to be a non-issue. ``` [root@ose-test-node-01 ~]# docker inspect --format='{{.Config.ExposedPorts}}' dd02066d4845 map[443/tcp:{} 53/tcp:{} 80/tcp:{} 8443/tcp:{}] ``` 4. Add label for sharding to the new router ``` [root@ose-test-master-01 ~]# oc set env dc/example ROUTE_LABELS="router=example" ``` 5. Add label to existing route for sharding. ``` [root@ose-test-master-01 ~]# oc label route v3simplebottle -n testing router=example ``` 6. Profit! ``` [root@ose-test-master-01 ~]# curl http://v3simplebottle-test.os.example.com:49999 <h1> hello OpenShift </h1> [root@ose-test-master-01 ~]# curl -k https://v3simplebottle-test.os.pixar.com:50000 <h1> hello OpenShift </h1> ``` The following things `oadm router` does not do: - properly set env "ROUTER_SERVICE_HTTPS_PORT" - properly set env "ROUTER_SERVICE_HTTP_PORT" - label the generated certs secret as it labels services, dc, and seemingly everything else it created I have replicated this in my test cluster. # openshift version openshift v3.3.1.7 kubernetes v1.3.0+52492b4 etcd 2.3.0+git
I think this is "functions as designed ... but confusing". The exposed ports are set up independently of the config for port binding. In fact the exposed ports really aren't used for much since typically you are using host networking so can't remap the ports anyway. I'm dropping te priority to "low" so we can clean up the command line help to make it clear that you need to set this and the environment variables.
As you said, these are host ports, so I was expecting the `--ports` arg to make it all the way into the haproxy bind config by whatever means. The goal was to run a second proxy (router) on a unique port while continuing to leverage the same ipfailover VIPs and DNS records.
We have the following documentation on this topic. Please review it. I can make changes to clear up inaccurate or confusing discussions. Documentation: https://docs.openshift.com/container-platform/3.5/ Architecture: Environment variables: ROUTER_SERVICE_HTTPS_PORT ROUTER_SERVICE_HTTP_PORT architecture/core_concepts/routes.html#routers architecture/core_concepts/routes.html#env-variables How to set it up: install_config/router/default_haproxy_router.html#customizing-the-router-service-ports -------- # oadm router --help --ports='80:80,443:443': A comma delimited list of ports or port pairs to expose on the router pod. The default is set for HAProxy. Port pairs are applied to the service and to host ports (if specified). The above oadm help is confusing (if not wrong). I will change it.
Ok, the docs you link make sense. I think as long as we clarify the oadm help info, that would be sufficient from my pov.
origin PR 13816 https://github.com/openshift/origin/pull/13816
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/0d4d887a808e95bdd2f0d4a71e747ceeb6d48557 The --ports flag does not modify dc env variables --ports is not intended to modify the env variables. That can be done using oc env or by editing the router's dc. Rather it sets the ports that are exposed in the router's service. The help message is confusing and misleading. This change rewords the help text. bug 1420543 https://bugzilla.redhat.com/show_bug.cgi?id=1420543
verified this bug on v3.7.0-0.143.1 oadm router --help --ports='80:80,443:443': A comma delimited list of ports or port pairs that set the port in the router pod containerPort and hostPort. It also sets service port and targetPort to expose on the router pod. This does not modify the env variables. That can be done using oc env or by editing the router's dc. This is used when host-network=false.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188