Description of problem: When you deploy a new app in OCP, the deployer pod contains a few environment variables like OPENSHIFT_MASTER & KUBERNETES_MASTER but their value is not the proper API url (in an HA environment, the LB), but just one of the masters API: $ oc new-project foobar $ oc new-app kubernetes/guestbook $ oc get pod -o yaml guestbook-1-deploy apiVersion: v1 kind: Pod ... spec: activeDeadlineSeconds: 21600 containers: - env: - name: KUBERNETES_MASTER value: https://master1.minwi.local:8443 - name: OPENSHIFT_MASTER value: https://master1.minwi.local:8443 $ oc get nodes NAME STATUS AGE master1.minwi.local Ready,SchedulingDisabled 192d master2.minwi.local Ready,SchedulingDisabled 192d master3.minwi.local Ready,SchedulingDisabled 192d node1.minwi.local Ready 192d node2.minwi.local Ready 192d nodeinfra.minwi.local Ready 192d master-config.yaml seems to be ok: [master1]# grep -i url /etc/origin/master/master-config.yaml loggingPublicURL: https://kibana.apps.minwi.com logoutURL: "" masterPublicURL: https://ocp.minwi.com:8443 metricsPublicURL: https://hawkular-metrics.apps.minwi.com/hawkular/metrics publicURL: https://ocp.minwi.com:8443/console/ urls: masterPublicURL: https://ocp.minwi.com:8443 assetPublicURL: https://ocp.minwi.com:8443/console/ masterPublicURL: https://ocp.minwi.com:8443 masterURL: https://ocp.minwi.local:8443 Version-Release number of selected component (if applicable): oc v3.5.5.26 kubernetes v1.5.2+43a9be4 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://emingueztst.eastus2.cloudapp.azure.com:8443 openshift v3.5.5.26 kubernetes v1.5.2+43a9be4 How reproducible: Create a new deployment and check the deployer pod environment variables Steps to Reproduce: 1. oc new-project foobar 2. oc new-app kubernetes/guestbook 3. oc get pod -o yaml guestbook-1-deploy apiVersion: v1 kind: Pod ... spec: activeDeadlineSeconds: 21600 containers: - env: - name: KUBERNETES_MASTER value: https://master1.minwi.local:8443 - name: OPENSHIFT_MASTER value: https://master1.minwi.local:8443 Actual results: Both variables values indicates just one of the masters instead the proper load balancer URL Expected results: Proper value for those variables. Additional info:
I am able to reproduce. From what I can tell, the master that the deploy pods use is determined by the which controller-manager on the HA masters wins the leader election. I think the controller-manager uses the /etc/origin/master/openshift-master.kubeconfig as its client config which uses itself as the server, not the LB. This means that API request from the controller-manager that won the election do not go through the LB, leading to higher stress on the master whose controller-manager is the leader. However, it probably also leads to lower latency on those requests since the master the controller-manager is talking to is colocated. I can see it either way, but the side effect on the deploy pods is unfortunate since the pod spec that the deployer controller generates uses the server from the controller-manager's client config as the OPENSHIFT_MASTER and KUBERNETES_MASTER.
sending to Master tl;dr OPENSHIFT_MASTER and KUBERNETES_MASTER env vars for builder pods are set to the master co-resident with the active controller-manager by the build controller, not the master LB.
To be clear, the supported deployer code constructs API clients using the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables. The OPENSHIFT_MASTER and KUBERNETES_MASTER variables are no longer used. Did somebody observe evidence of the deployer container actually communicating with the master outside the service IP, or was that just assumed based on the values of these defunct variables? Changing the KUBERNETES_MASTER and OPENSHIFT_MASTER variables should only have an effect on any custom deployer image code which happens to be using them (and any such usages should be ported to use KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT). Are we aware of any such usages? We should remove OPENSHIFT_MASTER and KUBERNETES_MASTER from the deployer pod container environment. If there are any compatibility concerns regarding custom deployer images, we could provide a transitional period during 3.7 where the values are set to a URL like: https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT Then we could remove the variables entirely in 3.8.
It also seems acceptable to leave the variables as-is and mark them deprecated in release notes until we're satisfied they're not in use by custom deployer/hook code.
(In reply to Dan Mace from comment #4) > To be clear, the supported deployer code constructs API clients using the > KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables. The > OPENSHIFT_MASTER and KUBERNETES_MASTER variables are no longer used. Did > somebody observe evidence of the deployer container actually communicating > with the master outside the service IP, or was that just assumed based on > the values of these defunct variables? I've just seen them as env vars in the deployer pod but to be honest, I don't know if they are used or not. > > Changing the KUBERNETES_MASTER and OPENSHIFT_MASTER variables should only > have an effect on any custom deployer image code which happens to be using > them (and any such usages should be ported to use KUBERNETES_SERVICE_HOST > and KUBERNETES_SERVICE_PORT). Are we aware of any such usages? > I have no idea. > We should remove OPENSHIFT_MASTER and KUBERNETES_MASTER from the deployer > pod container environment. If there are any compatibility concerns regarding > custom deployer images, we could provide a transitional period during 3.7 > where the values are set to a URL like: > > https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT > > Then we could remove the variables entirely in 3.8. I like to have a "transition period" just in case. Thanks
Release note to drive deprecation of these variables: https://github.com/openshift/openshift-docs/issues/4906
According the 3.7 release notes, this has been fixed, right?
KUBERNETES_MASTER and OPENSHIFT_MASTER in deployer pods were deprecated, and no further changes to their behavior will be made.