Created attachment 1408765 [details] The master config, nodes, namespace, dc and pods yaml file
(In reply to Anping Li from comment #1) > Created attachment 1408765 [details] > The master config, nodes, namespace, dc and pods yaml file The 'logging' namespace/project has following annotation: openshift.io/node-selector: "" That means the pods that run in that namespace are allowed to be scheduled anywhere, including master nodes (if they are schedulable).
Additionally, the 'openshift.io/node-selector: ""' does not mean the selector is not set, it means you explicitly allowing it to schedule on every node. You have to remove that annotation entirely to fallback to master-config.
The behavioral change between 3.7 -> 3.9 was that in 3.9 we made the masters schedulable.
@Michal, Are there any document/bug with this point (it means you explicitly allowing it to schedule on every node)?
@scott, The following namespace include "openshift.io/node-selector". Could we remove it in openshift-ansible both fresh install/upgrade? oc get namespaces/default -o json| grep node-selector "openshift.io/node-selector": "", oc get namespaces/kube-service-catalog -o json |grep node-selector "openshift.io/node-selector": "", oc get namespaces/logging -o json |grep node-selector "openshift.io/node-selector": "" oc get namespaces/openshift-infra -o json | grep node-selector "openshift.io/node-selector": "", oc get namespaces/openshift-template-service-broker -o json | grep node-selector "openshift.io/node-selector": "", oc get namespaces/openshift-web-console -o json | grep node-selector "openshift.io/node-selector": "",
@Michal, @scott, The doc https://github.com/openshift/openshift-docs/pull/8197/files. According the doc, we needn't remove openshift.io/node-selector from namespace. But how to deal with the other pods? Shall we set node-selector for all of them?
My recommendation would be to separate applications to namespaces based on if they need to disable/change the default project node selector. The rest of namespaces shouldn't disable it. There are 3 use cases to change the default node I can think of: 1. DaemonSet - Logging, Prometheus, ... needs to disable the default project node selector (it has it's own node selector and commonly needs to hit more nodes that the default one allows) 2. Webconsole need to change the default project node selector to select only masters 3. Infra - if we want to run something only on infra nodes need to change the default project node selector to select only infra nodes Creating a good namespace structure an separation allows you to avoid setting explicit node selector for applications in "unrestricted" namespaces. Also less error prone. > some pods are scheduled to masters when openshift.io/node-selector="" in namespace As Michal already pointed out this is not a bug in how the default project node selector works. I'd suggest moving this to the installer component to verify/fix the labeling and distribution to particular namespaces.
Hit the same issue for registry-console after upgrade. Before upgrade. # oc get pod -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE ... default registry-console-1-f7vq7 1/1 Running 0 1h 10.2.10.2 qe-jliu-ha2-node-primary-1 ... After upgrade # oc get pod -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE ... default registry-console-2-kqf72 1/1 Running 0 28m 10.2.0.3 qe-jliu-ha2-master-etcd-2 ... # oc describe project default|grep selector Annotations: openshift.io/node-selector= # cat /etc/origin/master/master-config.yaml|grep defaultNode defaultNodeSelector: region=primary
I believe it's expected that those projects have no node selector as they're administrative projects and leverage other means to ensure their pods land on the proper nodes. This combined with setting a default node selector should ensure that only administrative pods land on the masters. I think everything is working as expected here unless I'm missing something.
@scott, yes, it is expected in v3.9. A further question is how to set default node-selector for pod under those infra project comment 8.
Created PR to limit registry-console pods to masters in 3.9: https://github.com/openshift/openshift-ansible/pull/8767
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2213