Bug 1846669 - misleading scheduler.spec.defaultNodeSelector & pod.spec.nodeSelector
Summary: misleading scheduler.spec.defaultNodeSelector & pod.spec.nodeSelector
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-scheduler
Version: 4.4
Hardware: All
OS: All
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Mike Dame
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On:
Blocks: 1846671
TreeView+ depends on / blocked
 
Reported: 2020-06-12 14:36 UTC by Olimp Bockowski
Modified: 2020-10-27 16:07 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:06:58 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift api pull 663 0 None closed Bug 1846669: Update defaultNodeSelector description to clarify its behavior with existing nodeSelectors 2021-02-02 10:54:09 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:07:22 UTC

Internal Links: 1846671

Description Olimp Bockowski 2020-06-12 14:36:13 UTC
Description of problem:

It looks like scheduler.spec.defaultNodeSelector + pod.spec.nodeSelector are intersection (less likely union)
When a "nodeSelector" is already defined for the resource, the scheduler will still append the "defaultNodeSelector" value to the pod resource. This causes the pod to be "unschedulable" if (what is common) no node matches both selectors.
However, descriptions are misleading and one could think that it works in a different way.  

Version-Release number of selected component (if applicable):
OCP 4.x (atm 4.4 the latest)

Comment 2 Mike Dame 2020-06-15 17:39:21 UTC
Adding to this from offline discussion, to clarify the problem doesn't seem to be with the scheduler itself but the description that's provided here: https://github.com/openshift/api/blob/d0b31d707c464221d1eb24846b0d0bbe57040102/config/v1/types_scheduling.go#L31-L51

Will open a PR to update this

Comment 5 RamaKasturi 2020-06-22 11:46:11 UTC
Hi Mike,

   Saw that bug fix here is only the description and nothing related to the code itself, Below are the steps i performed to verify the bug, can you please take a look and let me know if the verification looks good ? Thanks !!

Test 1:
================================
1) Add defaultNodeSelector in the scheduler spec as below:
 spec:
  defaultNodeSelector: disktype=ssd
  mastersSchedulable: false
  policy:
    name: ""
status: {}

2) Add the label disktype=ssd to one of the node in the cluster by running the command below:
oc label nodes node1 disktype=sdd

3) Now schedule a pod using the spec below and see that pod gets scheduled.
[ramakasturinarra@dhcp35-60 ~]$ cat podtest.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx1
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

4) pod gets scheduled and runs with out any issues.

Test 2:
======================================
1) Keep the defaultNodeSelector in the scheduler spec
2) Now add the label to the node in the cluster by running the command below:
oc label nodes node2 disktype=hdd

3) Now schedule a pod using the spec below and see that pod gets scheduled.
[ramakasturinarra@dhcp35-60 ~]$ cat podtest.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx1
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    disktype: hdd

4) Verify that pod runs successfully on node2.

nginx1   1/1     Running   0          10s     x.x.x.x   <node2>  <none>           <none>

Comment 6 RamaKasturi 2020-06-22 11:46:58 UTC
Verified with the payload below:

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-06-20-011219   True        False         3h49m   Cluster version is 4.6.0-0.nightly-2020-06-20-011219

Comment 7 Mike Dame 2020-06-22 14:25:23 UTC
Yes, we can't change anything with the actual behavior for this bug so it is only testing that the description of this field now matches the observed behavior.

Comment 8 RamaKasturi 2020-06-23 10:28:19 UTC
Verified with the payload below and i see that description of the field now matches the observed behaviour.

[ramakasturinarra@dhcp35-60 verification-tests]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-06-23-053310   True        False         64m     Cluster version is 4.6.0-0.nightly-2020-06-23-053310


Tested the description by reading the "defaultNodeSelector" by running the command "oc get crd schedulers.config.openshift.io -o yaml" file and as per comment 5 the behaviour matches as well.

Based on the above moving the bug to verified state.

Comment 10 errata-xmlrpc 2020-10-27 16:06:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.