Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1557295

Summary: The default nodeSelector "node-role.kubernetes.io/compute=true" blocks daemonset create pods on master node
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MasterAssignee: Tomáš Nožička <tnozicka>
Status: CLOSED DUPLICATE QA Contact: Wang Haoran <haowang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, jokerman, mmccomas
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-16 13:23:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2018-03-16 11:40:06 UTC 
Description of problem:
This defect is split from https://bugzilla.redhat.com/show_bug.cgi?id=1548641#c21
the default node-selector is node-role.kubernetes.io/compute=true, there is a problem, see the following:

1) Create one project named 'hosa', node-selector is ""
# oc get project -n hosa -o yaml
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/description: ""
    openshift.io/display-name: ""
    openshift.io/requester: system:admin
    openshift.io/sa.scc.mcs: s0:c11,c10
    openshift.io/sa.scc.supplemental-groups: 1000130000/10000
    openshift.io/sa.scc.uid-range: 1000130000/10000
  creationTimestamp: 2018-03-16T09:42:39Z
*********************snipped**************************************

2) I have one hawkular-openshift-agent ds, which node-selector is not defined, this means it should start up one pod in each node, and there is not "node-role.kubernetes.io/compute=true" label on master node, the ds actually check if the master node is labeled with node-selector "node-role.kubernetes.io/compute=true", but it is not, so the hawkular-openshift-agent pod on master node would be recreated continuesly(defect is filed, https://bugzilla.redhat.com/show_bug.cgi?id=1543727), see the event

Checked the running pod, its nodeSelector is node-role.kubernetes.io/compute: "true"

# oc get hawkular-openshift-agent-6s6x4 -o yaml | grep nodeSelector -A 1
  nodeSelector:
    node-role.kubernetes.io/compute: "true"


Since the master node is not labeled with node-selector "node-role.kubernetes.io/compute=true", the pod on master node would be recreated continuesly

# oc get event
54m         54m          1         hawkular-openshift-agent-l88s4.151b51d4b370b454   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8bmg.151b4f89ec04badc   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
18m         18m          1         hawkular-openshift-agent-l8bnq.151b53d58897ac5d   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
18m         18m          1         hawkular-openshift-agent-l8cdn.151b53d37068bc5b   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8ct8.151b50c721482011   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
49m         49m          1         hawkular-openshift-agent-l8f9g.151b521a453dd798   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8gfg.151b5147fa1961db   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
31m         31m          1         hawkular-openshift-agent-l8gk9.151b531a79eff10e   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
18m         18m          1         hawkular-openshift-agent-l8hdz.151b53d2464b812a   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8hxs.151b4f15bcaf15af   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8m97.151b50156eaba408   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8mwv.151b4f3e049f205a   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8ndn.151b4f65e0b4f99a   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8nsz.151b4f34c26ea5c4   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed
1h          1h           1         hawkular-openshift-agent-l8r9w.151b4e61aaf37da9   Pod                     Warning   MatchNodeSelector   kubelet, 172.16.120.93   Predicate MatchNodeSelector failed


# oc get ds -n hosa
NAME                       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
hawkular-openshift-agent   2         2         1         2            1           <none>          15h


# oc get po -n hosa -o wide
NAME                             READY     STATUS    RESTARTS   AGE       IP            NODE
hawkular-openshift-agent-6s6x4   1/1       Running   0          16h       10.129.0.26   172.16.120.78
hawkular-openshift-agent-j2wq4   0/1       Pending   0          0s        <none>        172.16.120.93

# oc get po -n hosa -o wide
NAME                             READY     STATUS    RESTARTS   AGE       IP            NODE
hawkular-openshift-agent-6s6x4   1/1       Running   0          16h       10.129.0.26   172.16.120.78
hawkular-openshift-agent-hrbhf   0/1       Pending   0          0s        <none>        172.16.120.93


# oc get node --show-labels
NAME            STATUS    ROLES     AGE       VERSION             LABELS
172.16.120.78   Ready     compute   18h       v1.9.1+a0ce1bc657   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=431ac1fb-1463-4527-b3d1-79245dd698e1,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=regionOne,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=172.16.120.78,node-role.kubernetes.io/compute=true,registry=enabled,role=node,router=enabled
172.16.120.93   Ready     master    18h       v1.9.1+a0ce1bc657   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=431ac1fb-1463-4527-b3d1-79245dd698e1,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=regionOne,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=172.16.120.93,node-role.kubernetes.io/master=true,openshift-infra=apiserver,role=node

Version-Release number of selected component (if applicable):
openshift v3.9.9
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16


How reproducible:
Always

Steps to Reproduce:
1. See the description part
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Junqi Zhao 2018-03-16 11:41:31 UTC 
correction

1) Create one project named 'hosa', node-selector is ""

change to
1) Create one project named 'hosa', node-selector is not defined in project
Comment 3 Tomáš Nožička 2018-03-16 13:23:34 UTC 
> I have one hawkular-openshift-agent ds, which node-selector is not defined, this means it should start up one pod in each node

Before you start using daemonset you need to make sure to disable project default node selector in your namespace.

Documentation for 3.9.0 is being written here https://github.com/openshift/openshift-docs/pull/8125


oc patch namespace myproject -p \
   '{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'

*** This bug has been marked as a duplicate of bug 1501514 ***