1846358 – Machine Config Daemon Daemon Set does not set universal Toleration (and therefore gets booted if taints are set on a node)

Bug 1846358 - Machine Config Daemon Daemon Set does not set universal Toleration (and therefore gets booted if taints are set on a node)

Summary: Machine Config Daemon Daemon Set does not set universal Toleration (and there...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Machine Config Operator
Sub Component:
Version:	4.2.z
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.3.z
Assignee:	Antonio Murdaca
QA Contact:	Michael Nguyen
Docs Contact:
URL:
Whiteboard:
Depends On:	1846357
Blocks:
TreeView+	depends on / blocked

Reported:	2020-06-11 12:38 UTC by OpenShift BugZilla Robot
Modified:	2024-03-25 16:02 UTC (History)
CC List:	27 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-08-05 10:54:08 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift machine-config-operator pull 1816	0	None	closed	[release-4.3] Bug 1846358: MCD: tolerate all taints	2021-02-04 04:11:16 UTC
Red Hat Product Errata	RHBA-2020:3180	0	None	None	None	2020-08-05 10:54:47 UTC

Comment 5 Michael Nguyen 2020-07-13 16:05:03 UTC

Verified on 4.3.0-0.nightly-2020-07-12-052232

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-07-12-052232   True        False         57m     Cluster version is 4.3.0-0.nightly-2020-07-12-052232
$ oc -n openshift-machine-config-operator get ds machine-config-daemon -o yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  creationTimestamp: "2020-07-13T14:27:04Z"
  generation: 1
  name: machine-config-daemon
  namespace: openshift-machine-config-operator
  resourceVersion: "19385"
  selfLink: /apis/extensions/v1beta1/namespaces/openshift-machine-config-operator/daemonsets/machine-config-daemon
  uid: 157a8d96-86e9-4b6f-8ab5-3191464fe804
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: machine-config-daemon
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: machine-config-daemon
      name: machine-config-daemon
    spec:
      containers:
      - args:
        - start
        command:
        - /usr/bin/machine-config-daemon
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:352d56537cf89b5b693af564c7338907d69efbcdc0866bab85d331cb759c99cf
        imagePullPolicy: IfNotPresent
        name: machine-config-daemon
        resources:
          requests:
            cpu: 20m
            memory: 50Mi
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /rootfs
          name: rootfs
      - args:
        - --https-address=:9001
        - --provider=openshift
        - --openshift-service-account=machine-config-daemon
        - --upstream=http://127.0.0.1:8797
        - --tls-cert=/etc/tls/private/tls.crt
        - --tls-key=/etc/tls/private/tls.key
        - --cookie-secret-file=/etc/tls/cookie-secret/cookie-secret
        - '--openshift-sar={"resource": "namespaces", "verb": "get"}'
        - '--openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e7e5835605c605626f77ddc2f2d68fa8928ee819ee1464dd80f2e84dce98dff
        imagePullPolicy: IfNotPresent
        name: oauth-proxy
        ports:
        - containerPort: 9001
          hostPort: 9001
          name: metrics
          protocol: TCP
        resources:
          requests:
            cpu: 20m
            memory: 50Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/tls/private
          name: proxy-tls
        - mountPath: /etc/tls/cookie-secret
          name: cookie-secret
      dnsPolicy: ClusterFirst
      hostNetwork: true
      hostPID: true
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: machine-config-daemon
      serviceAccountName: machine-config-daemon
      terminationGracePeriodSeconds: 600
      tolerations:
      - operator: Exists
      volumes:
      - hostPath:
          path: /
          type: ""
        name: rootfs
      - name: proxy-tls
        secret:
          defaultMode: 420
          secretName: proxy-tls
      - name: cookie-secret
        secret:
          defaultMode: 420
          secretName: cookie-secret
  templateGeneration: 1
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 6
  desiredNumberScheduled: 6
  numberAvailable: 6
  numberMisscheduled: 0
  numberReady: 6
  observedGeneration: 1
  updatedNumberScheduled: 6
$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-128-54.us-west-2.compute.internal    Ready    master   94m   v1.16.2+117aea2
ip-10-0-132-57.us-west-2.compute.internal    Ready    worker   67m   v1.16.2+117aea2
ip-10-0-145-52.us-west-2.compute.internal    Ready    master   93m   v1.16.2+117aea2
ip-10-0-151-169.us-west-2.compute.internal   Ready    worker   67m   v1.16.2+117aea2
ip-10-0-161-193.us-west-2.compute.internal   Ready    worker   68m   v1.16.2+117aea2
ip-10-0-168-239.us-west-2.compute.internal   Ready    master   94m   v1.16.2+117aea2
$ oc adm taint node ip-10-0-132-57.us-west-2.compute.internal infra=reserved:NoExecute; oc adm taint ip-10-0-132-57.us-west-2.compute.internal infra=reserved:NoSchedule
node/ip-10-0-132-57.us-west-2.compute.internal tainted
error: invalid resource type ip-10-0-132-57.us-west-2.compute.internal, only ["node" "nodes"] are supported
$  oc adm taint node ip-10-0-132-57.us-west-2.compute.internal infra=reserved:NoSchedule
node/ip-10-0-132-57.us-west-2.compute.internal tainted
$ oc get node/ip-10-0-132-57.us-west-2.compute.internal -o yaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    machine.openshift.io/machine: openshift-machine-api/mnguyen43-qzhm9-worker-us-west-2a-t4dsf
    machineconfiguration.openshift.io/currentConfig: rendered-worker-7e4563a06b4ebbce4b4cb3ac4993a32a
    machineconfiguration.openshift.io/desiredConfig: rendered-worker-7e4563a06b4ebbce4b4cb3ac4993a32a
    machineconfiguration.openshift.io/reason: ""
    machineconfiguration.openshift.io/state: Done
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2020-07-13T14:52:09Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: m5.large
    beta.kubernetes.io/os: linux
    failure-domain.beta.kubernetes.io/region: us-west-2
    failure-domain.beta.kubernetes.io/zone: us-west-2a
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: ip-10-0-132-57
    kubernetes.io/os: linux
    node-role.kubernetes.io/worker: ""
    node.openshift.io/os_id: rhcos
  name: ip-10-0-132-57.us-west-2.compute.internal
  resourceVersion: "40352"
  selfLink: /api/v1/nodes/ip-10-0-132-57.us-west-2.compute.internal
  uid: 858d0c40-3ad0-42af-ba37-49d0a0a17be5
spec:
  providerID: aws:///us-west-2a/i-04827373ad5498889
  taints:
  - effect: NoSchedule
    key: infra
    value: reserved
  - effect: NoExecute
    key: infra
    value: reserved
status:
  addresses:
  - address: 10.0.132.57
    type: InternalIP
  - address: ip-10-0-132-57.us-west-2.compute.internal
    type: Hostname
  - address: ip-10-0-132-57.us-west-2.compute.internal
    type: InternalDNS
  allocatable:
    attachable-volumes-aws-ebs: "25"
    cpu: 1500m
    ephemeral-storage: "114381692328"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 6796216Ki
    pods: "250"
  capacity:
    attachable-volumes-aws-ebs: "25"
    cpu: "2"
    ephemeral-storage: 125277164Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 7947192Ki
    pods: "250"
  conditions:
  - lastHeartbeatTime: "2020-07-13T16:02:02Z"
    lastTransitionTime: "2020-07-13T14:52:09Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2020-07-13T16:02:02Z"
    lastTransitionTime: "2020-07-13T14:52:09Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2020-07-13T16:02:02Z"
    lastTransitionTime: "2020-07-13T14:52:09Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2020-07-13T16:02:02Z"
    lastTransitionTime: "2020-07-13T14:53:30Z"
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea2af4ca73d2c9e90aad155b47a0716b0d1bcec49ea3e4688051c4e663032268
    - <none>:<none>
    sizeBytes: 723644729
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b73442119b08f16869fc7e095d18802c0c4810e796c78d70d387e7ac371518c
    - <none>:<none>
    sizeBytes: 467961983
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa7ba380184d08ad9b8e116e550fce50c3fce86c289791bd4a02bcbc42733d0b
    - <none>:<none>
    sizeBytes: 438174647
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:352d56537cf89b5b693af564c7338907d69efbcdc0866bab85d331cb759c99cf
    - <none>:<none>
    sizeBytes: 420378743
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ca112ef33d8c84f18fc107cd304dd3de8ef1cc49e789f8059690853cfe80349
    - <none>:<none>
    sizeBytes: 372884983
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12731d48915cb01b381cb1135fcab224a0341713725161946e5d508b8a007d50
    - <none>:<none>
    sizeBytes: 369057138
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b1759194e9a8ae6065faae70965408956402805835cfbeabe9c1619f9a9e4bd
    - <none>:<none>
    sizeBytes: 351712488
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed380c7881f4971c83e891eee3a825594cc24405976fe228fc37c8a23e0d59bb
    - <none>:<none>
    sizeBytes: 341357933
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2205664361423eb6b18ecf1ff9881e82415b8093522a81938c02a59f901d602
    - <none>:<none>
    sizeBytes: 333960108
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abf5f37beebee88f098083ac40794a088fe2a03d0c172d6ccec42bb9c222a9a6
    - <none>:<none>
    sizeBytes: 320494886
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed3dcc2ff4f5c7fcb47139c4949d2c17149ae71eacab98fb204e1c50f93bec1a
    - <none>:<none>
    sizeBytes: 315282389
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ad97beeaba6e15b4f30fbe7c298788a9c3a8e210cf591f6b3bd4c3073fb8615
    - <none>:<none>
    sizeBytes: 312743594
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:434e35365acf9773d0df5ec8cfa69b1bed73bcb978da90000f2da67937a539d6
    - <none>:<none>
    sizeBytes: 309405044
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:02aaa70524c7fb426f23332fe70e868ccf4cc51053985052fdf8e04fc7942e94
    - <none>:<none>
    sizeBytes: 302077511
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3729a05e6a67e4ed502b266b8675baf5f2b806b71e2ada9060d4fa981c7ca5cd
    - <none>:<none>
    sizeBytes: 286095367
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53c647fc5c3a1c63267df314ae989700960d0f49675422ae716beef88732e573
    - <none>:<none>
    sizeBytes: 277966135
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e7e5835605c605626f77ddc2f2d68fa8928ee819ee1464dd80f2e84dce98dff
    - <none>:<none>
    sizeBytes: 271558107
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5495b7564ed27e820909e5a1ff427dea40a247f305281b26187bc0ce59cfecac
    - <none>:<none>
    sizeBytes: 258187813
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7564b93348848bec986df68347a99cfb09b88c4b09afa368128ffbc3216bbc25
    - <none>:<none>
    sizeBytes: 257157194
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e96b4335db1b080a5858726c9548a3e3822913f945895a1a2185e5905ce6d24
    - <none>:<none>
    sizeBytes: 256093557
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:305ba5c27aeb9feb8e3b81e99743cbd330e4bd54990ba446882b9205708f4a5d
    - <none>:<none>
    sizeBytes: 250971559
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4afb2d3aa2f79649bebbcae6893d62ff7dce62c7e3dfe031602c22d49e4b988d
    - <none>:<none>
    sizeBytes: 238959073
  - names:
    - quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b430486c364f8235333434b497d2f1a91d8fe8a5a2acd2d5f5d1513d52c7cc10
    - <none>:<none>
    sizeBytes: 237288172
  nodeInfo:
    architecture: amd64
    bootID: a2efb88d-f2ba-43b5-a62b-eed33c28a399
    containerRuntimeVersion: cri-o://1.16.6-17.rhaos4.3.git4936f44.el8
    kernelVersion: 4.18.0-147.20.1.el8_1.x86_64
    kubeProxyVersion: v1.16.2+117aea2
    kubeletVersion: v1.16.2+117aea2
    machineID: ec221c0c041e13159a57559c9b606558
    operatingSystem: linux
    osImage: Red Hat Enterprise Linux CoreOS 43.81.202007110253.0 (Ootpa)
    systemUUID: ec221c0c-041e-1315-9a57-559c9b606558
$ oc get pods -n openshift-machine-config-operator  --field-selector spec.nodeName=ip-10-0-132-57.us-west-2.compute.internal
NAME                          READY   STATUS    RESTARTS   AGE
machine-config-daemon-np6d4   2/2     Running   0          71m
$ oc  -n openshift-machine-config-operator  get ds
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
machine-config-daemon   6         6         6       6            6           kubernetes.io/os=linux            96m
machine-config-server   3         3         3       3            3           node-role.kubernetes.io/master=   95m

Comment 7 errata-xmlrpc 2020-08-05 10:54:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.3.31 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3180

Note You need to log in before you can comment on or make changes to this bug.

amurdaca
andcosta
clasohm
cmarches
cruhm
cshulman
enorling
insights-rule-dev
jmalde
jnaess
kgarriso
ktadimar
lbac
mharri
mvardhan
nstephan
obockows
palonsor
pamoedom
rahmed
rdomnu
rekhan
scott.worthington
smilner
sreber
susuresh
vfarias