Description of problem: Because ds/machine-config-daemon defaults to rollingUpdate maxUnavialable 1 the rollout is entirely serialized and thus very slow on large clusters. We can speed the rollout of daemonsets which don't immediately affect availability by allowing the maxUnavailable to scale with cluster size. A quick test on a 250 node cluster shows that the current behavior takes around 100 minutes where as with maxUnavailable 10% it takes under 10 minutes. Version-Release number of selected component (if applicable): 4.4 How reproducible: 100% Steps to Reproduce: 1. Install a cluster that's got 20 or more hosts 2. Perform an upgrade 3. Observe that only one pod is unavailable at once and the amount of time the upgrade takes. Actual results: 1 pod unavailable at a time, slow rollout Expected results: 10% pods unavailable at most, faster / more parallel rollout Additional info:
Verified on 4.7.0-0.nightly-2020-10-26-124513. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-10-26-124513 True False 126m Cluster version is 4.7.0-0.nightly-2020-10-26-124513 $ oc -n openshift-machine-config-operator get ds/machine-config-daemon -o yaml apiVersion: apps/v1 kind: DaemonSet metadata: annotations: deprecated.daemonset.template.generation: "1" creationTimestamp: "2020-10-27T18:13:47Z" generation: 1 managedFields: - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:deprecated.daemonset.template.generation: {} f:spec: f:revisionHistoryLimit: {} f:selector: f:matchLabels: .: {} f:k8s-app: {} f:template: f:metadata: f:labels: .: {} f:k8s-app: {} f:name: {} f:spec: f:containers: k:{"name":"machine-config-daemon"}: .: {} f:args: {} f:command: {} f:env: .: {} k:{"name":"NODE_NAME"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: .: {} f:apiVersion: {} f:fieldPath: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: .: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:privileged: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/rootfs"}: .: {} f:mountPath: {} f:name: {} k:{"name":"oauth-proxy"}: .: {} f:args: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:ports: .: {} k:{"containerPort":9001,"protocol":"TCP"}: .: {} f:containerPort: {} f:hostPort: {} f:name: {} f:protocol: {} f:resources: .: {} f:requests: .: {} f:cpu: {} f:memory: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/etc/tls/cookie-secret"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/etc/tls/private"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:hostNetwork: {} f:hostPID: {} f:nodeSelector: .: {} f:kubernetes.io/os: {} f:priorityClassName: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:serviceAccount: {} f:serviceAccountName: {} f:terminationGracePeriodSeconds: {} f:tolerations: {} f:volumes: .: {} k:{"name":"cookie-secret"}: .: {} f:name: {} f:secret: .: {} f:defaultMode: {} f:secretName: {} k:{"name":"proxy-tls"}: .: {} f:name: {} f:secret: .: {} f:defaultMode: {} f:secretName: {} k:{"name":"rootfs"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} f:updateStrategy: f:rollingUpdate: .: {} f:maxUnavailable: {} f:type: {} manager: machine-config-operator operation: Update time: "2020-10-27T18:13:47Z" - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:currentNumberScheduled: {} f:desiredNumberScheduled: {} f:numberAvailable: {} f:numberReady: {} f:observedGeneration: {} f:updatedNumberScheduled: {} manager: kube-controller-manager operation: Update time: "2020-10-27T18:22:53Z" name: machine-config-daemon namespace: openshift-machine-config-operator resourceVersion: "20769" selfLink: /apis/apps/v1/namespaces/openshift-machine-config-operator/daemonsets/machine-config-daemon uid: a2793d6b-dd9e-4211-a678-ed4aa1ba7820 spec: revisionHistoryLimit: 10 selector: matchLabels: k8s-app: machine-config-daemon template: metadata: creationTimestamp: null labels: k8s-app: machine-config-daemon name: machine-config-daemon spec: containers: - args: - start command: - /usr/bin/machine-config-daemon env: - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8eace99e813cad35dea2c6663b04b377e3a0c8f3ec2c759d1a268b26937bff47 imagePullPolicy: IfNotPresent name: machine-config-daemon resources: requests: cpu: 20m memory: 50Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /rootfs name: rootfs - args: - --https-address=:9001 - --provider=openshift - --openshift-service-account=machine-config-daemon - --upstream=http://127.0.0.1:8797 - --tls-cert=/etc/tls/private/tls.crt - --tls-key=/etc/tls/private/tls.key - --cookie-secret-file=/etc/tls/cookie-secret/cookie-secret - '--openshift-sar={"resource": "namespaces", "verb": "get"}' - '--openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}' image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a13d63927ac945990c930ef63789c2f5661962dfcfc2094de0774bc06a6435 imagePullPolicy: IfNotPresent name: oauth-proxy ports: - containerPort: 9001 hostPort: 9001 name: metrics protocol: TCP resources: requests: cpu: 20m memory: 50Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/tls/private name: proxy-tls - mountPath: /etc/tls/cookie-secret name: cookie-secret dnsPolicy: ClusterFirst hostNetwork: true hostPID: true nodeSelector: kubernetes.io/os: linux priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: machine-config-daemon serviceAccountName: machine-config-daemon terminationGracePeriodSeconds: 600 tolerations: - operator: Exists volumes: - hostPath: path: / type: "" name: rootfs - name: proxy-tls secret: defaultMode: 420 secretName: proxy-tls - name: cookie-secret secret: defaultMode: 420 secretName: cookie-secret updateStrategy: rollingUpdate: maxUnavailable: 10% type: RollingUpdate status: currentNumberScheduled: 6 desiredNumberScheduled: 6 numberAvailable: 6 numberMisscheduled: 0 numberReady: 6 observedGeneration: 1 updatedNumberScheduled: 6
Should not need a doc update as this only modifies how fast the updated pods themselves (not the pools) are rolled out
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633