Bug 1933159

Summary: multus DaemonSets should use maxUnavailable: 33%
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: NetworkingAssignee: Douglas Smith <dosmith>
Networking sub component: multus QA Contact: Weibin Liang <weliang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: ccoleman, dosmith
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:48:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1941212    

Description W. Trevor King 2021-02-25 20:29:32 UTC
Bug 1920209 made multus rollouts slightly faster, but they're still serial.  We want maxUnavailable: 10%, so we scale better on clusters with large node-counts.  Checking a recent 4.8 CI release:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.8/1364411741282242560/artifacts/e2e-gcp/daemonsets.json | jq -r '.items[] | select(.spec.template.spec.nodeSelector["node-role.kubernetes.io/master"] != "" and .spec.updateStrategy.rollingUpdate.maxUnavailable != "10%") | .metadata.namespace + " " + .metadata.name + " " + (.spec.template.spec.nodeSelector | tostring) + " " + (.spec.updateStrategy | tostring)'
...
openshift-multus multus {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-multus network-metrics-daemon {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
...

And we want to empty that query out before 4.8 GAs.

Comment 2 W. Trevor King 2021-02-26 22:55:53 UTC
cluster-network-operator#992 looks like it will address openshift-multus/multus, but per comment 0, this bug is also asking for openshift-multus/network-metrics-daemon to be bumped (around [1]).

[1]: https://github.com/openshift/cluster-network-operator/blob/dfea1dc53ed82f5e4a4e85b7e8e863af3e4bd54a/bindata/network/network-metrics/001-daemonset.yaml#L17

Comment 3 Douglas Smith 2021-03-01 19:03:15 UTC
Thanks W. Trevor, also for the file pointer. I've gone ahead and updated cluster-network-operator#992 to update the network-metrics-daemon as well.

Comment 5 Weibin Liang 2021-03-18 14:12:56 UTC
Used latest from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.8, the curl result is still not empty.

[weliang@weliang ~]$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.8/1366818372670459904/artifacts/e2e-gcp/daemonsets.json | jq -r '.items[] | select(.spec.template.spec.nodeSelector["node-role.kubernetes.io/master"] != "" and .spec.updateStrategy.rollingUpdate.maxUnavailable != "10%") | .metadata.namespace + " " + .metadata.name + " " + (.spec.template.spec.nodeSelector | tostring) + " " + (.spec.updateStrategy | tostring)'

...
openshift-cluster-csi-drivers gcp-pd-csi-driver-node {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-image-registry node-ca {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-multus multus {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-multus network-metrics-daemon {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-network-diagnostics network-check-target {"beta.kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-sdn ovs {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
openshift-sdn sdn {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}
...


@dosmith Is above curl result expected?

Comment 6 W. Trevor King 2021-03-18 20:41:26 UTC
The origin tests allow 33% as well as 10%, but your curl is still showing 1, which we don't want.  Looking at 4.8.0-0.nightly-2021-03-18-075013 [1,2], I see:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.8/1372456130126548992/artifacts/e2e-aws/daemonsets.json | jq -r '.items[] | select(.spec.template.spec.nodeSelector["node-role.kubernetes.io/master"] != "" and .spec.updateStrategy.rollingUpdate.maxUnavailable != "10%") | .metadata.namespace + " " + .metadata.name + " " + (
.spec.template.spec.nodeSelector | tostring) + " " + (.spec.updateStrategy | tostring)'
openshift-multus network-metrics-daemon {"kubernetes.io/os":"linux"} {"rollingUpdate":{"maxUnavailable":"33%"},"type":"RollingUpdate"}

So that's not empty, but the 33% is fine.

[1]: https://amd64.ocp.releases.ci.openshift.org/releasestream/4.8.0-0.nightly/release/4.8.0-0.nightly-2021-03-18-075013
[2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.8/1372456130126548992

Comment 7 Douglas Smith 2021-03-19 13:17:38 UTC
I'm seeing that the Multus daemonset is 10% (which is I think the conclusion that we came to in the PR), I'm using a cluster-bot ci cluster (4.8.0-0.ci-2021-03-19-084923), the steps I took to check are:

```
oc project openshift-multus
oc get daemonset multus -o yaml | grep -A3 -Pi "[^:]updateStrategy:"
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate
```

...Oh, looks like Weibin since verified *thumbs up* At any rate, just an fyi then.

Comment 10 errata-xmlrpc 2021-07-27 22:48:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438