Bug 2080123
| Summary: | Avoid update races between old and new NTO operands during cluster upgrades | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jiří Mencák <jmencak> | |
| Component: | Node Tuning Operator | Assignee: | Jiří Mencák <jmencak> | |
| Status: | CLOSED ERRATA | QA Contact: | liqcui | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.10 | CC: | dagray, liqcui | |
| Target Milestone: | --- | |||
| Target Release: | 4.10.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 2078969 | |||
| : | 2083482 (view as bug list) | Environment: | ||
| Last Closed: | 2022-05-18 11:51:02 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2078969 | |||
| Bug Blocks: | 2083482 | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10.14 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:2178 |
[ocpadmin@ec2-18-217-45-133 ~]$ oc create -f- <<'EOF' > apiVersion: tuned.openshift.io/v1 > kind: Tuned > metadata: > name: openshift-tuned-fight > namespace: openshift-cluster-node-tuning-operator > spec: > profile: > - data: | > [main] > summary=Custom OpenShift profile > [bootloader] > cmdline=+trigger_tuned_fight=${f:exec:/usr/bin/bash:-c:echo $RELEASE_VERSION} > name: openshift-tuned-fight > > recommend: > - machineConfigLabels: > machineconfiguration.openshift.io/role: "worker-rt" > priority: 20 > profile: openshift-tuned-fight > EOF tuned.tuned.openshift.io/openshift-tuned-fight created [ocpadmin@ec2-18-217-45-133 ~]$ for n in $(oc get no --selector=node-role.kubernetes.io/worker -o name) ; do oc label $n node-role.kubernetes.io/worker-rt= ; done node/ip-10-0-150-120.us-east-2.compute.internal labeled node/ip-10-0-165-14.us-east-2.compute.internal labeled node/ip-10-0-193-169.us-east-2.compute.internal labeled [ocpadmin@ec2-18-217-45-133 ~]$ oc get pods NAME READY STATUS RESTARTS AGE cluster-node-tuning-operator-74664f69fc-lhxmp 1/1 Running 0 35s tuned-85p9d 1/1 Running 0 27s tuned-c4cpq 1/1 Running 0 26s tuned-gxrp4 1/1 Running 0 31s tuned-zzrgl 1/1 Running 0 29s [ocpadmin@ec2-18-217-45-133 ~]$ oc logs -f cluster-node-tuning-operator-74664f69fc-lhxmp I0510 08:52:05.177282 1 main.go:25] Go Version: go1.17.5 I0510 08:52:05.177384 1 main.go:26] Go OS/Arch: linux/amd64 I0510 08:52:05.177395 1 main.go:27] node-tuning Version: v4.10.0-202205091737.p0.g73f9609.assembly.stream-0-g5faab36-dirty I0510 08:52:05.177642 1 server.go:51] starting metrics server I0510 08:52:05.178399 1 controller.go:1055] trying to become a leader I0510 08:52:06.230172 1 leaderelection.go:248] attempting to acquire leader lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock... I0510 08:52:06.238388 1 leaderelection.go:258] successfully acquired lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock I0510 08:52:06.238568 1 controller.go:1117] became leader: cluster-node-tuning-operator-74664f69fc-lhxmp_ece6211a-465f-48bf-bfac-99c879ca264f I0510 08:52:06.238585 1 controller.go:990] starting Tuned controller I0510 08:52:06.339417 1 controller.go:1042] started events processor/controller I0510 08:52:06.407943 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-150-120.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" I0510 08:52:06.410877 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-165-14.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" I0510 08:52:06.418223 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-193-169.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" I0510 08:52:06.443481 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-165-14.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" I0510 08:52:06.445203 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-193-169.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" I0510 08:52:06.452096 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-150-120.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" I0510 08:52:08.664185 1 controller.go:728] updated MachineConfig 50-nto-worker-rt with kernel parameters: [trigger_tuned_fight=my-4.10.0-0.nightly-2022-05-10-02092] I0510 08:52:13.772954 1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-165-14.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929" ^C [ocpadmin@ec2-18-217-45-133 ~]$ oc get pods NAME READY STATUS RESTARTS AGE cluster-node-tuning-operator-74664f69fc-lhxmp 1/1 Running 0 5m52s tuned-85p9d 1/1 Running 0 5m44s tuned-c4cpq 1/1 Running 1 5m43s tuned-gxrp4 1/1 Running 1 5m48s tuned-zzrgl 1/1 Running 1 5m46s