Bug 2080123 - Avoid update races between old and new NTO operands during cluster upgrades
Summary: Avoid update races between old and new NTO operands during cluster upgrades
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node Tuning Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.z
Assignee: Jiří Mencák
QA Contact: liqcui
URL:
Whiteboard:
Depends On: 2078969
Blocks: 2083482
TreeView+ depends on / blocked
 
Reported: 2022-04-29 05:12 UTC by Jiří Mencák
Modified: 2022-05-18 11:51 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2078969
: 2083482 (view as bug list)
Environment:
Last Closed: 2022-05-18 11:51:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-node-tuning-operator pull 357 0 None open Bug 2080123: Ignore Profile updates triggered by old operands 2022-05-06 12:56:39 UTC
Red Hat Product Errata RHBA-2022:2178 0 None None None 2022-05-18 11:51:13 UTC

Comment 3 liqcui 2022-05-10 09:21:10 UTC
[ocpadmin@ec2-18-217-45-133 ~]$ oc create -f- <<'EOF'
> apiVersion: tuned.openshift.io/v1
> kind: Tuned
> metadata:
>   name: openshift-tuned-fight
>   namespace: openshift-cluster-node-tuning-operator
> spec:
>   profile:
>   - data: |
>       [main]
>       summary=Custom OpenShift profile
>       [bootloader]
>       cmdline=+trigger_tuned_fight=${f:exec:/usr/bin/bash:-c:echo $RELEASE_VERSION}
>     name: openshift-tuned-fight
> 
>   recommend:
>   - machineConfigLabels:
>       machineconfiguration.openshift.io/role: "worker-rt"
>     priority: 20
>     profile: openshift-tuned-fight
> EOF
tuned.tuned.openshift.io/openshift-tuned-fight created
[ocpadmin@ec2-18-217-45-133 ~]$ for n in $(oc get no --selector=node-role.kubernetes.io/worker -o name) ; do oc label $n node-role.kubernetes.io/worker-rt= ; done
node/ip-10-0-150-120.us-east-2.compute.internal labeled
node/ip-10-0-165-14.us-east-2.compute.internal labeled
node/ip-10-0-193-169.us-east-2.compute.internal labeled
[ocpadmin@ec2-18-217-45-133 ~]$ oc get pods
NAME                                            READY   STATUS    RESTARTS   AGE
cluster-node-tuning-operator-74664f69fc-lhxmp   1/1     Running   0          35s
tuned-85p9d                                     1/1     Running   0          27s
tuned-c4cpq                                     1/1     Running   0          26s
tuned-gxrp4                                     1/1     Running   0          31s
tuned-zzrgl                                     1/1     Running   0          29s
[ocpadmin@ec2-18-217-45-133 ~]$ oc logs -f cluster-node-tuning-operator-74664f69fc-lhxmp
I0510 08:52:05.177282       1 main.go:25] Go Version: go1.17.5
I0510 08:52:05.177384       1 main.go:26] Go OS/Arch: linux/amd64
I0510 08:52:05.177395       1 main.go:27] node-tuning Version: v4.10.0-202205091737.p0.g73f9609.assembly.stream-0-g5faab36-dirty
I0510 08:52:05.177642       1 server.go:51] starting metrics server
I0510 08:52:05.178399       1 controller.go:1055] trying to become a leader
I0510 08:52:06.230172       1 leaderelection.go:248] attempting to acquire leader lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock...
I0510 08:52:06.238388       1 leaderelection.go:258] successfully acquired lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock
I0510 08:52:06.238568       1 controller.go:1117] became leader: cluster-node-tuning-operator-74664f69fc-lhxmp_ece6211a-465f-48bf-bfac-99c879ca264f
I0510 08:52:06.238585       1 controller.go:990] starting Tuned controller
I0510 08:52:06.339417       1 controller.go:1042] started events processor/controller
I0510 08:52:06.407943       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-150-120.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
I0510 08:52:06.410877       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-165-14.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
I0510 08:52:06.418223       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-193-169.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
I0510 08:52:06.443481       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-165-14.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
I0510 08:52:06.445203       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-193-169.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
I0510 08:52:06.452096       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-150-120.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
I0510 08:52:08.664185       1 controller.go:728] updated MachineConfig 50-nto-worker-rt with kernel parameters: [trigger_tuned_fight=my-4.10.0-0.nightly-2022-05-10-02092]
I0510 08:52:13.772954       1 controller.go:651] refusing to sync MachineConfig "50-nto-worker-rt" due to Profile "ip-10-0-165-14.us-east-2.compute.internal" change generated by operand version "4.10.0-0.nightly-2022-05-10-020929"
^C
[ocpadmin@ec2-18-217-45-133 ~]$ oc get pods
NAME                                            READY   STATUS    RESTARTS   AGE
cluster-node-tuning-operator-74664f69fc-lhxmp   1/1     Running   0          5m52s
tuned-85p9d                                     1/1     Running   0          5m44s
tuned-c4cpq                                     1/1     Running   1          5m43s
tuned-gxrp4                                     1/1     Running   1          5m48s
tuned-zzrgl                                     1/1     Running   1          5m46s

Comment 6 errata-xmlrpc 2022-05-18 11:51:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.10.14 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2178


Note You need to log in before you can comment on or make changes to this bug.