Bug 1960339 - manifests: unset "preemptionPolicy" makes CVO hotloop
Summary: manifests: unset "preemptionPolicy" makes CVO hotloop
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: config-operator
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.8.0
Assignee: Stefan Schimanski
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-13 17:12 UTC by Vadim Rutkovsky
Modified: 2021-07-27 23:08 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:08:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-config-operator pull 202 0 None closed Bug 1960339: manifests: add preemptionPolicy for openshift-user-critical 2021-06-07 10:01:06 UTC
Github openshift cluster-config-operator pull 205 0 None open Bug 1960339: openshift-user-critical: unset globalDefault 2021-06-07 10:01:06 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:08:37 UTC

Description Vadim Rutkovsky 2021-05-13 17:12:00 UTC
Manifest doesn't have preemptionPolicy set, so CVO tries to remove it on every sync cycle

Comment 2 Vadim Rutkovsky 2021-06-02 15:27:00 UTC
Moving to POST as another change in cluster-config-operator is required

Comment 5 Xingxing Xia 2021-06-29 12:05:13 UTC
Sorry did not timely check it. Was fully occupied by burning down other on_qa bugs and other new coming work.

Today checked in 4.8.0-0.nightly-2021-06-29-033219 fresh env:
oc get pc openshift-user-critical -o yaml
apiVersion: scheduling.k8s.io/v1
description: This priority class should be used for user facing OpenShift workload
  pods only.
kind: PriorityClass
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  creationTimestamp: "2021-06-29T09:11:37Z"
  generation: 1
  name: openshift-user-critical
  resourceVersion: "922"
  uid: 87054743-27b7-44d7-94e2-70904ffe8daa
preemptionPolicy: PreemptLowerPriority
value: 1000000000

It sets preemptionPolicy and has no globalDefault as the PRs do.
But checked CVO pod logs:
oc logs cluster-version-operator-694d98bc85-mwz5t -n openshift-cluster-version | grep -i priorityclass
...
I0629 11:19:09.549293       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:19:09.579146       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:22:25.764389       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:22:25.803163       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:25:42.095430       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:25:42.129337       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:28:58.312801       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:28:58.349362       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:32:14.523692       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:32:14.562926       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:35:30.746833       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:35:30.774766       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:38:47.053778       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:38:47.090787       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:42:03.322813       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:42:03.361631       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:45:19.533299       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:45:19.571770       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:48:35.753839       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:48:35.792080       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:51:51.960179       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:51:51.998018       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:55:08.181023       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:55:08.217326       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:58:24.391219       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 11:58:24.427691       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)
I0629 12:01:40.609428       1 sync_worker.go:752] Running sync for priorityclass "openshift-user-critical" (411 of 676)
I0629 12:01:40.662608       1 sync_worker.go:764] Done syncing for priorityclass "openshift-user-critical" (411 of 676)

Is this every-3-min repetition expected?

Comment 6 Vadim Rutkovsky 2021-06-29 15:57:22 UTC
(In reply to Xingxing Xia from comment #5)
> Is this every-3-min repetition expected?

The periodic sync is expected. These lines mean this manifest is being handled by CVO, but it doesn't mean its being hotlooping.

Check CVO logs, if it has "... updated due to diff..." periodically, it means we have a hotloop again.

Comment 7 Xingxing Xia 2021-06-30 02:29:26 UTC
Thx, no that periodical stuff is found, thus moving to VERIFIED

Comment 9 errata-xmlrpc 2021-07-27 23:08:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.