Bug 1907313 - ClusterOperator objects cannot be overriden with cvo-overrides
Summary: ClusterOperator objects cannot be overriden with cvo-overrides
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.7.0
Assignee: Guillaume Rose
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-14 08:18 UTC by Guillaume Rose
Modified: 2021-02-24 15:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:43:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 488 0 None closed Bug 1907313: Don't create ClusterOperator during precreation step if it's present in overrides 2021-01-29 09:45:20 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:43:41 UTC

Description Guillaume Rose 2020-12-14 08:18:16 UTC
Description of problem:

When using the installer, when I add a ClusterOperator in the overrides list, the object is still created by the CVO.
Precreation step doesn't check the overrides list before creating ClusterOperators.

I can see in the logs:

```
I1210 20:28:04.418715       1 sync_worker.go:687] Precreated resource clusteroperator "monitoring" (317 of 617)
I1210 20:28:32.633406       1 sync_worker.go:701] Running sync for clusteroperator "monitoring" (317 of 617)
I1210 20:28:32.633423       1 sync_worker.go:705] Skipping clusteroperator "monitoring" (317 of 617) as unmanaged
```

I would like to use this method to disable correctly monitoring in CodeReady Containers VM.

Comment 2 Johnny Liu 2020-12-16 07:39:17 UTC
Hi Guillaume,
Can you show me the detailed steps of adding a ClusterOperator in the overrides list? So that I can reproduce it and verify this bug?

Comment 3 Guillaume Rose 2020-12-16 08:20:05 UTC
Yes sure, so first, take the openshift-installer and a traditional install configuration.

* mkdir cluster-1
* cp install-config.yaml cluster-1/
* openshift-install --dir=cluster-1 create manifests  # warning: this target is unstable
* edit cluster-1/manifests/cvo-overrides.yaml and add overrides below spec:

```
apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
  name: version
spec:
  channel: ...
  clusterID: ...
  overrides:
  - kind: Deployment
    group: apps/v1
    name: cluster-monitoring-operator
    namespace: openshift-monitoring
    unmanaged: true
  - kind: ClusterOperator
    group: config.openshift.io/v1
    name: monitoring
    namespace: ""
    unmanaged: true
```

* openshift-install --dir=cluster-1 create cluster
* openshift-install --dir=cluster-1 wait-for install-complete
* observe that the monitoring stack doesn't exist. `oc get co` doesn't include monitoring. Monitoring pods, namespaces, etc. don't exist.

Comment 4 Johnny Liu 2020-12-16 11:48:48 UTC
Great, thanks for your detailed steps.

I can reproduce this issue on 4.7.0-0.nightly-2020-12-14-165231

[root@preserve-jialiu-ansible ~]# oc get -o json clusterversion version | jq .spec.overrides
[
  {
    "group": "apps/v1",
    "kind": "Deployment",
    "name": "cluster-monitoring-operator",
    "namespace": "openshift-monitoring",
    "unmanaged": true
  },
  {
    "group": "config.openshift.io/v1",
    "kind": "ClusterOperator",
    "name": "monitoring",
    "namespace": "",
    "unmanaged": true
  }
]

[root@preserve-jialiu-ansible ~]# oc get co monitoring
NAME         VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
monitoring 

[root@preserve-jialiu-ansible ~]# oc get ns|grep openshift-monitoring
openshift-monitoring                               Active   58m

[root@preserve-jialiu-ansible ~]# oc get all -n openshift-monitoring
No resources found in openshift-monitoring namespace.

[root@preserve-jialiu-ansible ~]# oc -n openshift-cluster-version logs cluster-version-operator-5c64ff97f-kkp6j |grep clusteroperator|grep monitoring
I1216 11:43:24.280891       1 sync_worker.go:699] Running sync for clusteroperator "monitoring" (330 of 663)
I1216 11:43:24.280898       1 sync_worker.go:703] Skipping clusteroperator "monitoring" (330 of 663) as unmanaged

Once I removed Deployment from overrides list, check log again, will see 'Precreated resource clusteroperator "monitoring"' log.
  - kind: Deployment
    group: apps/v1
    name: cluster-monitoring-operator
    namespace: openshift-monitoring
    unmanaged: true

[root@preserve-jialiu-ansible ~]# oc -n openshift-cluster-version logs cluster-version-operator-5c64ff97f-kkp6j |grep clusteroperator|grep monitoring
I1216 11:43:24.280891       1 sync_worker.go:699] Running sync for clusteroperator "monitoring" (330 of 663)
I1216 11:43:24.280898       1 sync_worker.go:703] Skipping clusteroperator "monitoring" (330 of 663) as unmanaged
I1216 11:45:35.815545       1 sync_worker.go:685] Precreated resource clusteroperator "monitoring" (330 of 663)
I1216 11:46:06.074255       1 sync_worker.go:699] Running sync for clusteroperator "monitoring" (330 of 663)
I1216 11:46:06.074266       1 sync_worker.go:703] Skipping clusteroperator "monitoring" (330 of 663) as unmanaged


Once next green nightly build available, I will run verification.

Comment 5 Johnny Liu 2020-12-18 04:01:32 UTC
Verified this bug with 4.7.0-0.nightly-2020-12-17-201522, and PASS.

[root@preserve-jialiu-ansible ~]# oc get co monitoring
Error from server (NotFound): clusteroperators.config.openshift.io "monitoring" not found

[root@preserve-jialiu-ansible ~]# oc get ns|grep openshift-monitoring
openshift-monitoring                               Active   58m

The monitoring is listed there, from my understanding, we did not add monitoring into override list, so this behaviour is expected.

[root@preserve-jialiu-ansible ~]# oc get all -n openshift-monitoring
No resources found in openshift-monitoring namespace.

[root@preserve-jialiu-ansible ~]# oc -n openshift-cluster-version logs  cluster-version-operator-57ccffd947-tknmr |grep clusteroperator|grep monitoring
I1218 03:32:44.628712       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:32:44.628718       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged

Once I removed Deployment from overrides list, check log again, will see 'Skipping precreation of clusteroperator "monitoring"' log.

[root@preserve-jialiu-ansible ~]# oc -n openshift-cluster-version logs  cluster-version-operator-57ccffd947-tknmr |grep clusteroperator|grep monitoring
I1218 03:32:44.628712       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:32:44.628718       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:37:17.098598       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:37:17.098604       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:40:50.462477       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:40:50.462484       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:45:06.876104       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:45:06.876111       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:48:42.440564       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:48:42.440571       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:52:13.105708       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:52:13.105714       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:53:47.134955       1 sync_worker.go:708] Skipping precreation of clusteroperator "monitoring" (329 of 660) as unmanaged
I1218 03:54:17.475356       1 sync_worker.go:729] Running sync for clusteroperator "monitoring" (329 of 660)
I1218 03:54:17.475366       1 sync_worker.go:733] Skipping clusteroperator "monitoring" (329 of 660) as unmanaged

[root@preserve-jialiu-ansible ~]# oc get co monitoring
NAME         VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
monitoring   4.7.0-0.nightly-2020-12-17-201522   True        False         False      3m24s


Now I am moving this bug to verified state, if anything wrong, pls let me know.

Comment 8 errata-xmlrpc 2021-02-24 15:43:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.