Bug 1966499
| Summary: | portworx-operator causes APIRemovedInNextReleaseInUse alert | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Stefan Schimanski <sttts> |
| Component: | Management Console | Assignee: | David Taylor <dtaylor> |
| Status: | CLOSED ERRATA | QA Contact: | Wei Duan <wduan> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.8 | CC: | anpicker, aos-bugs, erooth, hongyli, jhadvig, jokerman, jsafrane, juzhao, kewang, lcosic, mbukatov, mfojtik, scuppett, sippy, spadgett, stbenjam, surbania, xxia, yapei |
| Target Milestone: | --- | Keywords: | TestDisabled |
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1947719 | Environment: |
Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) creates a Couchbase Operator CouchbaseCluster instance via the form
Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) displays Couchbase Operator CouchbaseCluster creation form
Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) displays details about Couchbase Operator CouchbaseCluster instance on the "Details" tab
Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) displays empty message on the Couchbase Operator ClusterServiceVersion "All Instances" tab
|
| Last Closed: | 2021-10-18 17:32:28 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1984102 | ||
Storage team is not responsible for portworx-operator, it's a community operator installed during Console tests. Please use a different operator for console tests, preferably one that is maintained by Red Hat and we then can fix bugs there. Changed priority to medium as we have temporarily disabled the test which uses portworx-operator (https://github.com/openshift/console/pull/9100). Once upstream PR merges: https://github.com/libopenstorage/operator/pull/323, we can re-enable our e2e test which uses the portworx-operator. There is also discusson on switching all e2e operator tests to use a RH or 'hello world' operator for testing instead of using community contributed operators in our tests. Removed it as a blocker from 1947719. Raising the severity since we have disabled tests that we need to update and reenable. Moving this back to high priority since we've previously disabled OLM tests that we need to reenable. This fix should be backported to 4.8.z so the tests are reenabled in the z-stream. *** Bug 1979970 has been marked as a duplicate of this bug. *** 1. Install 'Service Binding Operator' and 'Red Hat CodeReady Workspaces' operator, check the CRDs apiVersion, they are using new apiVersion apiextensions.k8s.io/v1 $ oc get crd servicebindings.binding.operators.coreos.com -o json | jq .apiVersion "apiextensions.k8s.io/v1" $ oc get crd checlusters.org.eclipse.che -o json | jq .apiVersion "apiextensions.k8s.io/v1" 2. Check CI logs in https://search.ci.openshift.org/?search=Installing+%22Red+Hat+CodeReady+Workspaces%22+operator&maxAge=48h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job new operators are used in our CI test Moving to VERIFIED Also when two new operators are installed, APIRemovedInNextReleaseInUse is not triggered (In reply to Yadan Pei from comment #10) > Also when two new operators are installed, APIRemovedInNextReleaseInUse is not triggered Cool, this is enough. BTW a reminder, comment 9 checks `oc get crd ... -o json | jq .apiVersion`, this is not the verification, it will never display v1beta1 because `oc get` displays the default display version (v1). Rather, need check `oc get apirequestcounts | grep customresourcedefinitions`, if has v1beta1 output, then check `oc get apirequestcounts V1BETA1_OUTPUT_ITEM -o yaml` to see username & requestCount whether they match the installed operators. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |
From `oc get apirequestcounts` we see that portworx-operator accesses v1beta1 CRDs. These will be gone in 4.9 and trigger an alert in 4.8 because upgrades to 4.9 will be bumpy: - requestCount: 2 verb: create - requestCount: 1 verb: delete - requestCount: 2 verb: get requestCount: 5 userAgent: operator/v0.0.0 username: system:serviceaccount:openshift-operators:portworx-operator Found in https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_console-operator/548/pull-ci-openshift-console-operator-master-e2e-aws-console/1398186657894633472 +++ This bug was initially created as a clone of Bug #1947719 +++ Created attachment 1770482 [details] alert screen shot Created attachment 1770482 [details] alert screen shot Description of problem: 8 DeprecatedAPIInUse info alerts display Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-04-08-200632 How reproducible: always Steps to Reproduce: 1. open console-monitoring-alerts 2. 3. Actual results: 8 DeprecatedAPIInUse info alerts display Expected results: No other alerts display except watchdog Additional info: alert rule metrics: group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0 Element Value: {group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"} 1 {group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"} 1 {group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"} 1 {group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"} 1 {group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"} 1 {group="extensions",resource="ingresses",version="v1beta1"} 1 {group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"} 1 {group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"} 1 ---------------- # for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done clusterroles authorization.openshift.io/v1 false ClusterRole roles authorization.openshift.io/v1 true Role clusterroles rbac.authorization.k8s.io/v1 false ClusterRole roles rbac.authorization.k8s.io/v1 true Role mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest ingresses config.openshift.io/v1 false Ingress ingresses ing extensions/v1beta1 true Ingress ingresses ing networking.k8s.io/v1 true Ingress clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding rolebindings authorization.openshift.io/v1 true RoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding rolebindings rbac.authorization.k8s.io/v1 true RoleBinding --- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST --- alert details alert:DeprecatedAPIInUse expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0 for: 1h labels: severity: info annotations: message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload. --- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST --- --- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST --- Different issue from bug 1932165 which is about variable not translated to value --- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST --- # oc version Client Version: 4.8.0-0.nightly-2021-04-08-200632 Server Version: 4.8.0-0.nightly-2021-04-08-200632 Kubernetes Version: v1.21.0-rc.0+6d27558 checked from prometheus, query parameter: count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource) version is v1beta1 {group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1 {group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1 {group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1 {group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1 {group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1 but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result # for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest ingresses config.openshift.io/v1 false Ingress ingresses ing extensions/v1beta1 true Ingress ingresses ing networking.k8s.io/v1 true Ingress clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding rolebindings authorization.openshift.io/v1 true RoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding rolebindings rbac.authorization.k8s.io/v1 true RoleBinding clusterroles authorization.openshift.io/v1 false ClusterRole roles authorization.openshift.io/v1 true Role clusterroles rbac.authorization.k8s.io/v1 false ClusterRole roles rbac.authorization.k8s.io/v1 true Role mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition --- Additional comment from Stefan Schimanski on 2021-05-18 16:06:30 CEST --- --- Additional comment from Martin Bukatovic on 2021-05-19 17:18:07 CEST --- The alert this bug talks about is APIRemovedInNextEUSReleaseInUse. I'm mentioning it here so that it's possible to find this bug when one searches by content of bugzilla comments.