Bug 1966499 - portworx-operator causes APIRemovedInNextReleaseInUse alert
Summary: portworx-operator causes APIRemovedInNextReleaseInUse alert
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Management Console
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.9.0
Assignee: David Taylor
QA Contact: Wei Duan
URL:
Whiteboard:
: 1979970 (view as bug list)
Depends On:
Blocks: 1984102
TreeView+ depends on / blocked
 
Reported: 2021-06-01 09:49 UTC by Stefan Schimanski
Modified: 2021-10-18 17:32 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1947719
Environment:
Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) creates a Couchbase Operator CouchbaseCluster instance via the form Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) displays Couchbase Operator CouchbaseCluster creation form Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) displays details about Couchbase Operator CouchbaseCluster instance on the "Details" tab Interacting with a single namespace install mode Operator (Couchbase Operator).Interacting with a single namespace install mode Operator (Couchbase Operator) displays empty message on the Couchbase Operator ClusterServiceVersion "All Instances" tab
Last Closed: 2021-10-18 17:32:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift console pull 9420 0 None open Bug 1966499: Switch Cypress OLM tests from using community to Red Hat operators 2021-07-06 17:42:15 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:32:31 UTC

Internal Links: 1984102

Description Stefan Schimanski 2021-06-01 09:49:53 UTC
From `oc get apirequestcounts` we see that portworx-operator accesses v1beta1 CRDs. These will be gone in 4.9 and trigger an alert in 4.8 because upgrades to 4.9 will be bumpy:

        - requestCount: 2
          verb: create
        - requestCount: 1
          verb: delete
        - requestCount: 2
          verb: get
        requestCount: 5
        userAgent: operator/v0.0.0
        username: system:serviceaccount:openshift-operators:portworx-operator

Found in

  https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_console-operator/548/pull-ci-openshift-console-operator-master-e2e-aws-console/1398186657894633472

+++ This bug was initially created as a clone of Bug #1947719 +++

Created attachment 1770482 [details]
alert screen shot

Created attachment 1770482 [details]
alert screen shot

Description of problem:
8 DeprecatedAPIInUse info alerts display

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-08-200632

How reproducible:
always

Steps to Reproduce:
1. open console-monitoring-alerts
2.
3.

Actual results:
8 DeprecatedAPIInUse info alerts display

Expected results:
No other alerts display except watchdog

Additional info:

alert rule metrics:
group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0

Element	Value:
{group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"}	1
{group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"}	1
{group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"}	1
{group="extensions",resource="ingresses",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"}	1

----------------
# for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done
clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role
mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration
validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration
customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest
ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding

--- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST ---

alert details
alert:DeprecatedAPIInUse
expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0
for: 1h
labels:
  severity: info
annotations:
  message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload.

--- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST ---



--- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST ---

Different issue from bug 1932165 which is about variable not translated to value

--- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST ---

# oc version
Client Version: 4.8.0-0.nightly-2021-04-08-200632
Server Version: 4.8.0-0.nightly-2021-04-08-200632
Kubernetes Version: v1.21.0-rc.0+6d27558

checked from prometheus, query parameter:
count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource)
version is v1beta1
{group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1
{group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1
{group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1

but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result
# for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest


ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding


clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role


mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration


validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration


customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition

--- Additional comment from Stefan Schimanski on 2021-05-18 16:06:30 CEST ---



--- Additional comment from Martin Bukatovic on 2021-05-19 17:18:07 CEST ---

The alert this bug talks about is APIRemovedInNextEUSReleaseInUse. I'm mentioning it here so that it's possible to find this bug when one searches by content of bugzilla comments.

Comment 1 Jan Safranek 2021-06-01 12:23:06 UTC
Storage team is not responsible for portworx-operator, it's a community operator installed during Console tests. Please use a different operator for console tests, preferably one that is maintained by Red Hat and we then can fix bugs there.

Comment 2 David Taylor 2021-06-02 16:19:38 UTC
Changed priority to medium as we have temporarily disabled the test which uses portworx-operator (https://github.com/openshift/console/pull/9100).

Once upstream PR merges: https://github.com/libopenstorage/operator/pull/323, we can re-enable our e2e test which uses the portworx-operator.

There is also discusson on switching all e2e operator tests to use a RH or 'hello world' operator for testing instead of using community contributed operators in our tests.

Comment 3 Stefan Schimanski 2021-06-07 14:08:29 UTC
Removed it as a blocker from 1947719.

Comment 4 Samuel Padgett 2021-06-11 16:31:54 UTC
Raising the severity since we have disabled tests that we need to update and reenable.

Comment 6 Samuel Padgett 2021-06-29 18:30:30 UTC
Moving this back to high priority since we've previously disabled OLM tests that we need to reenable. This fix should be backported to 4.8.z so the tests are reenabled in the z-stream.

Comment 7 Robb Hamilton 2021-07-12 15:39:27 UTC
*** Bug 1979970 has been marked as a duplicate of this bug. ***

Comment 9 Yadan Pei 2021-07-26 05:51:28 UTC
1. Install 'Service Binding Operator' and 'Red Hat CodeReady Workspaces' operator, check the CRDs apiVersion, they are using new apiVersion apiextensions.k8s.io/v1

$ oc get crd servicebindings.binding.operators.coreos.com -o json | jq .apiVersion
"apiextensions.k8s.io/v1"

$ oc get crd checlusters.org.eclipse.che -o json | jq .apiVersion
"apiextensions.k8s.io/v1"

2. Check CI logs in https://search.ci.openshift.org/?search=Installing+%22Red+Hat+CodeReady+Workspaces%22+operator&maxAge=48h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job 
new operators are used in our CI test

Moving to VERIFIED

Comment 10 Yadan Pei 2021-07-26 05:56:42 UTC
Also when two new operators are installed, APIRemovedInNextReleaseInUse is not triggered

Comment 11 Xingxing Xia 2021-07-26 10:35:10 UTC
(In reply to Yadan Pei from comment #10)
> Also when two new operators are installed, APIRemovedInNextReleaseInUse is not triggered
Cool, this is enough. BTW a reminder, comment 9 checks `oc get crd ... -o json | jq .apiVersion`, this is not the verification, it will never display v1beta1 because `oc get` displays the default display version (v1). Rather, need check `oc get apirequestcounts | grep customresourcedefinitions`, if has v1beta1 output, then check `oc get apirequestcounts V1BETA1_OUTPUT_ITEM -o yaml` to see username & requestCount whether they match the installed operators.

Comment 18 errata-xmlrpc 2021-10-18 17:32:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.