Bug 1947785 - Cloud Compute: check (see bug 1947801#c4 steps) audit log to find deprecated API access related to this component to ensure this component won't access APIs that trigger APIRemovedInNextReleaseInUse alert
Summary: Cloud Compute: check (see bug 1947801#c4 steps) audit log to find deprecated ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Alexander Demicev
QA Contact: Milind Yadav
URL:
Whiteboard:
Depends On:
Blocks: 1947719
TreeView+ depends on / blocked
 
Reported: 2021-04-09 09:13 UTC by Stefan Schimanski
Modified: 2021-07-27 22:58 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1947719
Environment:
Last Closed: 2021-07-27 22:58:26 UTC
Target Upstream Version:
Embargoed:
xxia: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-autoscaler-operator pull 199 0 None closed Bug 1947785: Move webhook to v1 2021-05-18 08:41:36 UTC
Github openshift cluster-autoscaler-operator pull 202 0 None open Bug 1947785: Move all webhooks to v1 2021-05-18 08:41:36 UTC
Github openshift cluster-machine-approver pull 111 0 None closed Bug 1947785: Move certificates to v1 2021-05-18 08:41:37 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:58:46 UTC

Description Stefan Schimanski 2021-04-09 09:13:32 UTC
This component accesses APIs that will be removed in 4.9 (Kubernetes 1.22). It is causing the DeprecatedAPIInUse alert to fire in every 4.8 clusters permanently and hence must be fixed in 4.8 (blocker+).

The raw audit data can be found at https://gist.github.com/sttts/50a1429837f2448ce07f30174fa73cdb.

Here are the observed requests for this component:

system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=35147&timeoutSeconds=459&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=35147&timeoutSeconds=459&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=37397&timeoutSeconds=564&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=37397&timeoutSeconds=564&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=40137&timeoutSeconds=366&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=40137&timeoutSeconds=366&watch=true

system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=35147&timeoutSeconds=459&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=37397&timeoutSeconds=564&watch=true
system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=40137&timeoutSeconds=366&watch=true

system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=36524&timeoutSeconds=329&watch=true
system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=38131&timeoutSeconds=341&watch=true
system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=39758&timeoutSeconds=496&watch=true

system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/rolebindings/cluster-autoscaler-operator

system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/roles/cluster-autoscaler-operator


+++ This bug was initially created as a clone of Bug #1947719 +++

Created attachment 1770482 [details]
alert screen shot

Created attachment 1770482 [details]
alert screen shot

Description of problem:
8 DeprecatedAPIInUse info alerts display

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-08-200632

How reproducible:
always

Steps to Reproduce:
1. open console-monitoring-alerts
2.
3.

Actual results:
8 DeprecatedAPIInUse info alerts display

Expected results:
No other alerts display except watchdog

Additional info:

alert rule metrics:
group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0

Element	Value:
{group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"}	1
{group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"}	1
{group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"}	1
{group="extensions",resource="ingresses",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"}	1

----------------
# for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done
clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role
mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration
validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration
customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest
ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding

--- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST ---

alert details
alert:DeprecatedAPIInUse
expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0
for: 1h
labels:
  severity: info
annotations:
  message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload.

--- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST ---



--- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST ---

Different issue from bug 1932165 which is about variable not translated to value

--- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST ---

# oc version
Client Version: 4.8.0-0.nightly-2021-04-08-200632
Server Version: 4.8.0-0.nightly-2021-04-08-200632
Kubernetes Version: v1.21.0-rc.0+6d27558

checked from prometheus, query parameter:
count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource)
version is v1beta1
{group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1
{group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1
{group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1

but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result
# for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest


ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding


clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role


mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration


validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration


customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition

Comment 1 Stefan Schimanski 2021-04-09 09:26:33 UTC
Also:

system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io

Comment 4 Milind Yadav 2021-04-16 05:21:15 UTC
Cluster version used in validation for attached snap - 4.8.0-0.nightly-2021-04-15-202330

Comment 6 Ke Wang 2021-04-23 08:27:05 UTC
@miyadav, The requests of the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1947785#c0 are gone for the given component, you won't see the related alert in web-console.

Comment 7 Milind Yadav 2021-04-23 09:17:19 UTC
Checked on latest build - Cluster version is 4.8.0-0.nightly-2021-04-22-225832

Same as snap attached in comment#3 

Moved to ASSIGNED

Comment 8 Ke Wang 2021-04-27 11:04:47 UTC
Verification steps, you can refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4

Comment 9 Ke Wang 2021-04-30 03:17:46 UTC
Did a try, refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4, still some requests can be found for the given component.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-29-151418   True        False         58m     Cluster version is 4.8.0-0.nightly-2021-04-29-151418

$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep -E 'certificates|validating|cluster-autoscaler-operator'
system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/rolebindings/cluster-autoscaler-operator
system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/roles/cluster-autoscaler-operator
system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=39696&timeoutSeconds=503&watch=true

Comment 11 Milind Yadav 2021-05-19 14:39:19 UTC
Thanks Ke Wang for checking .

Version used - Cluster version is 4.8.0-0.ci-2021-05-19-015601

Not seeing any cluster-autoscaler-operator apis , but still seeing cloudcredential one 

[miyadav@miyadav ~]$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep cloud
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io


.
.
.
[miyadav@miyadav ~]$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep -E 'certificates|validating|cluster-autoscaler-operator'
[miyadav@miyadav ~]$ 


Please take a look .. not sure if that is already tracked ..

Followed the steps as mentioned by Ke Wang comment .

Comment 12 Joel Speed 2021-05-20 12:50:18 UTC
Cloud credential is under a different component, Milind if you have the full list there without grep I can take a look to make sure we've covered everything this team 

Looking at the original list, anything with `cluster-machine-approver` or `cluster-autoscaler-operator` would be a failure, if those are all gone we should be good.

Comment 13 Milind Yadav 2021-05-20 13:07:23 UTC
Thanks @Joel . 

There cluster-machine-approver and cluster-autoscaler-operator arent there .


Here is the file - https://url.corp.redhat.com/dep


If it looks good , will move this to VERIFIED .

Comment 14 Joel Speed 2021-05-20 13:30:18 UTC
Yep, looks like we've got our part covered, please move to verified

Comment 15 Ke Wang 2021-05-27 11:19:28 UTC
Hi miyadav, the results looks good.

Comment 18 errata-xmlrpc 2021-07-27 22:58:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.