Hide Forgot
This component accesses APIs that will be removed in 4.9 (Kubernetes 1.22). It is causing the DeprecatedAPIInUse alert to fire in every 4.8 clusters permanently and hence must be fixed in 4.8 (blocker+). The raw audit data can be found at https://gist.github.com/sttts/50a1429837f2448ce07f30174fa73cdb. Here are the observed requests for this component: system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=35147&timeoutSeconds=459&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=35147&timeoutSeconds=459&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=37397&timeoutSeconds=564&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=37397&timeoutSeconds=564&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=40137&timeoutSeconds=366&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=40137&timeoutSeconds=366&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=35147&timeoutSeconds=459&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=37397&timeoutSeconds=564&watch=true system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&resourceVersion=40137&timeoutSeconds=366&watch=true system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=36524&timeoutSeconds=329&watch=true system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=38131&timeoutSeconds=341&watch=true system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=39758&timeoutSeconds=496&watch=true system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/rolebindings/cluster-autoscaler-operator system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/roles/cluster-autoscaler-operator +++ This bug was initially created as a clone of Bug #1947719 +++ Created attachment 1770482 [details] alert screen shot Created attachment 1770482 [details] alert screen shot Description of problem: 8 DeprecatedAPIInUse info alerts display Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-04-08-200632 How reproducible: always Steps to Reproduce: 1. open console-monitoring-alerts 2. 3. Actual results: 8 DeprecatedAPIInUse info alerts display Expected results: No other alerts display except watchdog Additional info: alert rule metrics: group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0 Element Value: {group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"} 1 {group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"} 1 {group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"} 1 {group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"} 1 {group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"} 1 {group="extensions",resource="ingresses",version="v1beta1"} 1 {group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"} 1 {group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"} 1 ---------------- # for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done clusterroles authorization.openshift.io/v1 false ClusterRole roles authorization.openshift.io/v1 true Role clusterroles rbac.authorization.k8s.io/v1 false ClusterRole roles rbac.authorization.k8s.io/v1 true Role mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest ingresses config.openshift.io/v1 false Ingress ingresses ing extensions/v1beta1 true Ingress ingresses ing networking.k8s.io/v1 true Ingress clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding rolebindings authorization.openshift.io/v1 true RoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding rolebindings rbac.authorization.k8s.io/v1 true RoleBinding --- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST --- alert details alert:DeprecatedAPIInUse expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0 for: 1h labels: severity: info annotations: message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload. --- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST --- --- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST --- Different issue from bug 1932165 which is about variable not translated to value --- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST --- # oc version Client Version: 4.8.0-0.nightly-2021-04-08-200632 Server Version: 4.8.0-0.nightly-2021-04-08-200632 Kubernetes Version: v1.21.0-rc.0+6d27558 checked from prometheus, query parameter: count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource) version is v1beta1 {group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1 {group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1 {group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1 {group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1 {group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1 but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result # for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest ingresses config.openshift.io/v1 false Ingress ingresses ing extensions/v1beta1 true Ingress ingresses ing networking.k8s.io/v1 true Ingress clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding rolebindings authorization.openshift.io/v1 true RoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding rolebindings rbac.authorization.k8s.io/v1 true RoleBinding clusterroles authorization.openshift.io/v1 false ClusterRole roles authorization.openshift.io/v1 true Role clusterroles rbac.authorization.k8s.io/v1 false ClusterRole roles rbac.authorization.k8s.io/v1 true Role mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition
Also: system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io
Cluster version used in validation for attached snap - 4.8.0-0.nightly-2021-04-15-202330
@miyadav, The requests of the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1947785#c0 are gone for the given component, you won't see the related alert in web-console.
Checked on latest build - Cluster version is 4.8.0-0.nightly-2021-04-22-225832 Same as snap attached in comment#3 Moved to ASSIGNED
Verification steps, you can refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4
Did a try, refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4, still some requests can be found for the given component. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-29-151418 True False 58m Cluster version is 4.8.0-0.nightly-2021-04-29-151418 $ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep -E 'certificates|validating|cluster-autoscaler-operator' system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/rolebindings/cluster-autoscaler-operator system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-machine-api/roles/cluster-autoscaler-operator system:serviceaccount:openshift-machine-api:cluster-autoscaler-operator: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=39696&timeoutSeconds=503&watch=true
Thanks Ke Wang for checking . Version used - Cluster version is 4.8.0-0.ci-2021-05-19-015601 Not seeing any cluster-autoscaler-operator apis , but still seeing cloudcredential one [miyadav@miyadav ~]$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep cloud system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io . . . [miyadav@miyadav ~]$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep -E 'certificates|validating|cluster-autoscaler-operator' [miyadav@miyadav ~]$ Please take a look .. not sure if that is already tracked .. Followed the steps as mentioned by Ke Wang comment .
Cloud credential is under a different component, Milind if you have the full list there without grep I can take a look to make sure we've covered everything this team Looking at the original list, anything with `cluster-machine-approver` or `cluster-autoscaler-operator` would be a failure, if those are all gone we should be good.
Thanks @Joel . There cluster-machine-approver and cluster-autoscaler-operator arent there . Here is the file - https://url.corp.redhat.com/dep If it looks good , will move this to VERIFIED .
Yep, looks like we've got our part covered, please move to verified
Hi miyadav, the results looks good.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438