Description of problem: APIRemovedInNextReleaseInUse alerts for validatingwebhookconfigurations/customresourcedefinitions/ingresses/rolebindings/roles/priorityclasses, except priorityclasses, the issue with other resources are tracked with bug 1947719 and other bugs. this bug is only for priorityclasses. search with ALERTS{alertname="APIRemovedInNextReleaseInUse"} with prometheus API # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%3D%22APIRemovedInNextReleaseInUse%22%7D' | jq { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextReleaseInUse", "alertstate": "firing", "group": "admissionregistration.k8s.io", "resource": "validatingwebhookconfigurations", "severity": "info", "version": "v1beta1" }, "value": [ 1620977424.957, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextReleaseInUse", "alertstate": "firing", "group": "apiextensions.k8s.io", "resource": "customresourcedefinitions", "severity": "info", "version": "v1beta1" }, "value": [ 1620977424.957, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextReleaseInUse", "alertstate": "firing", "group": "extensions", "resource": "ingresses", "severity": "info", "version": "v1beta1" }, "value": [ 1620977424.957, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextReleaseInUse", "alertstate": "firing", "group": "rbac.authorization.k8s.io", "resource": "rolebindings", "severity": "info", "version": "v1beta1" }, "value": [ 1620977424.957, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextReleaseInUse", "alertstate": "firing", "group": "rbac.authorization.k8s.io", "resource": "roles", "severity": "info", "version": "v1beta1" }, "value": [ 1620977424.957, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextReleaseInUse", "alertstate": "firing", "group": "scheduling.k8s.io", "resource": "priorityclasses", "severity": "info", "version": "v1beta1" }, "value": [ 1620977424.957, "1" ] } ] } } checked the resources version, they are v1 # for i in validatingwebhookconfigurations customresourcedefinitions ingresses rolebindings roles priorityclasses; do echo $i; oc explain $i | head -n 2; echo -e "\n"; done validatingwebhookconfigurations KIND: ValidatingWebhookConfiguration VERSION: admissionregistration.k8s.io/v1 customresourcedefinitions KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io/v1 ingresses KIND: Ingress VERSION: networking.k8s.io/v1 rolebindings KIND: RoleBinding VERSION: rbac.authorization.k8s.io/v1 roles KIND: Role VERSION: rbac.authorization.k8s.io/v1 priorityclasses KIND: PriorityClass VERSION: scheduling.k8s.io/v1 Version-Release number of selected component (if applicable): # oc version Client Version: 4.8.0-0.nightly-2021-05-13-180601 Server Version: 4.8.0-0.nightly-2021-05-13-180601 Kubernetes Version: v1.21.0-rc.0+6998007 How reproducible: always Steps to Reproduce: 1. see the description 2. 3. Actual results: APIRemovedInNextReleaseInUse alert for priorityclasses Expected results: no alert Additional info: alert: APIRemovedInNextReleaseInUse expr: group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[4h]))) > 0 for: 1h labels: severity: info annotations: message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the apirequestcount.apiserver.openshift.io resources to identify the workload.
Please follow the APIRemovedInNextReleaseInUse alert message: "Refer to the apirequestcount.apiserver.openshift.io resources to identify the workload." The alert itself and this BZ against the apiserver is not helpful. Identify the workloads causing this and assign BZ to them.
# oc api-resources | grep priorityclasses priorityclasses pc scheduling.k8s.io/v1 false PriorityClass priorityclasses APIVERSION scheduling.k8s.io/v1, assign to kube-scheduler, please correct me if I'm wrong
Junqi Zhao, can you also provide output of "oc get apirequestcount"? Resp. "oc get apirequestcount priorityclasses.v1beta1.scheduling.k8s.io -o yaml" to see which component/client is requesting v1beta1 version? Can you also collect the audit logs (e.g. by running https://github.com/openshift/must-gather/blob/master/collection-scripts/gather_audit_logs)?
Created attachment 1784010 [details] oc get apirequestcount priorityclasses.v1beta1.scheduling.k8s.io -o yaml
Waiting for PR review
Closing Per https://github.com/openshift/cluster-logging-operator/pull/1023#issuecomment-845729529 Migrated to https://issues.redhat.com/browse/LOG-1385