Bug 1960549 - APIRemovedInNextReleaseInUse alert for priorityclasses
Summary: APIRemovedInNextReleaseInUse alert for priorityclasses
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-scheduler
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.8.0
Assignee: Jan Chaloupka
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On:
Blocks: 1947719
TreeView+ depends on / blocked
 
Reported: 2021-05-14 07:57 UTC by Junqi Zhao
Modified: 2021-06-08 16:58 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-21 13:02:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
oc get apirequestcount priorityclasses.v1beta1.scheduling.k8s.io -o yaml (6.52 KB, text/plain)
2021-05-17 10:46 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 1023 0 None open bug 1960549: priorityclasses.v1beta1.scheduling.k8s.io is removed in 1.22 and replaced by priorityclasses.v1.scheduling.... 2021-05-18 08:44:02 UTC

Description Junqi Zhao 2021-05-14 07:57:44 UTC
Description of problem:
APIRemovedInNextReleaseInUse alerts for validatingwebhookconfigurations/customresourcedefinitions/ingresses/rolebindings/roles/priorityclasses, except priorityclasses, the issue with other resources are tracked with bug 1947719 and other bugs. this bug is only for priorityclasses.

search with ALERTS{alertname="APIRemovedInNextReleaseInUse"} with prometheus API
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%3D%22APIRemovedInNextReleaseInUse%22%7D' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextReleaseInUse",
          "alertstate": "firing",
          "group": "admissionregistration.k8s.io",
          "resource": "validatingwebhookconfigurations",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1620977424.957,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextReleaseInUse",
          "alertstate": "firing",
          "group": "apiextensions.k8s.io",
          "resource": "customresourcedefinitions",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1620977424.957,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextReleaseInUse",
          "alertstate": "firing",
          "group": "extensions",
          "resource": "ingresses",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1620977424.957,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextReleaseInUse",
          "alertstate": "firing",
          "group": "rbac.authorization.k8s.io",
          "resource": "rolebindings",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1620977424.957,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextReleaseInUse",
          "alertstate": "firing",
          "group": "rbac.authorization.k8s.io",
          "resource": "roles",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1620977424.957,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextReleaseInUse",
          "alertstate": "firing",
          "group": "scheduling.k8s.io",
          "resource": "priorityclasses",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1620977424.957,
          "1"
        ]
      }
    ]
  }
}

checked the resources version, they are v1
# for i in validatingwebhookconfigurations customresourcedefinitions ingresses rolebindings roles priorityclasses; do echo $i; oc explain $i | head -n 2; echo -e "\n"; done
validatingwebhookconfigurations
KIND:     ValidatingWebhookConfiguration
VERSION:  admissionregistration.k8s.io/v1


customresourcedefinitions
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1


ingresses
KIND:     Ingress
VERSION:  networking.k8s.io/v1


rolebindings
KIND:     RoleBinding
VERSION:  rbac.authorization.k8s.io/v1


roles
KIND:     Role
VERSION:  rbac.authorization.k8s.io/v1


priorityclasses
KIND:     PriorityClass
VERSION:  scheduling.k8s.io/v1


Version-Release number of selected component (if applicable):
# oc version
Client Version: 4.8.0-0.nightly-2021-05-13-180601
Server Version: 4.8.0-0.nightly-2021-05-13-180601
Kubernetes Version: v1.21.0-rc.0+6998007


How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
APIRemovedInNextReleaseInUse alert for priorityclasses

Expected results:
no alert

Additional info:
alert: APIRemovedInNextReleaseInUse
expr: group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[4h]))) > 0
for: 1h
labels:
  severity: info
annotations:
  message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the apirequestcount.apiserver.openshift.io resources to identify the workload.

Comment 1 Stefan Schimanski 2021-05-17 07:44:12 UTC
Please follow the APIRemovedInNextReleaseInUse alert message: "Refer to the apirequestcount.apiserver.openshift.io resources to identify the workload."

The alert itself and this BZ against the apiserver is not helpful. Identify the workloads causing this and assign BZ to them.

Comment 2 Junqi Zhao 2021-05-17 08:23:18 UTC
# oc api-resources | grep priorityclasses
priorityclasses                       pc               scheduling.k8s.io/v1                          false        PriorityClass

priorityclasses APIVERSION scheduling.k8s.io/v1, assign to kube-scheduler, please correct me if I'm wrong

Comment 3 Jan Chaloupka 2021-05-17 09:30:26 UTC
Junqi Zhao,

can you also provide output of "oc get apirequestcount"? Resp. "oc get apirequestcount priorityclasses.v1beta1.scheduling.k8s.io -o yaml" to see which component/client is requesting v1beta1 version?
Can you also collect the audit logs (e.g. by running https://github.com/openshift/must-gather/blob/master/collection-scripts/gather_audit_logs)?

Comment 4 Junqi Zhao 2021-05-17 10:46:06 UTC
Created attachment 1784010 [details]
oc get apirequestcount priorityclasses.v1beta1.scheduling.k8s.io -o yaml

Comment 7 Jan Chaloupka 2021-05-20 14:09:43 UTC
Waiting for PR review


Note You need to log in before you can comment on or make changes to this bug.