Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2079314

Summary: [4.10] APIRemovedInNextEUSReleaseInUse alert for poddisruptionbudgets/cronjobs
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Arunprasad Rajkumar <arajkuma>
Status: CLOSED DUPLICATE QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: low    
Version: 4.10CC: amuller, anpicker, aos-bugs, arajkuma, erooth, knakayam, spasquie
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 15:05:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2022-04-27 11:31:37 UTC 
Description of problem:
4.10.0-0.nightly-2022-04-26-204343 cluster, APIRemovedInNextEUSReleaseInUse alerts for poddisruptionbudgets/cronjobs are caused by kube-state-metrics
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "batch",
          "namespace": "openshift-kube-apiserver",
          "resource": "cronjobs",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651057963.326,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "policy",
          "namespace": "openshift-kube-apiserver",
          "resource": "poddisruptionbudgets",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651057963.326,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "policy",
          "namespace": "openshift-kube-apiserver",
          "resource": "podsecuritypolicies",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651057963.326,
          "1"
        ]
      }
    ]
  }
}
# oc -n openshift-monitoring get pod | grep kube-state-metrics
kube-state-metrics-7d65ff6c55-hdj79            3/3     Running   5 (6h13m ago)   6h20m


# oc get apirequestcounts poddisruptionbudgets.v1beta1.policy -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-04-27T05:12:05Z"
  generation: 1
  name: poddisruptionbudgets.v1beta1.policy
  resourceVersion: "154609"
  uid: 97ece6ec-1d0a-4faa-a7e2-35864b49b5f7
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  last24h:
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 6
          verb: watch
        requestCount: 6
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 6
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.76.223
      requestCount: 2
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 9
          verb: watch
        requestCount: 9
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 9
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 9
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 7
          verb: watch
        requestCount: 7
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 7
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 7
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 9
          verb: watch
        requestCount: 9
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 9
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 9
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 7
          verb: watch
        requestCount: 7
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 7
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 7
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 7
          verb: watch
        requestCount: 7
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 7
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 7
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 47


# oc explain poddisruptionbudgets
KIND:     PodDisruptionBudget
VERSION:  policy/v1

APIRemovedInNextEUSReleaseInUse alert detail
**********************
        - alert: APIRemovedInNextEUSReleaseInUse
          annotations:
            description: Deprecated API that will be removed in the next EUS version is
              being used. Removing the workload that is using the {{ $labels.group }}.{{
              $labels.version }}/{{ $labels.resource }} API might be necessary for a successful
              upgrade to the next EUS cluster version. Refer to `oc get apirequestcounts
              {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml`
              to identify the workload.
            summary: Deprecated API that will be removed in the next EUS version is being
              used.
          expr: |
            group(apiserver_requested_deprecated_apis{removed_release=~"1\\.2[45]"}) by (group,version,resource) and (sum by(group,version,resource) (rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h]))) > 0
          for: 1h
          labels:
            namespace: openshift-kube-apiserver
            severity: info
**********************
Version-Release number of selected component (if applicable):
# oc -n openshift-monitoring exec -c kube-state-metrics kube-state-metrics-7d65ff6c55-hdj79 -- kube-state-metrics --version
kube-state-metrics, version v2.3.0 (branch: rhaos-4.10-rhel-8, revision: 6bd963d)
  build user:       root@4c33318b8ee0
  build date:       2022-04-21T19:20:51Z
  go version:       go1.17.5
  platform:         linux/amd64
# oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-04-26-204343   True        False         6h4m    Cluster version is 4.10.0-0.nightly-2022-04-26-204343
kubernetes version: v1.23.5+70fb84c

How reproducible:
always

Steps to Reproduce:
1. check alerts
2.
3.

Actual results:
APIRemovedInNextEUSReleaseInUse alert for poddisruptionbudgets/cronjobs

Expected results:
no such alert

Additional info:
Comment 1 Junqi Zhao 2022-04-27 11:32:45 UTC 
# oc get apirequestcounts cronjobs.v1beta1.batch -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-04-27T05:11:44Z"
  generation: 1
  name: cronjobs.v1beta1.batch
  resourceVersion: "155899"
  uid: d06f176a-9148-48b0-91d4-b62d4425f46f
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 1
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 1
  last24h:
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 6
          verb: watch
        requestCount: 6
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 6
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.76.223
      requestCount: 2
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 8
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 7
          verb: watch
        requestCount: 7
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 7
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 7
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 8
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 8
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 8
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: v2.3.0
        username: system:serviceaccount:openshift-monitoring:kube-state-metrics
      nodeName: 10.0.52.79
      requestCount: 1
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 1
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 48

# oc explain cronjobs
KIND:     CronJob
VERSION:  batch/v1
Comment 2 Arunprasad Rajkumar 2022-04-28 12:40:29 UTC 
I think this alert matters for user workloads not for the core system components like KSM., because we can handle this situation much before the next EUS version goes to GA. Please feel free to reopen if you feel otherwise.
Comment 3 Kenjiro Nakayama 2022-05-16 06:05:35 UTC 
Since bz#2079318 fixed the same issue for podsecuritypolicies, I expected this the alert is fixed but any reason why it will not be fixed?
We have seen this issue on our serverless-operator CI and it is a blocker now, so we have to ignore the alert message somehow if the alert keeps producing.
Comment 5 Simon Pasquier 2022-05-17 15:05:44 UTC 

*** This bug has been marked as a duplicate of bug 2085326 ***
Comment 6 Simon Pasquier 2022-05-17 15:07:51 UTC 
https://github.com/openshift/kube-state-metrics/pull/72 should already fix this bug. Can you point me to a counter-example if this isn't the case for you?
Comment 7 Kenjiro Nakayama 2022-05-17 22:47:19 UTC 
Ah, I see. I wrongly assumed that the fix would remove the alert itself but actually it updates pdbs deployments.