Description of problem: 4.10.0-0.nightly-2022-04-26-204343 cluster, APIRemovedInNextEUSReleaseInUse alerts for poddisruptionbudgets/cronjobs are caused by kube-state-metrics # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse"}' | jq { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextEUSReleaseInUse", "alertstate": "firing", "group": "batch", "namespace": "openshift-kube-apiserver", "resource": "cronjobs", "severity": "info", "version": "v1beta1" }, "value": [ 1651057963.326, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextEUSReleaseInUse", "alertstate": "firing", "group": "policy", "namespace": "openshift-kube-apiserver", "resource": "poddisruptionbudgets", "severity": "info", "version": "v1beta1" }, "value": [ 1651057963.326, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextEUSReleaseInUse", "alertstate": "firing", "group": "policy", "namespace": "openshift-kube-apiserver", "resource": "podsecuritypolicies", "severity": "info", "version": "v1beta1" }, "value": [ 1651057963.326, "1" ] } ] } } # oc -n openshift-monitoring get pod | grep kube-state-metrics kube-state-metrics-7d65ff6c55-hdj79 3/3 Running 5 (6h13m ago) 6h20m # oc get apirequestcounts poddisruptionbudgets.v1beta1.policy -o yaml apiVersion: apiserver.openshift.io/v1 kind: APIRequestCount metadata: creationTimestamp: "2022-04-27T05:12:05Z" generation: 1 name: poddisruptionbudgets.v1beta1.policy resourceVersion: "154609" uid: 97ece6ec-1d0a-4faa-a7e2-35864b49b5f7 spec: numberOfUsersToReport: 10 status: currentHour: byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 last24h: - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - byUser: - byVerb: - requestCount: 6 verb: watch requestCount: 6 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 6 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 1 verb: list - requestCount: 1 verb: watch requestCount: 2 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.76.223 requestCount: 2 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 9 verb: watch requestCount: 9 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 9 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 9 - byNode: - byUser: - byVerb: - requestCount: 7 verb: watch requestCount: 7 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 7 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 7 - byNode: - byUser: - byVerb: - requestCount: 9 verb: watch requestCount: 9 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 9 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 9 - byNode: - byUser: - byVerb: - requestCount: 7 verb: watch requestCount: 7 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 7 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 7 - byNode: - byUser: - byVerb: - requestCount: 7 verb: watch requestCount: 7 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 7 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 7 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 removedInRelease: "1.25" requestCount: 47 # oc explain poddisruptionbudgets KIND: PodDisruptionBudget VERSION: policy/v1 APIRemovedInNextEUSReleaseInUse alert detail ********************** - alert: APIRemovedInNextEUSReleaseInUse annotations: description: Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload. summary: Deprecated API that will be removed in the next EUS version is being used. expr: | group(apiserver_requested_deprecated_apis{removed_release=~"1\\.2[45]"}) by (group,version,resource) and (sum by(group,version,resource) (rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h]))) > 0 for: 1h labels: namespace: openshift-kube-apiserver severity: info ********************** Version-Release number of selected component (if applicable): # oc -n openshift-monitoring exec -c kube-state-metrics kube-state-metrics-7d65ff6c55-hdj79 -- kube-state-metrics --version kube-state-metrics, version v2.3.0 (branch: rhaos-4.10-rhel-8, revision: 6bd963d) build user: root@4c33318b8ee0 build date: 2022-04-21T19:20:51Z go version: go1.17.5 platform: linux/amd64 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-04-26-204343 True False 6h4m Cluster version is 4.10.0-0.nightly-2022-04-26-204343 kubernetes version: v1.23.5+70fb84c How reproducible: always Steps to Reproduce: 1. check alerts 2. 3. Actual results: APIRemovedInNextEUSReleaseInUse alert for poddisruptionbudgets/cronjobs Expected results: no such alert Additional info:
# oc get apirequestcounts cronjobs.v1beta1.batch -o yaml apiVersion: apiserver.openshift.io/v1 kind: APIRequestCount metadata: creationTimestamp: "2022-04-27T05:11:44Z" generation: 1 name: cronjobs.v1beta1.batch resourceVersion: "155899" uid: d06f176a-9148-48b0-91d4-b62d4425f46f spec: numberOfUsersToReport: 10 status: currentHour: byNode: - byUser: - byVerb: - requestCount: 1 verb: watch requestCount: 1 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 1 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 1 last24h: - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - byUser: - byVerb: - requestCount: 6 verb: watch requestCount: 6 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 6 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 1 verb: list - requestCount: 1 verb: watch requestCount: 2 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.76.223 requestCount: 2 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 8 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 7 verb: watch requestCount: 7 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 7 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 7 - byNode: - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 8 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 8 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 8 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 1 verb: watch requestCount: 1 userAgent: v2.3.0 username: system:serviceaccount:openshift-monitoring:kube-state-metrics nodeName: 10.0.52.79 requestCount: 1 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 1 - requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 removedInRelease: "1.25" requestCount: 48 # oc explain cronjobs KIND: CronJob VERSION: batch/v1
I think this alert matters for user workloads not for the core system components like KSM., because we can handle this situation much before the next EUS version goes to GA. Please feel free to reopen if you feel otherwise.
Since bz#2079318 fixed the same issue for podsecuritypolicies, I expected this the alert is fixed but any reason why it will not be fixed? We have seen this issue on our serverless-operator CI and it is a blocker now, so we have to ignore the alert message somehow if the alert keeps producing.
*** This bug has been marked as a duplicate of bug 2085326 ***
https://github.com/openshift/kube-state-metrics/pull/72 should already fix this bug. Can you point me to a counter-example if this isn't the case for you?
Ah, I see. I wrongly assumed that the fix would remove the alert itself but actually it updates pdbs deployments.