Description of problem: 4.10.0-0.nightly-2022-04-26-204343 cluster, APIRemovedInNextEUSReleaseInUse alert for podsecuritypolicies is caused by openshift-insights # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse"}' | jq { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextEUSReleaseInUse", "alertstate": "firing", "group": "batch", "namespace": "openshift-kube-apiserver", "resource": "cronjobs", "severity": "info", "version": "v1beta1" }, "value": [ 1651057963.326, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextEUSReleaseInUse", "alertstate": "firing", "group": "policy", "namespace": "openshift-kube-apiserver", "resource": "poddisruptionbudgets", "severity": "info", "version": "v1beta1" }, "value": [ 1651057963.326, "1" ] }, { "metric": { "__name__": "ALERTS", "alertname": "APIRemovedInNextEUSReleaseInUse", "alertstate": "firing", "group": "policy", "namespace": "openshift-kube-apiserver", "resource": "podsecuritypolicies", "severity": "info", "version": "v1beta1" }, "value": [ 1651057963.326, "1" ] } ] } } # oc get apirequestcounts podsecuritypolicies.v1beta1.policy -o yaml apiVersion: apiserver.openshift.io/v1 kind: APIRequestCount metadata: creationTimestamp: "2022-04-27T05:01:51Z" generation: 1 name: podsecuritypolicies.v1beta1.policy resourceVersion: "163664" uid: dc255820-a67d-4bbe-a134-62e45db2c6aa spec: numberOfUsersToReport: 10 status: currentHour: byNode: - byUser: - byVerb: - requestCount: 1 verb: list requestCount: 1 userAgent: insights-operator/v0.0.0 username: system:serviceaccount:openshift-insights:gather nodeName: 10.0.52.79 requestCount: 1 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 4 verb: watch requestCount: 4 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 4 requestCount: 5 last24h: - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 1 verb: list - requestCount: 1 verb: watch requestCount: 2 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:admin nodeName: 10.0.59.222 requestCount: 2 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 2 - byNode: - byUser: - byVerb: - requestCount: 1 verb: list - requestCount: 1 verb: watch requestCount: 2 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager - byVerb: - requestCount: 1 verb: list requestCount: 1 userAgent: insights-operator/v0.0.0 username: system:serviceaccount:openshift-insights:gather nodeName: 10.0.52.79 requestCount: 3 - byUser: - byVerb: - requestCount: 1 verb: list - requestCount: 2 verb: watch requestCount: 3 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.53.91 requestCount: 3 - byUser: - byVerb: - requestCount: 2 verb: list - requestCount: 7 verb: watch requestCount: 9 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 9 requestCount: 15 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 9 verb: watch requestCount: 9 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 9 requestCount: 9 - byNode: - byUser: - byVerb: - requestCount: 1 verb: list requestCount: 1 userAgent: insights-operator/v0.0.0 username: system:serviceaccount:openshift-insights:gather nodeName: 10.0.52.79 requestCount: 1 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 7 verb: watch requestCount: 7 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 7 requestCount: 8 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 8 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 1 verb: list requestCount: 1 userAgent: insights-operator/v0.0.0 username: system:serviceaccount:openshift-insights:gather nodeName: 10.0.52.79 requestCount: 1 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 8 requestCount: 9 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 8 verb: watch requestCount: 8 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 8 requestCount: 8 - byNode: - byUser: - byVerb: - requestCount: 1 verb: list requestCount: 1 userAgent: insights-operator/v0.0.0 username: system:serviceaccount:openshift-insights:gather nodeName: 10.0.52.79 requestCount: 1 - nodeName: 10.0.53.91 requestCount: 0 - byUser: - byVerb: - requestCount: 4 verb: watch requestCount: 4 userAgent: kube-controller-manager/v1.23.5+70fb84c username: system:kube-controller-manager nodeName: 10.0.76.223 requestCount: 4 requestCount: 5 - requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 - byNode: - nodeName: 10.0.52.79 requestCount: 0 - nodeName: 10.0.53.91 requestCount: 0 - nodeName: 10.0.59.222 requestCount: 0 - nodeName: 10.0.76.223 requestCount: 0 requestCount: 0 removedInRelease: "1.25" requestCount: 64 # oc get podsecuritypolicies Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ # oc explain podsecuritypolicies KIND: PodSecurityPolicy VERSION: policy/v1beta1 APIRemovedInNextEUSReleaseInUse alert detail ********************** - alert: APIRemovedInNextEUSReleaseInUse annotations: description: Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload. summary: Deprecated API that will be removed in the next EUS version is being used. expr: | group(apiserver_requested_deprecated_apis{removed_release=~"1\\.2[45]"}) by (group,version,resource) and (sum by(group,version,resource) (rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h]))) > 0 for: 1h labels: namespace: openshift-kube-apiserver severity: info ********************** Version-Release number of selected component (if applicable): # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-04-26-204343 True False 6h4m Cluster version is 4.10.0-0.nightly-2022-04-26-204343 kubernetes version: v1.23.5+70fb84c How reproducible: always Steps to Reproduce: 1. check alerts 2. 3. Actual results: APIRemovedInNextEUSReleaseInUse alert for podsecuritypolicies Expected results: no such alert Additional info:
Verified on 4.10.0-0.ci-2022-05-02-231040. Verification steps: 1. oc get apirequestcounts podsecuritypolicies.v1beta1.policy -o yaml | grep insights Command returns empty. 2. oc get apirequestcounts podsecuritypolicies.v1beta1.policy -o yaml Command does not return empty.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10.13 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1690