Bug 2079318

Summary: [4.10] APIRemovedInNextEUSReleaseInUse alert for podsecuritypolicies
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: Insights OperatorAssignee: Tomas Remes <tremes>
Status: CLOSED ERRATA QA Contact: Joao Fula <jfula>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.10CC: inecas, mklika, sburhade, tremes
Target Milestone: ---   
Target Release: 4.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-11 10:31:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2073945    
Bug Blocks:    

Description Junqi Zhao 2022-04-27 11:44:31 UTC 
Description of problem:
4.10.0-0.nightly-2022-04-26-204343 cluster, APIRemovedInNextEUSReleaseInUse alert for podsecuritypolicies is caused by openshift-insights
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "batch",
          "namespace": "openshift-kube-apiserver",
          "resource": "cronjobs",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651057963.326,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "policy",
          "namespace": "openshift-kube-apiserver",
          "resource": "poddisruptionbudgets",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651057963.326,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "policy",
          "namespace": "openshift-kube-apiserver",
          "resource": "podsecuritypolicies",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651057963.326,
          "1"
        ]
      }
    ]
  }
}

# oc get apirequestcounts podsecuritypolicies.v1beta1.policy -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-04-27T05:01:51Z"
  generation: 1
  name: podsecuritypolicies.v1beta1.policy
  resourceVersion: "163664"
  uid: dc255820-a67d-4bbe-a134-62e45db2c6aa
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        requestCount: 1
        userAgent: insights-operator/v0.0.0
        username: system:serviceaccount:openshift-insights:gather
      nodeName: 10.0.52.79
      requestCount: 1
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 4
    requestCount: 5
  last24h:
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:admin
      nodeName: 10.0.59.222
      requestCount: 2
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 2
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      - byVerb:
        - requestCount: 1
          verb: list
        requestCount: 1
        userAgent: insights-operator/v0.0.0
        username: system:serviceaccount:openshift-insights:gather
      nodeName: 10.0.52.79
      requestCount: 3
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 2
          verb: watch
        requestCount: 3
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.53.91
      requestCount: 3
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: list
        - requestCount: 7
          verb: watch
        requestCount: 9
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 9
    requestCount: 15
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 9
          verb: watch
        requestCount: 9
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 9
    requestCount: 9
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        requestCount: 1
        userAgent: insights-operator/v0.0.0
        username: system:serviceaccount:openshift-insights:gather
      nodeName: 10.0.52.79
      requestCount: 1
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 7
          verb: watch
        requestCount: 7
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 7
    requestCount: 8
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 8
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        requestCount: 1
        userAgent: insights-operator/v0.0.0
        username: system:serviceaccount:openshift-insights:gather
      nodeName: 10.0.52.79
      requestCount: 1
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 8
    requestCount: 9
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 8
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        requestCount: 1
        userAgent: insights-operator/v0.0.0
        username: system:serviceaccount:openshift-insights:gather
      nodeName: 10.0.52.79
      requestCount: 1
    - nodeName: 10.0.53.91
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: kube-controller-manager/v1.23.5+70fb84c
        username: system:kube-controller-manager
      nodeName: 10.0.76.223
      requestCount: 4
    requestCount: 5
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.52.79
      requestCount: 0
    - nodeName: 10.0.53.91
      requestCount: 0
    - nodeName: 10.0.59.222
      requestCount: 0
    - nodeName: 10.0.76.223
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 64

# oc get podsecuritypolicies
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+

# oc explain podsecuritypolicies
KIND:     PodSecurityPolicy
VERSION:  policy/v1beta1

APIRemovedInNextEUSReleaseInUse alert detail
**********************
        - alert: APIRemovedInNextEUSReleaseInUse
          annotations:
            description: Deprecated API that will be removed in the next EUS version is
              being used. Removing the workload that is using the {{ $labels.group }}.{{
              $labels.version }}/{{ $labels.resource }} API might be necessary for a successful
              upgrade to the next EUS cluster version. Refer to `oc get apirequestcounts
              {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml`
              to identify the workload.
            summary: Deprecated API that will be removed in the next EUS version is being
              used.
          expr: |
            group(apiserver_requested_deprecated_apis{removed_release=~"1\\.2[45]"}) by (group,version,resource) and (sum by(group,version,resource) (rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h]))) > 0
          for: 1h
          labels:
            namespace: openshift-kube-apiserver
            severity: info
**********************
Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-04-26-204343   True        False         6h4m    Cluster version is 4.10.0-0.nightly-2022-04-26-204343
kubernetes version: v1.23.5+70fb84c

How reproducible:
always

Steps to Reproduce:
1. check alerts
2.
3.

Actual results:
APIRemovedInNextEUSReleaseInUse alert for podsecuritypolicies

Expected results:
no such alert

Additional info:
Comment 3 Joao Fula 2022-05-03 07:48:17 UTC 
Verified on 4.10.0-0.ci-2022-05-02-231040.

Verification steps:
1. oc get apirequestcounts podsecuritypolicies.v1beta1.policy -o yaml | grep insights
 Command returns empty.
2. oc get apirequestcounts podsecuritypolicies.v1beta1.policy -o yaml
 Command does not return empty.
Comment 7 errata-xmlrpc 2022-05-11 10:31:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.10.13 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1690