Bug 2098581

Summary: APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: Compliance OperatorAssignee: Matt Rogers <mrogers>
Status: CLOSED ERRATA QA Contact:
Severity: low Docs Contact: Jeana Routh <jrouth>
Priority: low    
Version: 4.10CC: jhrozek, jrouth, lbragsta, mrogers, wenshen, xiyuan
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
* Previously, the Compliance Operator used an old version of the Operator SDK, which is a dependency for building Operators. This caused alerts about deprecated Kubernetes functionality used by the Operator SDK. With this release, the Compliance Operator is upgraded to version 0.1.55, which includes an updated version of the Operator SDK. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2098581[*BZ#2098581*])
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-02 16:00:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2022-06-20 06:54:03 UTC
Description of problem:
found in 4.10.17 AWS OSD cluster, APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs
$ oc -n openshift-compliance get pod
NAME                                              READY   STATUS    RESTARTS        AGE
compliance-operator-594c9d547f-fndhc              1/1     Running   13 (163m ago)   4h50m
ocp4-openshift-compliance-pp-5cd896b74c-tk86s     1/1     Running   0               4h48m
rhcos4-openshift-compliance-pp-78bf7c5bf9-z96g4   1/1     Running   0               4h48m

$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-compliance exec compliance-operator-594c9d547f-fndhc -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "batch",
          "namespace": "openshift-kube-apiserver",
          "resource": "cronjobs",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1655707821.174,
          "1"
        ]
      }
    ]
  }
}


$ oc get apirequestcounts cronjobs.v1beta1.batch -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-06-20T01:01:47Z"
  generation: 1
  name: cronjobs.v1beta1.batch
  resourceVersion: "287502"
  uid: 35c1d2b3-02d3-47e2-bc7e-747911d92c4f
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 4
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 4
  last24h:
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 5
          verb: create
        - requestCount: 5
          verb: get
        requestCount: 10
        userAgent: manager/v0.0.0
        username: system:admin
      - byVerb:
        - requestCount: 1
          verb: create
        - requestCount: 1
          verb: get
        requestCount: 2
        userAgent: Go-http-client/2.0
        username: system:admin
      nodeName: 10.0.179.188
      requestCount: 12
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 12
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: create
        - requestCount: 2
          verb: get
        requestCount: 4
        userAgent: manager/v0.0.0
        username: system:admin
      nodeName: 10.0.179.188
      requestCount: 4
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 4
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: get
        requestCount: 8
        userAgent: manager/v0.0.0
        username: system:admin
      nodeName: 10.0.138.228
      requestCount: 8
    - nodeName: 10.0.179.188
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: create
        - requestCount: 2
          verb: list
        - requestCount: 6
          verb: watch
        requestCount: 9
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.195.85
      requestCount: 9
    requestCount: 17
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 8
          verb: watch
        requestCount: 9
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 9
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 9
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 8
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: get
        requestCount: 8
        userAgent: manager/v0.0.0
        username: system:admin
      nodeName: 10.0.195.85
      requestCount: 8
    requestCount: 16
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 4
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 4
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 62

cronjobs version now is v1
$ oc explain cronjobs
KIND:     CronJob
VERSION:  batch/v1



Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.10.17
Server Version: 4.10.17
Kubernetes Version: v1.23.5+3afdacb


How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs

Expected results:
no such alert

Additional info:
reason maybe in https://github.com/openshift/compliance-operator/blob/release-4.10/pkg/controller/compliancesuite/suitererunner.go#L10

Comment 1 Jakub Hrozek 2022-06-20 08:45:49 UTC
This will be fixed once we merge the operator-sdk upgrade

Comment 2 Jakub Hrozek 2022-06-20 08:46:44 UTC
Assigning to Matt who works on the SDK upgrade.

Comment 7 xiyuan 2022-09-26 03:19:01 UTC
Hi Jakub, 
Tried to verify with 4.12.0-0.nightly-2022-09-25-071630 and compliance-operator.v0.1.55, the alert still exists. Could you please help to double check? Thanks.
$ token=`oc  create token prometheus-k8s -n openshift-monitoring`
$  oc -n openshift-compliance exec compliance-operator-7489d57b55-6c2j5  -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   403    0   308  100    95   1974    608 --:--:-- --:--:-- --:--:--  2583
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "pending",
          "group": "batch",
          "namespace": "openshift-kube-apiserver",
          "resource": "cronjobs",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1664161876.437,
          "1"
        ]
      }
    ]
  }
}
$ oc get apirequestcounts cronjobs.v1beta1.batch -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-09-26T02:35:22Z"
  generation: 1
  name: cronjobs.v1beta1.batch
  resourceVersion: "66324"
  uid: 4ff4faaf-1b43-4ed6-8523-54bbd1d83e66
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.0.6
      requestCount: 1
    requestCount: 1
  last24h:
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: create
        - requestCount: 1
          verb: delete
        - requestCount: 1
          verb: list
        - requestCount: 4
          verb: watch
        requestCount: 8
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.0.6
      requestCount: 8
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.0.6
      requestCount: 1
    requestCount: 1
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 9
$ oc explain cronjobs
KIND:     CronJob
VERSION:  batch/v1

DESCRIPTION:
     CronJob represents the configuration of a single cron job.

...

Comment 8 Jakub Hrozek 2022-09-26 09:39:09 UTC
Thanks, I posted a new patch.

Comment 9 xiyuan 2022-09-26 13:46:12 UTC
bug verification pass with pre-merge process.The APIRemovedInNextEUSReleaseInUse alert won't fired for openshift-compliance cronjobs

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-09-25-071630   True        False         11h     Cluster version is 4.12.0-0.nightly-2022-09-25-071630
$ oc get pod
NAME                                              READY   STATUS    RESTARTS        AGE
compliance-operator-d47877fbd-4db77               1/1     Running   1 (6m25s ago)   6m33s
ocp4-openshift-compliance-pp-7cb94bcdc6-nhjwj     1/1     Running   0               6m13s
rhcos4-openshift-compliance-pp-67c79bc97b-4qtl4   1/1     Running   0               6m13s
$ token=`oc  create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-compliance exec compliance-operator-d47877fbd-4db77  -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
time="2022-09-26T13:17:59Z" level=error msg="exec failed: unable to start container process: exec: \"curl\": executable file not found in $PATH"
command terminated with exit code 255

Comment 11 xiyuan 2022-09-29 04:55:53 UTC
Verification pass with 4.12.0-0.nightly-2022-09-26-111919 + compliance-operator.v0.1.56

$  token=`oc  create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-compliance exec compliance-operator-7799fd4cd9-wsb9b  -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   158    0    63  100    95   1285   1938 --:--:-- --:--:-- --:--:--  3224
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": []
  }
}

Comment 13 errata-xmlrpc 2022-11-02 16:00:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6657