Bug 2098581 - APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs
Summary: APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Compliance Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.12.0
Assignee: Matt Rogers
QA Contact:
Jeana Routh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-20 06:54 UTC by Junqi Zhao
Modified: 2022-12-22 19:22 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
* Previously, the Compliance Operator used an old version of the Operator SDK, which is a dependency for building Operators. This caused alerts about deprecated Kubernetes functionality used by the Operator SDK. With this release, the Compliance Operator is upgraded to version 0.1.55, which includes an updated version of the Operator SDK. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2098581[*BZ#2098581*])
Clone Of:
Environment:
Last Closed: 2022-11-02 16:00:53 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ComplianceAsCode compliance-operator pull 143 0 None open bug 2098581: Don't import batchv1beta1 anymore 2022-09-26 09:38:29 UTC
Github ComplianceAsCode compliance-operator pull 41 0 None open [WIP] operator-sdk upgrade 2022-06-20 08:45:49 UTC
Red Hat Product Errata RHBA-2022:6657 0 None None None 2022-11-02 16:01:09 UTC

Description Junqi Zhao 2022-06-20 06:54:03 UTC 
Description of problem:
found in 4.10.17 AWS OSD cluster, APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs
$ oc -n openshift-compliance get pod
NAME                                              READY   STATUS    RESTARTS        AGE
compliance-operator-594c9d547f-fndhc              1/1     Running   13 (163m ago)   4h50m
ocp4-openshift-compliance-pp-5cd896b74c-tk86s     1/1     Running   0               4h48m
rhcos4-openshift-compliance-pp-78bf7c5bf9-z96g4   1/1     Running   0               4h48m

$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-compliance exec compliance-operator-594c9d547f-fndhc -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "batch",
          "namespace": "openshift-kube-apiserver",
          "resource": "cronjobs",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1655707821.174,
          "1"
        ]
      }
    ]
  }
}


$ oc get apirequestcounts cronjobs.v1beta1.batch -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-06-20T01:01:47Z"
  generation: 1
  name: cronjobs.v1beta1.batch
  resourceVersion: "287502"
  uid: 35c1d2b3-02d3-47e2-bc7e-747911d92c4f
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 4
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 4
  last24h:
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 5
          verb: create
        - requestCount: 5
          verb: get
        requestCount: 10
        userAgent: manager/v0.0.0
        username: system:admin
      - byVerb:
        - requestCount: 1
          verb: create
        - requestCount: 1
          verb: get
        requestCount: 2
        userAgent: Go-http-client/2.0
        username: system:admin
      nodeName: 10.0.179.188
      requestCount: 12
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 12
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: create
        - requestCount: 2
          verb: get
        requestCount: 4
        userAgent: manager/v0.0.0
        username: system:admin
      nodeName: 10.0.179.188
      requestCount: 4
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 4
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: get
        requestCount: 8
        userAgent: manager/v0.0.0
        username: system:admin
      nodeName: 10.0.138.228
      requestCount: 8
    - nodeName: 10.0.179.188
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: create
        - requestCount: 2
          verb: list
        - requestCount: 6
          verb: watch
        requestCount: 9
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.195.85
      requestCount: 9
    requestCount: 17
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 8
          verb: watch
        requestCount: 9
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 9
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 9
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 8
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: get
        requestCount: 8
        userAgent: manager/v0.0.0
        username: system:admin
      nodeName: 10.0.195.85
      requestCount: 8
    requestCount: 16
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.179.188
      requestCount: 4
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 4
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.138.228
      requestCount: 0
    - nodeName: 10.0.179.188
      requestCount: 0
    - nodeName: 10.0.195.85
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 62

cronjobs version now is v1
$ oc explain cronjobs
KIND:     CronJob
VERSION:  batch/v1



Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.10.17
Server Version: 4.10.17
Kubernetes Version: v1.23.5+3afdacb


How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
APIRemovedInNextEUSReleaseInUse alert fired for openshift-compliance cronjobs

Expected results:
no such alert

Additional info:
reason maybe in https://github.com/openshift/compliance-operator/blob/release-4.10/pkg/controller/compliancesuite/suitererunner.go#L10
Comment 1 Jakub Hrozek 2022-06-20 08:45:49 UTC 
This will be fixed once we merge the operator-sdk upgrade
Comment 2 Jakub Hrozek 2022-06-20 08:46:44 UTC 
Assigning to Matt who works on the SDK upgrade.
Comment 7 xiyuan 2022-09-26 03:19:01 UTC 
Hi Jakub, 
Tried to verify with 4.12.0-0.nightly-2022-09-25-071630 and compliance-operator.v0.1.55, the alert still exists. Could you please help to double check? Thanks.
$ token=`oc  create token prometheus-k8s -n openshift-monitoring`
$  oc -n openshift-compliance exec compliance-operator-7489d57b55-6c2j5  -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   403    0   308  100    95   1974    608 --:--:-- --:--:-- --:--:--  2583
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "pending",
          "group": "batch",
          "namespace": "openshift-kube-apiserver",
          "resource": "cronjobs",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1664161876.437,
          "1"
        ]
      }
    ]
  }
}
$ oc get apirequestcounts cronjobs.v1beta1.batch -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-09-26T02:35:22Z"
  generation: 1
  name: cronjobs.v1beta1.batch
  resourceVersion: "66324"
  uid: 4ff4faaf-1b43-4ed6-8523-54bbd1d83e66
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.0.6
      requestCount: 1
    requestCount: 1
  last24h:
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: create
        - requestCount: 1
          verb: delete
        - requestCount: 1
          verb: list
        - requestCount: 4
          verb: watch
        requestCount: 8
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.0.6
      requestCount: 8
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: compliance-operator/v0.0.0
        username: system:serviceaccount:openshift-compliance:compliance-operator
      nodeName: 10.0.0.6
      requestCount: 1
    requestCount: 1
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 9
$ oc explain cronjobs
KIND:     CronJob
VERSION:  batch/v1

DESCRIPTION:
     CronJob represents the configuration of a single cron job.

...
Comment 8 Jakub Hrozek 2022-09-26 09:39:09 UTC 
Thanks, I posted a new patch.
Comment 9 xiyuan 2022-09-26 13:46:12 UTC 
bug verification pass with pre-merge process.The APIRemovedInNextEUSReleaseInUse alert won't fired for openshift-compliance cronjobs

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-09-25-071630   True        False         11h     Cluster version is 4.12.0-0.nightly-2022-09-25-071630
$ oc get pod
NAME                                              READY   STATUS    RESTARTS        AGE
compliance-operator-d47877fbd-4db77               1/1     Running   1 (6m25s ago)   6m33s
ocp4-openshift-compliance-pp-7cb94bcdc6-nhjwj     1/1     Running   0               6m13s
rhcos4-openshift-compliance-pp-67c79bc97b-4qtl4   1/1     Running   0               6m13s
$ token=`oc  create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-compliance exec compliance-operator-d47877fbd-4db77  -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
time="2022-09-26T13:17:59Z" level=error msg="exec failed: unable to start container process: exec: \"curl\": executable file not found in $PATH"
command terminated with exit code 255
Comment 11 xiyuan 2022-09-29 04:55:53 UTC 
Verification pass with 4.12.0-0.nightly-2022-09-26-111919 + compliance-operator.v0.1.56

$  token=`oc  create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-compliance exec compliance-operator-7799fd4cd9-wsb9b  -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="cronjobs"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   158    0    63  100    95   1285   1938 --:--:-- --:--:-- --:--:--  3224
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": []
  }
}
Comment 13 errata-xmlrpc 2022-11-02 16:00:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6657
Note You need to log in before you can comment on or make changes to this bug.