Bug 1982142 - Level up the resource for Compliance Operator Suite Cronjob
Summary: Level up the resource for Compliance Operator Suite Cronjob
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Compliance Operator
Version: 4.7
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.9.0
Assignee: Jakub Hrozek
QA Contact: Prashant Dhamdhere
URL:
Whiteboard:
Depends On: 1990836
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-14 10:18 UTC by chrzhang
Modified: 2024-10-01 18:58 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-07 06:05:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:3214 0 None None None 2021-09-07 06:05:28 UTC

Description chrzhang 2021-07-14 10:18:39 UTC
Description of problem:

The default cronjob rerunner shows error "OutOfMemory". The customer hope to level up the default resource memory for scan cronjob. 

Version-Release number of selected component (if applicable):

Customer CSV: compliance-operator.v0.1.32

How reproducible:

Can be changed manually by editing the cronjob. 

Steps to Reproduce:
1.oc get cronjob -n openshift-comliance
2.oc edit xxxx-xx-rerunner
3.Update limit/request memory/cpu.

Actual results:

The "OOM" error will show up. 

Expected results:

No warning when job running.

Additional info:

Comment 1 Jakub Hrozek 2021-07-20 07:38:06 UTC
Sorry for the vacation-induced delay. I'll send a patch soon.

Comment 2 Jakub Hrozek 2021-07-23 19:00:05 UTC
We doubled the resources to 20Mi request and 100Mi limit. Can you please check with the customer if this would be OK in their environment (what values did they use)?

Comment 3 chrzhang 2021-07-27 08:31:14 UTC
Hi Jakub,

The customer thinks it's acceptable. The customer used the default resources at the moment. Thank you.

Comment 7 Prashant Dhamdhere 2021-08-25 11:31:59 UTC
[Bug_Verification]

Looks good. The suitererunner resource requests and limits doubled now as per 
https://github.com/openshift/compliance-operator/pull/669 

Verified on:
4.8.0-0.nightly-2021-08-23-234834 + compliance-operator.v0.1.39


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-08-23-234834   True        False         7h4m    Cluster version is 4.8.0-0.nightly-2021-08-23-234834

$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.39       Compliance Operator                0.1.39                Succeeded
elasticsearch-operator.5.1.1-43   OpenShift Elasticsearch Operator   5.1.1-43              Succeeded

$ oc create -f - << EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSetting
> metadata:
>   name: master-scansetting
> # Suite-specific settings
> autoApplyRemediations: true
> schedule: "*/5 * * * *"
> # Scan-specific settings
> rawResultStorage:
>   size: "2Gi"
>   rotation: 5
> # For each role, a separate scan will be created pointing
> # to a node-role specified in roles
> roles:
>   - master
> EOF
scansetting.compliance.openshift.io/master-scansetting created


$ oc get ss -w
NAME                 AGE
default              4h16m
default-auto-apply   4h16m
master-scansetting   15s

$ oc create -f - <<EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSettingBinding 
> metadata:
>   name: compliance-requirements
> profiles:
>   # Cluster checks
>   - name: ocp4-cis
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
>   # Node checks
>   - name: ocp4-cis-node
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
> settingsRef:
>   name: master-scansetting
>   kind: ScanSetting
>   apiGroup: compliance.openshift.io/v1alpha1
> EOF
scansettingbinding.compliance.openshift.io/compliance-requirements created


$ oc get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-cis                                 0/1     Completed   0          3m3s
aggregator-pod-ocp4-cis-node-master                     0/1     Completed   0          3m3s
compliance-operator-55b5469c8f-slqwq                    1/1     Running     1          4h33m
compliance-requirements-rerunner-27164805-487tw         0/1     Completed   0          9m1s
compliance-requirements-rerunner-27164810-6vfnh         0/1     Completed   0          4m1s
ocp4-cis-api-checks-pod                                 0/2     Completed   0          3m37s
ocp4-openshift-compliance-pp-6c9c674b6d-c5j6p           1/1     Running     0          4h31m
openscap-pod-56a57de315ef98cfacb33b08733221f08d3dc750   0/2     Completed   0          3m33s
openscap-pod-a133fa609bd9a9a6bbe0eeb69171a90e879ccc6f   0/2     Completed   0          3m33s
openscap-pod-c2813930046bf4f45a77372d6f8461e4e05d18c3   0/2     Completed   0          3m33s
rhcos4-openshift-compliance-pp-66c856ddd8-4b7ll         1/1     Running     0          4h31m


$ oc get cronjob
NAME                               SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
compliance-requirements-rerunner   */5 * * * *   False     0        89s             12m

$ oc get jobs
NAME                                        COMPLETIONS   DURATION   AGE
compliance-requirements-rerunner-27164810   1/1           8s         12m
compliance-requirements-rerunner-27164815   1/1           9s         7m2s
compliance-requirements-rerunner-27164820   1/1           9s         2m2s


$ for job in $(oc get jobs -lworkload=suitererunner --no-headers |awk '{print $1}'); do echo -e "\n\n >>>>> Print resources request & limit for " $job ; oc get jobs $job -ojsonpath={.spec.template.spec.containers[0].resources}; done


 >>>>> Print resources request & limit for  compliance-requirements-rerunner-27164825
{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"20Mi"}}

 >>>>> Print resources request & limit for  compliance-requirements-rerunner-27164830
{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"20Mi"}}

 >>>>> Print resources request & limit for  compliance-requirements-rerunner-27164835
{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"20Mi"}}


$ oc get pods -lworkload=suitererunner --no-headers
compliance-requirements-rerunner-27164830-vssk4   0/1   Completed   0     12m
compliance-requirements-rerunner-27164835-kqr9k   0/1   Completed   0     7m24s
compliance-requirements-rerunner-27164840-svq7n   0/1   Completed   0     2m24s


$ for pod in $(oc get pods -lworkload=suitererunner --no-headers |awk '{print $1}'); do echo -e "\n\n >>>>> Print resources request & limit for " $pod ; oc get pod $pod -ojsonpath={.spec.containers[0].resources}; done


 >>>>> Print resources request & limit for  compliance-requirements-rerunner-27164830-vssk4
{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"20Mi"}}

 >>>>> Print resources request & limit for  compliance-requirements-rerunner-27164835-kqr9k
{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"20Mi"}}

 >>>>> Print resources request & limit for  compliance-requirements-rerunner-27164840-svq7n
{"limits":{"cpu":"50m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"20Mi"}}

Comment 9 errata-xmlrpc 2021-09-07 06:05:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3214


Note You need to log in before you can comment on or make changes to this bug.