Bug 1862434 - [OCP v46] The Compliance-Operator raw result rotation policy does not rotate old scan results
Summary: [OCP v46] The Compliance-Operator raw result rotation policy does not rotate ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Compliance Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Jakub Hrozek
QA Contact: Prashant Dhamdhere
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-31 12:35 UTC by Prashant Dhamdhere
Modified: 2020-10-27 16:22 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:21:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:22:11 UTC

Description Prashant Dhamdhere 2020-07-31 12:35:48 UTC
Description of problem 

The Compliance-Operator(v0.1.11) raw result rotation policy does not rotate old scan results.
After 4 to 5 scan completes, it rotates or deletes scan result directory from 2 instead of 0 

o On 4th scan the status is like below: 

$ oc get pods 
NAME                                                         READY   STATUS      RESTARTS   AGE 
aggregator-pod-worker-scan                                   0/1     Completed   0          2m6s  <<--[4] 
compliance-operator-869646dd4f-dpg6l                         1/1     Running     0          5h43m 
compliance-operator-869646dd4f-gjfp9                         1/1     Running     0          5h43m 
compliance-operator-869646dd4f-sm5q7                         1/1     Running     0          5h43m 
example-compliancesuite-rerunner-1596195000-pv4n9            0/1     Completed   0          8m58s <<-- [1] 
example-compliancesuite-rerunner-1596195180-j7j85            0/1     Completed   0          5m57s <<-- [2] 
example-compliancesuite-rerunner-1596195360-2nrhl            0/1     Completed   0          2m56s <<-- [3] 
ocp4-pp-dcb8bc5b5-wn6qh                                      1/1     Running     0          5h42m 
pv-extract                                                   1/1     Running     0          9m56s 
rhcos4-pp-58466496cf-lm4rr                                   1/1     Running     0          5h42m 
worker-scan-ip-10-0-150-75.us-east-2.compute.internal-pod    0/2     Completed   0          2m46s 
worker-scan-ip-10-0-165-73.us-east-2.compute.internal-pod    0/2     Completed   0          2m46s 
worker-scan-ip-10-0-196-245.us-east-2.compute.internal-pod   0/2     Completed   0          2m46s 

o The scan performs successfully and the index value is 3 

$ oc describe compliancesuite example-compliancesuite |grep -A14 Status: 
Status: 
  Aggregated Phase:   DONE 
  Aggregated Result:  COMPLIANT 
  Scan Statuses: 
    Current Index:  3               <<----- 
    Name:           worker-scan 
    Phase:          DONE 
    Result:         COMPLIANT 
    Results Storage: 
      Name:       worker-scan 
      Namespace:  openshift-compliance 
Events: 
  Type    Reason           Age                   From       Message 
  ----    ------           ----                  ----       ------- 
  Normal  ResultAvailable  49s (x15 over 9m53s)  suitectrl  ComplianceSuite's result is: COMPLIANT 

o The following raw result of scan stores in /workers-scan-results/ directory

sh-4.4# ls -l /workers-scan-results/ 
total 32 
drwxrws---. 2 1000560000 1000560000  4096 Jul 31 11:27 0 
drwxrws---. 2 1000560000 1000560000  4096 Jul 31 11:30 1 
drwxrws---. 2 1000560000 1000560000  4096 Jul 31 11:33 2 
drwxr-s---. 2 1000560000 1000560000  4096 Jul 31 11:36 3 
drwxrws---. 2 root       1000560000 16384 Jul 31 11:26 lost+found 

o On 5th scan the status is like below : 

$ oc get pods 
NAME                                                         READY   STATUS      RESTARTS   AGE 
aggregator-pod-worker-scan                                   0/1     Completed   0          40s   <<-- [5] 
compliance-operator-869646dd4f-dpg6l                         1/1     Running     0          5h45m 
compliance-operator-869646dd4f-gjfp9                         1/1     Running     0          5h45m 
compliance-operator-869646dd4f-sm5q7                         1/1     Running     0          5h45m 
example-compliancesuite-rerunner-1596195180-j7j85            0/1     Completed   0          7m32s <<-- [2] 
example-compliancesuite-rerunner-1596195360-2nrhl            0/1     Completed   0          4m31s <<-- [3] 
example-compliancesuite-rerunner-1596195540-hqg84            0/1     Completed   0          90s   <<-- [4] 
ocp4-pp-dcb8bc5b5-wn6qh                                      1/1     Running     0          5h44m 
pv-extract                                                   1/1     Running     0          11m 
rhcos4-pp-58466496cf-lm4rr                                   1/1     Running     0          5h44m 
worker-scan-ip-10-0-150-75.us-east-2.compute.internal-pod    0/2     Completed   0          81s 
worker-scan-ip-10-0-165-73.us-east-2.compute.internal-pod    0/2     Completed   0          81s 
worker-scan-ip-10-0-196-245.us-east-2.compute.internal-pod   0/2     Completed   0          81s 

o The scan performs successfully and the index value is 4 

$ oc describe compliancesuite example-compliancesuite |grep -A14 Status: 
Status: 
  Aggregated Phase:   RUNNING 
  Aggregated Result:  NOT-AVAILABLE 
  Scan Statuses: 
    Current Index:  4 
    Name:           worker-scan 
    Phase:          RUNNING 
    Result:         NOT-AVAILABLE 
    Results Storage: 
      Name:       worker-scan 
      Namespace:  openshift-compliance 
Events: 
  Type    Reason           Age                 From       Message 
  ----    ------           ----                ----       ------- 
  Normal  ResultAvailable  10s (x18 over 11m)  suitectrl  ComplianceSuite's result is: COMPLIANT 

o The following raw result of scan stores in /workers-scan-results/ but instead of rotates directory 0, 
  it starts rotation from directory 2

sh-4.4# ls -l /workers-scan-results/   
total 32 
drwxrws---. 2 1000560000 1000560000  4096 Jul 31 11:27 0 
drwxrws---. 2 1000560000 1000560000  4096 Jul 31 11:30 1 
drwxrws---. 2 1000560000 1000560000  4096 Jul 31 11:36 3 
drwxr-s---. 2 1000560000 1000560000  4096 Jul 31 11:39 4 
drwxrws---. 2 root       1000560000 16384 Jul 31 11:26 lost+found 



Version-Release -Cluster version  

4.6.0-0.nightly-2020-07-25-091217 

How reproducible: 

Always  

Steps: 

1 clone compliance-operator git repo 

$ git clone https://github.com/openshift/compliance-operator.git 

2 Create 'openshift-compliance' namespace 

$ oc create -f compliance-operator/deploy/ns.yaml   

3 Switch to 'openshift-compliance' namespace 

$ oc project openshift-compliance 

4 Deploy CustomResourceDefinition. 

$ for f in $(ls -1 compliance-operator/deploy/crds/*crd.yaml); do oc create -f $f; done 

5. Deploy compliance-operator. 

$ oc create -f compliance-operator/deploy/ 

6.Set rotation policy 4 & schedule a scan after every 3 mins. Deploy ComplianceSuite CR 

$ oc create -f - <<EOF 
apiVersion: compliance.openshift.io/v1alpha1 
kind: ComplianceSuite 
metadata: 
  name: example-compliancesuite 
spec: 
  autoApplyRemediations: false 
  schedule: "*/3 * * * *" 
  scans: 
    - name: worker-scan 
      profile: xccdf_org.ssgproject.content_profile_moderate 
      content: ssg-rhcos4-ds.xml 
      contentImage: quay.io/complianceascode/ocp4:latest 
      rule: "xccdf_org.ssgproject.content_rule_no_netrc_files" 
      nodeSelector: 
        node-role.kubernetes.io/worker: "" 
      rawResultStorage: 
        rotation: 4 
EOF 

7. Verify the scan performs successfully

$ oc get pods 
$ oc get compliancesuite 
$ oc describe compliancesuite example-compliancesuite|grep -A14 Status: 

8. Create a pod to mount PV 

$ oc create -f - <<EOF 
apiVersion: "v1" 
kind: Pod 
metadata: 
  name: pv-extract 
spec: 
  containers: 
    - name: pv-extract-pod 
      image: registry.access.redhat.com/ubi8/ubi 
      command: ["sleep", "3000"] 
      volumeMounts: 
        - mountPath: "/workers-scan-results" 
          name: workers-scan-vol 
  volumes: 
    - name: workers-scan-vol 
      persistentVolumeClaim: 
        claimName: worker-scan   
EOF 

9. Login to the pv-extract pod and list the raw scan result directory to monitor if the raw 
   result rotates after 4,5 scan as rotation policy set to 4

$ oc rsh pv-extract 
sh-4.4# ls -l /workers-scan-results/ 


Actual result: 

The Compliance-Operator raw result rotation policy does not rotate or delete old scan results. 
It starts rotation from raw scan result directory 2 instead of 0. 


Expected results 

The Compliance-Operator raw result rotation policy should rotate or delete old results first i.e directory 0 & 1.  

Additional info

Comment 2 Jakub Hrozek 2020-08-28 13:43:58 UTC
Thanks for the bug report, I can reproduce.

Comment 3 Jakub Hrozek 2020-08-28 19:44:34 UTC
PR: https://github.com/openshift/compliance-operator/pull/419

Comment 7 Prashant Dhamdhere 2020-09-10 17:37:50 UTC
Yes. Now, The Compliance-Operator raw result rotation policy does rotate old scan results and it 
starts delete scan result directory from 0 instead of 2.


Verified on:

OCP version: 4.6.0-0.nightly-2020-09-10-054902
Compliance Operator : v0.1.15


$ oc get pods 
NAME                                                READY   STATUS              RESTARTS   AGE
aggregator-pod-worker-scan                          0/1     Completed           0          116s
compliance-operator-869646dd4f-n7lxn                1/1     Running             0          3h40m
example-compliancesuite-rerunner-1599758280-cw9zz   0/1     Completed           0          9m3s
example-compliancesuite-rerunner-1599758460-xkgjb   0/1     Completed           0          6m2s
example-compliancesuite-rerunner-1599758640-cczpc   0/1     Completed           0          3m10s
example-compliancesuite-rerunner-1599758820-dxtdc   0/1     ContainerCreating   0          2s
ocp4-pp-6786c5f5b-vnk6c                             1/1     Running             0          3h39m
pv-extract                                          1/1     Running             0          10m
rhcos4-pp-78c8cc9d44-r4zns                          1/1     Running             0          3h39m
worker-scan-osp-pdhamdhe-4kndt-worker-0-4rqz7-pod   0/2     Completed           0          2m57s
worker-scan-osp-pdhamdhe-4kndt-worker-0-7pg4x-pod   0/2     Completed           0          2m56s
worker-scan-osp-pdhamdhe-4kndt-worker-0-bpwvw-pod   0/2     Completed           0          2m57s

$ oc describe compliancesuite example-compliancesuite |grep -A14 Status: 
Status:
  Phase:   DONE
  Result:  COMPLIANT
  Scan Statuses:
    Current Index:  3
    Name:           worker-scan
    Phase:          DONE
    Result:         COMPLIANT
    Results Storage:
      Name:       worker-scan
      Namespace:  openshift-compliance
Events:
  Type    Reason           Age                   From       Message
  ----    ------           ----                  ----       -------
  Normal  ResultAvailable  78s (x12 over 9m41s)  suitectrl  ComplianceSuite's result is: COMPLIANT


sh-4.4#  ls -l /workers-scan-results/ 
total 32
drwxrws---. 2 1000590000 1000590000  4096 Sep 10 17:16 0
drwxrws---. 2 1000590000 1000590000  4096 Sep 10 17:19 1
drwxrws---. 2 1000590000 1000590000  4096 Sep 10 17:22 2
drwxr-s---. 2 1000590000 1000590000  4096 Sep 10 17:25 3
drwxrws---. 2 root       1000590000 16384 Sep 10 17:16 lost+found


$ oc get pods 
NAME                                                READY   STATUS      RESTARTS   AGE
aggregator-pod-worker-scan                          0/1     Completed   0          64s
compliance-operator-869646dd4f-n7lxn                1/1     Running     0          3h42m
example-compliancesuite-rerunner-1599758460-xkgjb   0/1     Completed   0          8m14s
example-compliancesuite-rerunner-1599758640-cczpc   0/1     Completed   0          5m22s
example-compliancesuite-rerunner-1599758820-dxtdc   0/1     Completed   0          2m14s
ocp4-pp-6786c5f5b-vnk6c                             1/1     Running     0          3h41m
pv-extract                                          1/1     Running     0          13m
rhcos4-pp-78c8cc9d44-r4zns                          1/1     Running     0          3h41m
worker-scan-osp-pdhamdhe-4kndt-worker-0-4rqz7-pod   0/2     Completed   0          2m5s
worker-scan-osp-pdhamdhe-4kndt-worker-0-7pg4x-pod   0/2     Completed   0          2m4s
worker-scan-osp-pdhamdhe-4kndt-worker-0-bpwvw-pod   0/2     Completed   0          2m4s


$ oc describe compliancesuite example-compliancesuite |grep -A14 Status: 
Status:
  Phase:   DONE
  Result:  COMPLIANT
  Scan Statuses:
    Current Index:  4
    Name:           worker-scan
    Phase:          DONE
    Result:         COMPLIANT
    Results Storage:
      Name:       worker-scan
      Namespace:  openshift-compliance
Events:
  Type    Reason           Age                 From       Message
  ----    ------           ----                ----       -------
  Normal  ResultAvailable  46s (x15 over 12m)  suitectrl  ComplianceSuite's result is: COMPLIANT


sh-4.4#  ls -l /workers-scan-results/ 
total 32
drwxrws---. 2 1000590000 1000590000  4096 Sep 10 17:19 1
drwxrws---. 2 1000590000 1000590000  4096 Sep 10 17:22 2
drwxrws---. 2 1000590000 1000590000  4096 Sep 10 17:25 3
drwxr-s---. 2 1000590000 1000590000  4096 Sep 10 17:28 4
drwxrws---. 2 root       1000590000 16384 Sep 10 17:16 lost+found

Comment 9 errata-xmlrpc 2020-10-27 16:21:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.