Bug 1975825 - [v4.8] The `oc compliance fetch-raw` is unable to process results from suite: unexpected EOF
Summary: [v4.8] The `oc compliance fetch-raw` is unable to process results from suite:...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc-compliance
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.9.0
Assignee: Juan Antonio Osorio
QA Contact: Prashant Dhamdhere
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-24 13:59 UTC by Prashant Dhamdhere
Modified: 2021-10-18 17:36 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-18 17:36:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc-compliance pull 63 0 None open Bug 1975825: Only wait for the pod that was generated by us 2021-07-12 13:38:33 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:36:48 UTC

Description Prashant Dhamdhere 2021-06-24 13:59:09 UTC
Description of problem:

The `oc compliance fetch-raw` is unable to process results from suite: unexpected EOF

$ oc-compliance fetch-raw scansettingbinding compliance-requirements -o resultsdir/ 
Fetching results for compliance-requirements scans: ocp4-cis, ocp4-cis-node-master
Fetching raw compliance results for scan 'ocp4-cis'...
Error: Unable to process results from suite compliance-requirements: unexpected EOF  <<----

$ oc-compliance fetch-raw scansettingbinding compliance-requirements -o resultsdir/ 
Fetching results for compliance-requirements scans: ocp4-cis, ocp4-cis-node-master
Fetching raw compliance results for scan 'ocp4-cis'.
error: unable to upgrade connection: container not found ("pv-extract-pod")  <<----


$ oc get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-cis                                 0/1     Completed   0          140m
aggregator-pod-ocp4-cis-node-master                     0/1     Completed   0          119m
compliance-operator-5bd49c8dd5-m7qkj                    1/1     Running     0          5h20m
ocp4-cis-api-checks-pod                                 0/2     Completed   0          141m
ocp4-openshift-compliance-pp-7ddd9fd849-mxlw2           1/1     Running     0          5h19m
openscap-pod-7754b35cb80f0e6a8670ba6fc2974e1f642c99ab   0/2     Completed   0          119m
openscap-pod-cbcd43e9ac9a5558b49de21cda9d425d74aa8ccb   0/2     Completed   0          119m
openscap-pod-fb3a8ebc9431561caf9d4a5d5b0dfe6996e122dc   0/2     Completed   0          119m
raw-result-extractor-2zbfd                              1/1     Running     0          96m  <<---
raw-result-extractor-kwcbr                              1/1     Running     0          52s   <<---
raw-result-extractor-pn98l                              1/1     Running     0          97m   <<---
raw-result-extractor-pp7wt                              1/1     Running     0          94m <<---
raw-result-extractor-slfzb                              1/1     Running     0          13s   <<---
rhcos4-openshift-compliance-pp-577699c677-qzh24         1/1     Running     0          5h19m


Version-Release number of selected components (if applicable):
4.8.0-0.nightly-2021-06-23-201305 + compliance-operator.v0.1.35
Latest oc-compliance build


How reproducible:
always 


Steps to Reproduce:

1. Install compliance operator
2. Create a scansettingbinding:

oc create -f -<<EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: my-ssb-r
profiles:
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
EOF

3. Check scan status

$ oc get compliancesuite -w

4. Fetch raw result using oc-compliance
$ mkdir resultsdir
$ oc-compliance fetch-raw scansettingbinding compliance-requirements -o resultsdir/


Actual results:
The `oc compliance fetch-raw` is unable to process results from suite and unexpected EOF reported


Expected results:
The `oc compliance fetch-raw` should able to process results from suite and raw result store in
directory. Also the raw-result-extractor pod should get terminated.


Additional info:
So far this issue is noticed on IPI on GCP cluster with OVN-IPSec enabled.

Comment 3 Prashant Dhamdhere 2021-10-12 14:23:48 UTC
[Bug_verification]

Looks good. The `oc compliance fetch-raw` is able to fetch results from compliance objects
and also, the raw-result-extractor pod is getting terminated once the result gets fetched.

Verified on:
4.9.0-0.nightly-2021-10-11-151207 + compliance-operator.v0.1.41 
oc-compliance build https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1672674 
IPI on AWS cluster with OVN-IPSec enabled


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-10-11-151207   True        False         9h      Cluster version is 4.9.0-0.nightly-2021-10-11-151207

$ oc get ds -n openshift-ovn-kubernetes ovn-ipsec
NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
ovn-ipsec   6         6         6       6            6           beta.kubernetes.io/os=linux   10h

$ oc get pod -n openshift-ovn-kubernetes -o wide | grep ipsec
ovn-ipsec-5vqps        1/1     Running   9 (23m ago)     10h   10.0.183.232   ip-10-0-183-232.us-east-2.compute.internal   <none>           <none>
ovn-ipsec-7d8r9        1/1     Running   9 (10m ago)     10h   10.0.159.101   ip-10-0-159-101.us-east-2.compute.internal   <none>           <none>
ovn-ipsec-8xwk2        0/1     Running   12 (2m3s ago)   10h   10.0.176.61    ip-10-0-176-61.us-east-2.compute.internal    <none>           <none>
ovn-ipsec-j8lcn        1/1     Running   9 (20m ago)     10h   10.0.195.186   ip-10-0-195-186.us-east-2.compute.internal   <none>           <none>
ovn-ipsec-mz4xh        1/1     Running   42 (31m ago)    10h   10.0.216.124   ip-10-0-216-124.us-east-2.compute.internal   <none>           <none>
ovn-ipsec-rzr26        1/1     Running   9 (27m ago)     10h   10.0.159.205   ip-10-0-159-205.us-east-2.compute.internal   <none>           <none>


$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.41       Compliance Operator                0.1.41                Succeeded
elasticsearch-operator.5.2.2-26   OpenShift Elasticsearch Operator   5.2.2-26              Succeeded

$ oc get pods
NAME                                              READY   STATUS    RESTARTS   AGE
compliance-operator-5db46f875-g2zcs               1/1     Running   0          2m15s
ocp4-openshift-compliance-pp-646d77dd55-vrxzx     1/1     Running   0          12m
rhcos4-openshift-compliance-pp-54d6dd49cf-46zm5   1/1     Running   0          2m15s

$ oc create -f -<<EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSettingBinding
> metadata:
>   name: my-ssb-r
> profiles:
>   - name: ocp4-cis
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
>   - name: ocp4-cis-node
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
> settingsRef:
>   name: default
>   kind: ScanSetting
>   apiGroup: compliance.openshift.io/v1alpha1
> EOF
scansettingbinding.compliance.openshift.io/my-ssb-r created


$ oc get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-cis                                 0/1     Completed   0          2m13s
aggregator-pod-ocp4-cis-node-master                     0/1     Completed   0          2m13s
aggregator-pod-ocp4-cis-node-worker                     0/1     Completed   0          2m13s
compliance-operator-5db46f875-g2zcs                     1/1     Running     0          5m52s
ocp4-cis-api-checks-pod                                 0/2     Completed   0          2m56s
ocp4-openshift-compliance-pp-646d77dd55-vrxzx           1/1     Running     0          15m
openscap-pod-1866eafd430eda0bb820d2981b9223c2597cb0e4   0/2     Completed   0          2m54s
openscap-pod-25ce34be408cd359a2bd1510213627f38c5365f0   0/2     Completed   0          2m53s
openscap-pod-3487beed560f0324690e87c03ac39786e0adfc2b   0/2     Completed   0          2m53s
openscap-pod-d3e918518c996371fe9203f791904bb39b73f02f   0/2     Completed   0          2m53s
openscap-pod-e53aa1b9e0eff4a588a79d1f5ececc0c11474ba2   0/2     Completed   0          2m53s
openscap-pod-fa22b18334ef5e4f2749972678144e6b3c1b8301   0/2     Completed   0          2m53s
rhcos4-openshift-compliance-pp-54d6dd49cf-46zm5         1/1     Running     0          5m52s


$ oc get compliancesuite
NAME       PHASE   RESULT
my-ssb-r   DONE    NON-COMPLIANT


$ oc-compliance fetch-raw scansettingbinding my-ssb-r -o resultsdir/
Fetching results for my-ssb-r scans: ocp4-cis, ocp4-cis-node-worker, ocp4-cis-node-master
Fetching raw compliance results for scan 'ocp4-cis'......
The raw compliance results are avaliable in the following directory: resultsdir/ocp4-cis
Fetching raw compliance results for scan 'ocp4-cis-node-worker'......
The raw compliance results are avaliable in the following directory: resultsdir/ocp4-cis-node-worker
Fetching raw compliance results for scan 'ocp4-cis-node-master'...........
The raw compliance results are avaliable in the following directory: resultsdir/ocp4-cis-node-master


$ oc get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-cis                                 0/1     Completed   0          5m10s
aggregator-pod-ocp4-cis-node-master                     0/1     Completed   0          5m10s
aggregator-pod-ocp4-cis-node-worker                     0/1     Completed   0          5m10s
compliance-operator-5db46f875-g2zcs                     1/1     Running     0          8m49s
ocp4-cis-api-checks-pod                                 0/2     Completed   0          5m53s
ocp4-openshift-compliance-pp-646d77dd55-vrxzx           1/1     Running     0          18m
openscap-pod-1866eafd430eda0bb820d2981b9223c2597cb0e4   0/2     Completed   0          5m51s
openscap-pod-25ce34be408cd359a2bd1510213627f38c5365f0   0/2     Completed   0          5m50s
openscap-pod-3487beed560f0324690e87c03ac39786e0adfc2b   0/2     Completed   0          5m50s
openscap-pod-d3e918518c996371fe9203f791904bb39b73f02f   0/2     Completed   0          5m50s
openscap-pod-e53aa1b9e0eff4a588a79d1f5ececc0c11474ba2   0/2     Completed   0          5m50s
openscap-pod-fa22b18334ef5e4f2749972678144e6b3c1b8301   0/2     Completed   0          5m50s
raw-result-extractor-lzdqn                              1/1     Running     0          25s       <<------
rhcos4-openshift-compliance-pp-54d6dd49cf-46zm5         1/1     Running     0          8m49s



$ ls resultsdir/
ocp4-cis  ocp4-cis-node-master  ocp4-cis-node-worker

$ ls resultsdir/ocp4-cis/
ocp4-cis-api-checks-pod.xml.bzip2

$ ls resultsdir/ocp4-cis-node-master/
openscap-pod-25ce34be408cd359a2bd1510213627f38c5365f0.xml.bzip2  openscap-pod-d3e918518c996371fe9203f791904bb39b73f02f.xml.bzip2  openscap-pod-fa22b18334ef5e4f2749972678144e6b3c1b8301.xml.bzip2

$ ls resultsdir/ocp4-cis-node-worker/
openscap-pod-1866eafd430eda0bb820d2981b9223c2597cb0e4.xml.bzip2  openscap-pod-3487beed560f0324690e87c03ac39786e0adfc2b.xml.bzip2  openscap-pod-e53aa1b9e0eff4a588a79d1f5ececc0c11474ba2.xml.bzip2



$ oc get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-cis                                 0/1     Completed   0          5m25s
aggregator-pod-ocp4-cis-node-master                     0/1     Completed   0          5m25s
aggregator-pod-ocp4-cis-node-worker                     0/1     Completed   0          5m25s
compliance-operator-5db46f875-g2zcs                     1/1     Running     0          9m4s
ocp4-cis-api-checks-pod                                 0/2     Completed   0          6m8s
ocp4-openshift-compliance-pp-646d77dd55-vrxzx           1/1     Running     0          18m
openscap-pod-1866eafd430eda0bb820d2981b9223c2597cb0e4   0/2     Completed   0          6m6s
openscap-pod-25ce34be408cd359a2bd1510213627f38c5365f0   0/2     Completed   0          6m5s
openscap-pod-3487beed560f0324690e87c03ac39786e0adfc2b   0/2     Completed   0          6m5s
openscap-pod-d3e918518c996371fe9203f791904bb39b73f02f   0/2     Completed   0          6m5s
openscap-pod-e53aa1b9e0eff4a588a79d1f5ececc0c11474ba2   0/2     Completed   0          6m5s
openscap-pod-fa22b18334ef5e4f2749972678144e6b3c1b8301   0/2     Completed   0          6m5s
rhcos4-openshift-compliance-pp-54d6dd49cf-46zm5         1/1     Running     0          9m4s

Comment 7 errata-xmlrpc 2021-10-18 17:36:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.