Bug 1899479

Summary: Aggregator pod tries to parse ConfigMaps without results
Product: OpenShift Container Platform Reporter: Juan Antonio Osorio <josorior>
Component: Compliance OperatorAssignee: Jakub Hrozek <jhrozek>
Status: CLOSED ERRATA QA Contact: Prashant Dhamdhere <pdhamdhe>
Severity: medium Docs Contact:
Priority: low    
Version: 4.6.zCC: josorior, mrogers, nkinder, pdhamdhe, xiyuan
Target Milestone: ---Keywords: Bugfix, UpcomingSprint
Target Release: 4.6.z   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1898819 Environment:
Last Closed: 2021-01-19 13:53:52 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1898819    
Bug Blocks:    

Description Juan Antonio Osorio 2020-11-19 11:30:56 UTC
+++ This bug was initially created as a clone of Bug #1898819 +++

Description of problem:

The aggregator pod tries to parse ConfigMaps which don't contain results, thus adding stack-traces to the logs which make the logs harder to follow.

Version-Release number of selected component (if applicable):

master


How reproducible:

always


Steps to Reproduce:
1. Create a ScanSettingBinding
2. Follow logs
3. View aggregator pod logs

Actual results:

The logs show:

I1118 07:35:01.246412       1 request.go:645] Throttling request took 1.006521051s, request: GET:https://172.30.0.1:443/apis/template.openshift.io/v1?timeout=32s
{"level":"info","ts":1605684903.565532,"logger":"cmd","msg":"Scan has results","ComplianceScan.Name":"test-scan-w-missing-tailoring-cm","results-length":10}
{"level":"info","ts":1605684910.9794436,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-4605af40730567de960963679a9a4eefe2a4acc1"}
{"level":"info","ts":1605684911.1973448,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-4605af40730567de960963679a9a4eefe2a4acc1","results":1}
{"level":"info","ts":1605684911.197434,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-5c7423272dc4c89a0a94b99fc99495fba69b0cc3"}
{"level":"info","ts":1605684911.4358816,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-5c7423272dc4c89a0a94b99fc99495fba69b0cc3","results":1}
{"level":"info","ts":1605684911.4369462,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-7c1bf5e2c9d25368967ea7bbd4dafa22e54983de"}
{"level":"info","ts":1605684911.6557343,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-7c1bf5e2c9d25368967ea7bbd4dafa22e54983de","results":1}
{"level":"info","ts":1605684911.6562061,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-7d5faf29662c131b119ca3b2176a5dea60c76b24"}
{"level":"info","ts":1605684911.8935153,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-7d5faf29662c131b119ca3b2176a5dea60c76b24","results":1}
{"level":"info","ts":1605684911.8944597,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-db1601e6f10759eb70b985f2948b974dbc124b2e"}
{"level":"info","ts":1605684912.1987154,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-db1601e6f10759eb70b985f2948b974dbc124b2e","results":1}
{"level":"info","ts":1605684912.1997826,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-ip-10-0-148-7.ec2.internal-pod"}
{"level":"info","ts":1605684912.4199076,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-ip-10-0-148-7.ec2.internal-pod","results":1}
{"level":"info","ts":1605684912.420428,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-container-entrypoint"}
{"level":"error","ts":1605684912.4205093,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-container-entrypoint","error":"no results in configmap test-scan-w-missing-tailoring-cm-openscap-container-entrypoint","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4207253,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-container-entrypoint","results":0}
{"level":"info","ts":1605684912.4208217,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map"}
{"level":"error","ts":1605684912.4208686,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map","error":"no results in configmap test-scan-w-missing-tailoring-cm-openscap-env-map","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4210587,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map","results":0}
{"level":"info","ts":1605684912.4211433,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map-platform"}
{"level":"error","ts":1605684912.4211922,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map-platform","error":"no results in configmap test-scan-w-missing-tailoring-cm-openscap-env-map-platform","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4213772,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map-platform","results":0}
{"level":"info","ts":1605684912.42146,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"tp-test-scan-w-missing-tailoring-cm"}
{"level":"error","ts":1605684912.4214966,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"tp-test-scan-w-missing-tailoring-cm","error":"no results in configmap tp-test-scan-w-missing-tailoring-cm","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4216502,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"tp-test-scan-w-missing-tailoring-cm","results":0}
{"level":"info","ts":1605684912.4217007,"logger":"cmd","msg":"Creating result objects"}
{"level":"info","ts":1605684912.4217348,"logger":"cmd","msg":"Will create result objects","objects":1}
{"level":"info","ts":1605684912.4217718,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"test-scan-w-missing-tailoring-cm-no-netrc-files","ComplianceCheckResult.Namespace":"osdk-e2e-4eae2ba8-e4a7-4964-8665-36533b58280d"}
{"level":"info","ts":1605684912.430759,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"test-scan-w-missing-tailoring-cm-no-netrc-files"}
{"level":"info","ts":1605684912.4458978,"logger":"cmd","msg":"Annotating ConfigMaps"}


Expected results:

It shouldn't show the stack-traces of if trying to parse ConfigMaps that are not relevant.


Additional info:

Comment 3 Prashant Dhamdhere 2020-12-08 16:24:31 UTC
This looks good. Now, the aggregator pod does not parse any ConfigMaps which don't contain results.


Verified on:

4.6.0-0.nightly-2020-12-06-095114
Compliance Operator v0.1.22

$ oc get clusterversion

NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-12-06-095114   True        False         12h     Cluster version is 4.6.0-0.nightly-2020-12-06-095114


$ oc get csv

NAME                          DISPLAY               VERSION   REPLACES   PHASE
compliance-operator.v0.1.22   Compliance Operator   0.1.22               Succeeded


$ oc get pods 

NAME                                             READY   STATUS    RESTARTS   AGE
compliance-operator-5885db4685-qrjgd             1/1     Running   0          35m
ocp4-openshift-compliance-pp-654ffcbc57-qzhl6    1/1     Running   0          34m
rhcos4-openshift-compliance-pp-5ddf57b65-c89c7   1/1     Running   0          34m


$ oc create -f - << EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSettingBinding
> metadata:
>   name: my-companys-compliance-requirements
> profiles:
>   # Node checks                            
>   - name: rhcos4-ncp
>     kind: Profile                             
>     apiGroup: compliance.openshift.io/v1alpha1
>   # Cluster checks        
>   - name: ocp4-moderate
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
> settingsRef:
>   name: default                             
>   kind: ScanSetting
>   apiGroup: compliance.openshift.io/v1alpha1
> EOF

scansettingbinding.compliance.openshift.io/my-companys-compliance-requirements created


$ oc get pods

NAME                                                              READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-moderate                                      0/1     Completed   0          5m35s
aggregator-pod-rhcos4-ncp-master                                  0/1     Completed   0          85s
aggregator-pod-rhcos4-ncp-worker                                  0/1     Completed   0          2m46s
compliance-operator-5885db4685-qrjgd                              1/1     Running     0          42m
ocp4-moderate-api-checks-pod                                      0/2     Completed   0          6m15s
ocp4-openshift-compliance-pp-654ffcbc57-qzhl6                     1/1     Running     0          41m
rhcos4-ncp-master-ip-10-0-59-193.us-east-2.compute.internal-pod   0/2     Completed   0          6m16s
rhcos4-ncp-master-ip-10-0-61-71.us-east-2.compute.internal-pod    0/2     Completed   0          6m16s
rhcos4-ncp-master-ip-10-0-71-191.us-east-2.compute.internal-pod   0/2     Completed   0          6m16s
rhcos4-ncp-worker-ip-10-0-61-73.us-east-2.compute.internal-pod    0/2     Completed   0          6m18s
rhcos4-ncp-worker-ip-10-0-62-204.us-east-2.compute.internal-pod   0/2     Completed   0          6m18s
rhcos4-ncp-worker-ip-10-0-69-99.us-east-2.compute.internal-pod    0/2     Completed   0          6m18s
rhcos4-openshift-compliance-pp-5ddf57b65-c89c7                    1/1     Running     0          41m


$ oc get compliancesuite

NAME                                  PHASE   RESULT
my-companys-compliance-requirements   DONE    NON-COMPLIANT


$ oc get cm

NAME                                                              DATA   AGE
compliance-operator-lock                                          0      42m
ocp4-moderate-api-checks-pod                                      3      6m24s
ocp4-moderate-openscap-container-entrypoint                       1      6m56s
ocp4-moderate-openscap-env-map                                    4      6m56s
ocp4-moderate-openscap-env-map-platform                           3      6m56s
rhcos4-ncp-master-ip-10-0-59-193.us-east-2.compute.internal-pod   3      2m4s
rhcos4-ncp-master-ip-10-0-61-71.us-east-2.compute.internal-pod    3      3m23s
rhcos4-ncp-master-ip-10-0-71-191.us-east-2.compute.internal-pod   3      2m16s
rhcos4-ncp-master-openscap-container-entrypoint                   1      6m58s
rhcos4-ncp-master-openscap-env-map                                4      6m58s
rhcos4-ncp-master-openscap-env-map-platform                       3      6m58s
rhcos4-ncp-worker-ip-10-0-61-73.us-east-2.compute.internal-pod    3      4m4s
rhcos4-ncp-worker-ip-10-0-62-204.us-east-2.compute.internal-pod   3      3m24s
rhcos4-ncp-worker-ip-10-0-69-99.us-east-2.compute.internal-pod    3      3m44s
rhcos4-ncp-worker-openscap-container-entrypoint                   1      7m1s
rhcos4-ncp-worker-openscap-env-map                                4      7m1s
rhcos4-ncp-worker-openscap-env-map-platform                       3      7m1s


$ oc logs aggregator-pod-ocp4-moderate

I1208 16:10:01.807521       1 request.go:645] Throttling request took 1.004881696s, request: GET:https://172.30.0.1:443/apis/template.openshift.io/v1?timeout=32s
{"level":"info","ts":1607443803.3811357,"logger":"cmd","msg":"Scan has results","ComplianceScan.Name":"ocp4-moderate","results-length":1}
{"level":"info","ts":1607443804.352349,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"ocp4-moderate-api-checks-pod"}
{"level":"info","ts":1607443804.399674,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"ocp4-moderate-api-checks-pod","results":4}
{"level":"info","ts":1607443804.3997748,"logger":"cmd","msg":"Creating result objects"}
{"level":"info","ts":1607443804.3997974,"logger":"cmd","msg":"Will create result objects","objects":4}
{"level":"info","ts":1607443804.3998368,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-ocp-allowed-registries","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607443804.436267,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-ocp-allowed-registries"}
{"level":"info","ts":1607443804.476187,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-ocp-allowed-registries-for-import","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607443804.4808517,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-ocp-allowed-registries-for-import"}
{"level":"info","ts":1607443804.4890094,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-audit-log-forwarding-enabled","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607443804.493227,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-audit-log-forwarding-enabled"}
{"level":"info","ts":1607443804.5019896,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-ocp-idp-no-htpasswd","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607443804.506309,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-ocp-idp-no-htpasswd"}
{"level":"info","ts":1607443804.5137842,"logger":"cmd","msg":"Annotating ConfigMaps"}


$ oc logs aggregator-pod-rhcos4-ncp-master |grep "ConfigMap"


{"level":"info","ts":1607444056.9028938,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-59-193.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607444057.367897,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-59-193.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607444057.3704486,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-61-71.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607444057.7479532,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-61-71.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607444057.8333783,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-71-191.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607444058.2135365,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-71-191.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607444079.6047883,"logger":"cmd","msg":"Annotating ConfigMaps"}


$ oc logs aggregator-pod-rhcos4-ncp-worker |grep "ConfigMap"


{"level":"info","ts":1607443975.6787863,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-61-73.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607443976.078594,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-61-73.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607443976.0817118,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-62-204.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607443976.45594,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-62-204.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607443976.5348775,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-69-99.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607443976.921332,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-69-99.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607443998.3091395,"logger":"cmd","msg":"Annotating ConfigMaps"}

Comment 7 errata-xmlrpc 2021-01-19 13:53:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6 compliance-operator security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0190