Bug 1898819 - Aggregator pod tries to parse ConfigMaps without results
Summary: Aggregator pod tries to parse ConfigMaps without results
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Compliance Operator
Version: 4.7
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
: 4.7.0
Assignee: Juan Antonio Osorio
QA Contact: Prashant Dhamdhere
URL:
Whiteboard:
Depends On:
Blocks: 1899479
TreeView+ depends on / blocked
 
Reported: 2020-11-18 07:40 UTC by Juan Antonio Osorio
Modified: 2021-02-24 19:46 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1899479 (view as bug list)
Environment:
Last Closed: 2021-02-24 19:45:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift compliance-operator pull 473 0 None closed Bug 1898819: Aggregator: Only parse ConfigMaps with results 2021-01-12 20:21:24 UTC
Red Hat Product Errata RHBA-2021:0435 0 None None None 2021-02-24 19:46:07 UTC

Description Juan Antonio Osorio 2020-11-18 07:40:07 UTC
Description of problem:

The aggregator pod tries to parse ConfigMaps which don't contain results, thus adding stack-traces to the logs which make the logs harder to follow.

Version-Release number of selected component (if applicable):

master


How reproducible:

always


Steps to Reproduce:
1. Create a ScanSettingBinding
2. Follow logs
3. View aggregator pod logs

Actual results:

The logs show:

I1118 07:35:01.246412       1 request.go:645] Throttling request took 1.006521051s, request: GET:https://172.30.0.1:443/apis/template.openshift.io/v1?timeout=32s
{"level":"info","ts":1605684903.565532,"logger":"cmd","msg":"Scan has results","ComplianceScan.Name":"test-scan-w-missing-tailoring-cm","results-length":10}
{"level":"info","ts":1605684910.9794436,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-4605af40730567de960963679a9a4eefe2a4acc1"}
{"level":"info","ts":1605684911.1973448,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-4605af40730567de960963679a9a4eefe2a4acc1","results":1}
{"level":"info","ts":1605684911.197434,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-5c7423272dc4c89a0a94b99fc99495fba69b0cc3"}
{"level":"info","ts":1605684911.4358816,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-5c7423272dc4c89a0a94b99fc99495fba69b0cc3","results":1}
{"level":"info","ts":1605684911.4369462,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-7c1bf5e2c9d25368967ea7bbd4dafa22e54983de"}
{"level":"info","ts":1605684911.6557343,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-7c1bf5e2c9d25368967ea7bbd4dafa22e54983de","results":1}
{"level":"info","ts":1605684911.6562061,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-7d5faf29662c131b119ca3b2176a5dea60c76b24"}
{"level":"info","ts":1605684911.8935153,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-7d5faf29662c131b119ca3b2176a5dea60c76b24","results":1}
{"level":"info","ts":1605684911.8944597,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"openscap-pod-db1601e6f10759eb70b985f2948b974dbc124b2e"}
{"level":"info","ts":1605684912.1987154,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"openscap-pod-db1601e6f10759eb70b985f2948b974dbc124b2e","results":1}
{"level":"info","ts":1605684912.1997826,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-ip-10-0-148-7.ec2.internal-pod"}
{"level":"info","ts":1605684912.4199076,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-ip-10-0-148-7.ec2.internal-pod","results":1}
{"level":"info","ts":1605684912.420428,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-container-entrypoint"}
{"level":"error","ts":1605684912.4205093,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-container-entrypoint","error":"no results in configmap test-scan-w-missing-tailoring-cm-openscap-container-entrypoint","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4207253,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-container-entrypoint","results":0}
{"level":"info","ts":1605684912.4208217,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map"}
{"level":"error","ts":1605684912.4208686,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map","error":"no results in configmap test-scan-w-missing-tailoring-cm-openscap-env-map","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4210587,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map","results":0}
{"level":"info","ts":1605684912.4211433,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map-platform"}
{"level":"error","ts":1605684912.4211922,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map-platform","error":"no results in configmap test-scan-w-missing-tailoring-cm-openscap-env-map-platform","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4213772,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"test-scan-w-missing-tailoring-cm-openscap-env-map-platform","results":0}
{"level":"info","ts":1605684912.42146,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"tp-test-scan-w-missing-tailoring-cm"}
{"level":"error","ts":1605684912.4214966,"logger":"cmd","msg":"Cannot parse ConfigMap into remediations","ConfigMap.Name":"tp-test-scan-w-missing-tailoring-cm","error":"no results in configmap tp-test-scan-w-missing-tailoring-cm","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nmain.aggregator\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/aggregator.go:551\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/openshift/compliance-operator/vendor/github.com/spf13/cobra/command.go:887\nmain.main\n\t/go/src/github.com/openshift/compliance-operator/cmd/manager/main.go:34\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1605684912.4216502,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"tp-test-scan-w-missing-tailoring-cm","results":0}
{"level":"info","ts":1605684912.4217007,"logger":"cmd","msg":"Creating result objects"}
{"level":"info","ts":1605684912.4217348,"logger":"cmd","msg":"Will create result objects","objects":1}
{"level":"info","ts":1605684912.4217718,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"test-scan-w-missing-tailoring-cm-no-netrc-files","ComplianceCheckResult.Namespace":"osdk-e2e-4eae2ba8-e4a7-4964-8665-36533b58280d"}
{"level":"info","ts":1605684912.430759,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"test-scan-w-missing-tailoring-cm-no-netrc-files"}
{"level":"info","ts":1605684912.4458978,"logger":"cmd","msg":"Annotating ConfigMaps"}


Expected results:

It shouldn't show the stack-traces of if trying to parse ConfigMaps that are not relevant.


Additional info:

Comment 2 Prashant Dhamdhere 2020-12-08 15:20:15 UTC
This looks good. The aggregator pod does not parse any ConfigMaps which don't contain results.


Verified on:

4.7.0-0.nightly-2020-12-04-013308
Compliance Operator v0.1.22

$ oc get clusterversion

NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-12-04-013308   True        False         11h     Cluster version is 4.7.0-0.nightly-2020-12-04-013308


$ oc get csv

NAME                          DISPLAY               VERSION   REPLACES   PHASE
compliance-operator.v0.1.22   Compliance Operator   0.1.22               Succeeded


$ oc get pods

NAME                                             READY   STATUS    RESTARTS   AGE
compliance-operator-5885db4685-j7p7z             1/1     Running   0          2m19s
ocp4-openshift-compliance-pp-654ffcbc57-rfqd5    1/1     Running   0          95s
rhcos4-openshift-compliance-pp-5ddf57b65-55lx4   1/1     Running   0          95s


$ oc create -f - << EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: my-companys-compliance-requirements
profiles:
  # Node checks
                      
  - name: rhcos4-ncp
    kind: Profile                             
    apiGroup: compliance.openshift.io/v1alpha1
  # Cluster checks
                                     
  - name: ocp4-moderate
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default                             
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
EOF

scansettingbinding.compliance.openshift.io/my-companys-compliance-requirements created


$ oc get pods

NAME                                                              READY   STATUS      RESTARTS   AGE
aggregator-pod-ocp4-moderate                                      0/1     Completed   0          4m59s
aggregator-pod-rhcos4-ncp-master                                  0/1     Completed   0          49s
aggregator-pod-rhcos4-ncp-worker                                  0/1     Completed   0          2m29s
compliance-operator-5885db4685-j7p7z                              1/1     Running     0          12m
ocp4-moderate-api-checks-pod                                      0/2     Completed   0          6m9s
ocp4-openshift-compliance-pp-654ffcbc57-rfqd5                     1/1     Running     0          11m
rhcos4-ncp-master-ip-10-0-48-247.us-east-2.compute.internal-pod   0/2     Completed   0          6m9s
rhcos4-ncp-master-ip-10-0-55-206.us-east-2.compute.internal-pod   0/2     Completed   0          6m10s
rhcos4-ncp-master-ip-10-0-64-146.us-east-2.compute.internal-pod   0/2     Completed   0          6m9s
rhcos4-ncp-worker-ip-10-0-48-112.us-east-2.compute.internal-pod   0/2     Completed   0          6m10s
rhcos4-ncp-worker-ip-10-0-68-119.us-east-2.compute.internal-pod   0/2     Completed   0          6m10s
rhcos4-openshift-compliance-pp-5ddf57b65-55lx4                    1/1     Running     0          11m

$ oc get cm

NAME                                                              DATA   AGE
compliance-operator-lock                                          0      12m
ocp4-moderate-api-checks-pod                                      3      5m40s
ocp4-moderate-openscap-container-entrypoint                       1      6m49s
ocp4-moderate-openscap-env-map                                    4      6m49s
ocp4-moderate-openscap-env-map-platform                           3      6m49s
rhcos4-ncp-master-ip-10-0-48-247.us-east-2.compute.internal-pod   3      111s
rhcos4-ncp-master-ip-10-0-55-206.us-east-2.compute.internal-pod   3      87s
rhcos4-ncp-master-ip-10-0-64-146.us-east-2.compute.internal-pod   3      2m44s
rhcos4-ncp-master-openscap-container-entrypoint                   1      6m51s
rhcos4-ncp-master-openscap-env-map                                4      6m51s
rhcos4-ncp-master-openscap-env-map-platform                       3      6m51s
rhcos4-ncp-worker-ip-10-0-48-112.us-east-2.compute.internal-pod   3      3m3s
rhcos4-ncp-worker-ip-10-0-68-119.us-east-2.compute.internal-pod   3      4m34s
rhcos4-ncp-worker-openscap-container-entrypoint                   1      6m53s
rhcos4-ncp-worker-openscap-env-map                                4      6m53s
rhcos4-ncp-worker-openscap-env-map-platform                       3      6m53s


$ oc logs aggregator-pod-ocp4-moderate

I1208 14:53:51.861957       1 request.go:645] Throttling request took 1.010803346s, request: GET:https://172.30.0.1:443/apis/samples.operator.openshift.io/v1?timeout=32s
{"level":"info","ts":1607439233.3312569,"logger":"cmd","msg":"Scan has results","ComplianceScan.Name":"ocp4-moderate","results-length":1}
{"level":"info","ts":1607439234.2119365,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"ocp4-moderate-api-checks-pod"}
{"level":"info","ts":1607439234.254056,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"ocp4-moderate-api-checks-pod","results":4}
{"level":"info","ts":1607439234.25542,"logger":"cmd","msg":"Creating result objects"}
{"level":"info","ts":1607439234.2554567,"logger":"cmd","msg":"Will create result objects","objects":4}
{"level":"info","ts":1607439234.2555091,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-ocp-idp-no-htpasswd","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607439234.2617884,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-ocp-idp-no-htpasswd"}
{"level":"info","ts":1607439234.2942863,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-ocp-allowed-registries","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607439234.2984052,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-ocp-allowed-registries"}
{"level":"info","ts":1607439234.3086221,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-ocp-allowed-registries-for-import","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607439234.3125076,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-ocp-allowed-registries-for-import"}
{"level":"info","ts":1607439234.322319,"logger":"cmd","msg":"Getting ComplianceCheckResult","ComplianceCheckResult.Name":"ocp4-moderate-audit-log-forwarding-enabled","ComplianceCheckResult.Namespace":"openshift-compliance"}
{"level":"info","ts":1607439234.3260422,"logger":"cmd","msg":"Creating object","kind":"&TypeMeta{Kind:,APIVersion:,}","name":"ocp4-moderate-audit-log-forwarding-enabled"}
{"level":"info","ts":1607439234.3345127,"logger":"cmd","msg":"Annotating ConfigMaps"}


$ oc logs aggregator-pod-rhcos4-ncp-worker|grep "ConfigMap"


{"level":"info","ts":1607439388.525723,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-48-112.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607439388.8904777,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-48-112.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607439388.8923085,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-68-119.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607439389.2463346,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-worker-ip-10-0-68-119.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607439410.6369548,"logger":"cmd","msg":"Annotating ConfigMaps"}

$ oc logs aggregator-pod-rhcos4-ncp-master|grep "ConfigMap"


{"level":"info","ts":1607439489.8447337,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-48-247.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607439490.2151859,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-48-247.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607439490.2167296,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-55-206.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607439490.5732436,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-55-206.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607439490.6512868,"logger":"cmd","msg":"processing ConfigMap","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-64-146.us-east-2.compute.internal-pod"}
{"level":"info","ts":1607439491.0059912,"logger":"cmd","msg":"ConfigMap contained parsed results","ConfigMap.Name":"rhcos4-ncp-master-ip-10-0-64-146.us-east-2.compute.internal-pod","results":228}
{"level":"info","ts":1607439512.4048753,"logger":"cmd","msg":"Annotating ConfigMaps"}

Comment 7 errata-xmlrpc 2021-02-24 19:45:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7 compliance-operator image update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0435


Note You need to log in before you can comment on or make changes to this bug.