Description of problem: On a Cluster with >500 namespaces and >500 networkpolicies (cluster-wide) the compliancecheck "ocp4-cis-configure-network-policies-namespaces" fails reproducibly, although the rule criteria is met: the list of namespaces-with-netpols and the list of all-namespaces-with-some-exceptions are identical $ oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' | sha1sum 0903d57b28b7b66018ba77b8ccb1e2f964b55d5e - $ oc get --all-namespaces networkpolicies -o json | jq '[.items[] | select((.metadata.namespace | startswith("openshift") | not) and (.metadata.namespace | startswith("kube-") | not) and .metadata.namespace != "default") | .metadata.namespace] | unique' | sha1sum 0903d57b28b7b66018ba77b8ccb1e2f964b55d5e - I suspect that this might be due to the "limit=500" parameter in the corresponding check [1] (but admit I'm not sure if it's a real limit or rather a pagination feature). As there currently are 617 namespaces and it's a production cluster I unfortunately can't just delete 117 namespaces to verify this fixes the issue ;) On an identically configured cluster with less workload (number of namespaces and netpols are each below 500) the check evaluates correctly. Version-Release number of selected component (if applicable): compliance-operator.v0.1.47 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: compliancecheckresult ocp4-cis-configure-network-policies-namespaces with status "FAIL" Expected results: compliancecheckresult ocp4-cis-configure-network-policies-namespaces with status "PASS" Additional info: [1] https://github.com/ComplianceAsCode/content/blob/master/applications/openshift/networking/configure_network_policies_namespaces/rule.yml#L28-L29
More testing needed. I think you might be right about the pagination. Either way (as discussed with Lance and Vincent on Slack) we just can't get away from fetching all the objects if we need to have a qualified answer.
I just quickly reproduced this on another (non-prod) cluster as follows: $ oc get compliancecheckresults.compliance.openshift.io |grep -i fail corp-ocp4-cis-api-server-no-adm-ctrl-plugins-disabled FAIL medium $ oc get ns --no-headers | wc -l 135 $ for i in {001..400}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy-allow-all-ingress.yaml ; done [...] namespace/test-compliancecheck-400 created networkpolicy.networking.k8s.io/allow-all-ingress created $ oc -n openshift-compliance annotate compliancescans/corp-ocp4-cis compliance.openshift.io/rescan= compliancescan.compliance.openshift.io/corp-ocp4-cis annotated [... a couple minutes later ...] $ oc get compliancecheckresults.compliance.openshift.io |grep -i fail corp-ocp4-cis-api-server-no-adm-ctrl-plugins-disabled FAIL medium corp-ocp4-cis-configure-network-policies-namespaces FAIL high $ for i in {001..400}; do oc delete ns test-compliancecheck-${i} --wait=false ; done [...] namespace "test-compliancecheck-400" deleted $ oc delete compliancecheckresults.compliance.openshift.io corp-ocp4-cis-configure-network-policies-namespaces compliancecheckresult.compliance.openshift.io "corp-ocp4-cis-configure-network-policies-namespaces" deleted $ oc -n openshift-compliance annotate compliancescans/corp-ocp4-cis compliance.openshift.io/rescan= compliancescan.compliance.openshift.io/corp-ocp4-cis annotated [... a couple minutes later ...] $ oc get compliancecheckresults.compliance.openshift.io -w |grep -i fail corp-ocp4-cis-api-server-no-adm-ctrl-plugins-disabled FAIL medium $ (ignore the cis-api-server-no-adm-ctrl-plugins-disabled result, it's unrelated) so to me this looks strongly related to the number of namespaces present. Would it be a possibility to raise the pagination limit to i.e. 1000 for the time being, until a proper fix is found?
I can try to take out the 500 limit, see if that fixes the issue
Fix: https://github.com/ComplianceAsCode/content/commit/e2e7e9fb59b54d2f4d3bdb9ef8c9f79ac8ffa870
[Bug Verification] Looks good. Now, The rule ocp4-cis-configure-network-policies-namespaces is getting passed on cluster with >1000 namespaces. Verified on: 4.10.0-0.nightly-2022-03-29-163038 + compliance-operator.v0.1.49 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-03-29-163038 True False 6h53m Cluster version is 4.10.0-0.nightly-2022-03-29-163038 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE compliance-operator.v0.1.49 Compliance Operator 0.1.49 Succeeded elasticsearch-operator.5.4.0-127 OpenShift Elasticsearch Operator 5.4.0-127 Succeeded $ oc get pods NAME READY STATUS RESTARTS AGE compliance-operator-9bf58698f-g7w9d 1/1 Running 1 (3h10m ago) 3h11m ocp4-openshift-compliance-pp-59cd7665d6-w5czh 1/1 Running 0 3h10m rhcos4-openshift-compliance-pp-5c85d4d5c8-8xtk6 1/1 Running 0 3h10m $ oc get rule |grep policies ocp4-configure-network-policies 3h10m ocp4-configure-network-policies-namespaces 3h10m $ oc get rule ocp4-configure-network-policies-namespaces -o=jsonpath={.instructions} Verify that the every non-control plane namespace has an appropriate NetworkPolicy. To get all the non-control plane namespaces, you can do the following command oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' To get all the non-control plane namespaces with a NetworkPolicy, you can do the following command oc get --all-namespaces networkpolicies -o json | jq '[.items[] | select((.metadata.namespace | startswith("openshift") | not) and (.metadata.namespace | startswith("kube-") | not) and .metadata.namespace != "default") | .metadata.namespace] | unique' Make sure that the namespaces displayed in the commands of the commands match. $ oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' [] $ oc get --all-namespaces networkpolicies -o json | jq '[.items[] | select((.metadata.namespace | startswith("openshift") | not) and (.metadata.namespace | startswith("kube-") | not) and .metadata.namespace != "default") | .metadata.namespace] | unique' [] $ oc create -f - << EOF apiVersion: compliance.openshift.io/v1alpha1 kind: ScanSettingBinding metadata: name: my-ssb-moderate profiles: - name: ocp4-moderate kind: Profile apiGroup: compliance.openshift.io/v1alpha1 settingsRef: name: default kind: ScanSetting apiGroup: compliance.openshift.io/v1alpha1 EOF scansettingbinding.compliance.openshift.io/my-ssb-moderate created $ oc get suite -w NAME PHASE RESULT my-ssb-moderate LAUNCHING NOT-AVAILABLE my-ssb-moderate RUNNING NOT-AVAILABLE my-ssb-moderate AGGREGATING NOT-AVAILABLE my-ssb-moderate DONE NON-COMPLIANT my-ssb-moderate DONE NON-COMPLIANT $ oc get scan NAME PHASE RESULT ocp4-moderate DONE NON-COMPLIANT $ oc get checkresult ocp4-moderate-configure-network-policies-namespaces NAME STATUS SEVERITY ocp4-moderate-configure-network-policies-namespaces FAIL high $ oc get ns --no-headers | wc -l 68 $ for i in {001..400}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy.yaml ; done namespace/test-compliancecheck-001 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-002 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-003 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-004 created networkpolicy.networking.k8s.io/allow-same-namespace created .... .... namespace/test-compliancecheck-398 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-399 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-400 created networkpolicy.networking.k8s.io/allow-same-namespace created $ oc get ns --no-headers | wc -l 468 $ oc get suite NAME PHASE RESULT my-ssb-moderate DONE NON-COMPLIANT $ oc-compliance rerun-now compliancesuite/my-ssb-moderate -nopenshift-compliance Rerunning scans from 'my-ssb-moderate': ocp4-moderate Re-running scan 'openshift-compliance/ocp4-moderate' $ oc get suite -w NAME PHASE RESULT my-ssb-moderate RUNNING NOT-AVAILABLE my-ssb-moderate AGGREGATING NOT-AVAILABLE my-ssb-moderate DONE NON-COMPLIANT my-ssb-moderate DONE NON-COMPLIANT $ oc get checkresult ocp4-moderate-configure-network-policies-namespaces NAME STATUS SEVERITY ocp4-moderate-configure-network-policies-namespaces PASS high $ oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |head [ "test-compliancecheck-001", "test-compliancecheck-002", "test-compliancecheck-003", "test-compliancecheck-004", "test-compliancecheck-005", "test-compliancecheck-006", "test-compliancecheck-007", "test-compliancecheck-008", "test-compliancecheck-009", $ oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |tail "test-compliancecheck-392", "test-compliancecheck-393", "test-compliancecheck-394", "test-compliancecheck-395", "test-compliancecheck-396", "test-compliancecheck-397", "test-compliancecheck-398", "test-compliancecheck-399", "test-compliancecheck-400" ] $ for i in {401..600}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy.yaml ; done namespace/test-compliancecheck-401 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-402 created networkpolicy.networking.k8s.io/allow-same-namespace created .... namespace/test-compliancecheck-598 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-599 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-600 created networkpolicy.networking.k8s.io/allow-same-namespace created $ oc get ns --no-headers | wc -l 668 $ oc-compliance rerun-now compliancesuite/my-ssb-moderate -nopenshift-compliance Rerunning scans from 'my-ssb-moderate': ocp4-moderate Re-running scan 'openshift-compliance/ocp4-moderate' $ oc get suite -w NAME PHASE RESULT my-ssb-moderate LAUNCHING NOT-AVAILABLE my-ssb-moderate RUNNING NOT-AVAILABLE my-ssb-moderate AGGREGATING NOT-AVAILABLE my-ssb-moderate DONE NON-COMPLIANT my-ssb-moderate DONE NON-COMPLIANT $ oc get checkresult ocp4-moderate-configure-network-policies-namespaces NAME STATUS SEVERITY ocp4-moderate-configure-network-policies-namespaces PASS high $ oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |tail "test-compliancecheck-592", "test-compliancecheck-593", "test-compliancecheck-594", "test-compliancecheck-595", "test-compliancecheck-596", "test-compliancecheck-597", "test-compliancecheck-598", "test-compliancecheck-599", "test-compliancecheck-600" ] $ for i in {601..999}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy.yaml ; done namespace/test-compliancecheck-601 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-602 created networkpolicy.networking.k8s.io/allow-same-namespace created .... .... namespace/test-compliancecheck-998 created networkpolicy.networking.k8s.io/allow-same-namespace created namespace/test-compliancecheck-999 created networkpolicy.networking.k8s.io/allow-same-namespace created $ oc get ns --no-headers | wc -l 1067 $ oc-compliance rerun-now compliancesuite/my-ssb-moderate -nopenshift-compliance Rerunning scans from 'my-ssb-moderate': ocp4-moderate Re-running scan 'openshift-compliance/ocp4-moderate' $ oc get suite -w NAME PHASE RESULT my-ssb-moderate RUNNING NOT-AVAILABLE my-ssb-moderate AGGREGATING NOT-AVAILABLE my-ssb-moderate DONE NON-COMPLIANT my-ssb-moderate DONE NON-COMPLIANT $ oc get scan NAME PHASE RESULT ocp4-moderate DONE NON-COMPLIANT $ oc get checkresult ocp4-moderate-configure-network-policies-namespaces NAME STATUS SEVERITY ocp4-moderate-configure-network-policies-namespaces PASS high $ oc get namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |tail "test-compliancecheck-991", "test-compliancecheck-992", "test-compliancecheck-993", "test-compliancecheck-994", "test-compliancecheck-995", "test-compliancecheck-996", "test-compliancecheck-997", "test-compliancecheck-998", "test-compliancecheck-999" ]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1148