Bug 2038909

Summary:	ocp4-cis-configure-network-policies-namespaces compliancecheck fails on cluster with >500 namespaces
Product:	OpenShift Container Platform	Reporter:	Felix Krohn <fkrohn>
Component:	Compliance Operator	Assignee:	Vincent Shen <wenshen>
Status:	CLOSED ERRATA	QA Contact:	Prashant Dhamdhere <pdhamdhe>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.8	CC:	jhrozek, lbragsta, mrogers, wenshen, xiyuan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: ocp-cis-configure-network-policies-namespace used pagniation when requesting namespaces Consequence: This caused the rule to fail because the list of namespaces was truncated in deployments with more than 500 namespaces. Fix: Use the updated content, which requests the entire namespace list and doesn't use pagination Result: The rule for checking configured network policies for namespaces will work for deployments with more than 500 namespaces.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-04-18 07:54:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Felix Krohn 2022-01-10 14:22:26 UTC

Description of problem:
On a Cluster with >500 namespaces and >500 networkpolicies (cluster-wide) the compliancecheck "ocp4-cis-configure-network-policies-namespaces" fails reproducibly, although the rule criteria is met: the list of namespaces-with-netpols and the list of all-namespaces-with-some-exceptions are identical

  $ oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | 
startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' | sha1sum                                                                                                                           
  0903d57b28b7b66018ba77b8ccb1e2f964b55d5e  -                                                                                                                                                                                                                                                                                                                                
  $ oc get --all-namespaces networkpolicies -o json | jq '[.items[] | select((.metadata.namespace | startswith("openshift") | not) and (.metadata.namespace | startswith("kube-") | not) and .metadata.namespace != "default") | .metadata.namespace] | unique' | sha1sum                                                                          
  0903d57b28b7b66018ba77b8ccb1e2f964b55d5e  -

I suspect that this might be due to the "limit=500" parameter in the corresponding check [1] (but admit I'm not sure if it's a real limit or rather a pagination feature). As there currently are 617 namespaces and it's a production cluster I unfortunately can't just delete 117 namespaces to verify this fixes the issue ;)

On an identically configured cluster with less workload (number of namespaces and netpols are each below 500) the check evaluates correctly.

Version-Release number of selected component (if applicable):
compliance-operator.v0.1.47


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
compliancecheckresult ocp4-cis-configure-network-policies-namespaces with status "FAIL"

Expected results:
compliancecheckresult ocp4-cis-configure-network-policies-namespaces with status "PASS"

Additional info:
[1] https://github.com/ComplianceAsCode/content/blob/master/applications/openshift/networking/configure_network_policies_namespaces/rule.yml#L28-L29

Comment 1 Jakub Hrozek 2022-01-27 21:02:57 UTC

More testing needed. I think you might be right about the pagination. Either way (as discussed with Lance and Vincent on Slack) we just can't get away from fetching all the objects if we need to have a qualified answer.

Comment 2 Felix Krohn 2022-02-02 15:30:12 UTC

I just quickly reproduced this on another (non-prod) cluster as follows:


$ oc get compliancecheckresults.compliance.openshift.io |grep -i fail
corp-ocp4-cis-api-server-no-adm-ctrl-plugins-disabled                   FAIL     medium
$ oc get ns --no-headers | wc -l
135

$ for i in {001..400}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy-allow-all-ingress.yaml ; done
[...]
namespace/test-compliancecheck-400 created
networkpolicy.networking.k8s.io/allow-all-ingress created

$ oc -n openshift-compliance  annotate compliancescans/corp-ocp4-cis compliance.openshift.io/rescan=
compliancescan.compliance.openshift.io/corp-ocp4-cis annotated

[... a couple minutes later ...]

$ oc get compliancecheckresults.compliance.openshift.io |grep -i fail
corp-ocp4-cis-api-server-no-adm-ctrl-plugins-disabled                   FAIL     medium
corp-ocp4-cis-configure-network-policies-namespaces                     FAIL     high

$ for i in {001..400}; do oc delete ns test-compliancecheck-${i} --wait=false ; done
[...]
namespace "test-compliancecheck-400" deleted
$ oc delete compliancecheckresults.compliance.openshift.io corp-ocp4-cis-configure-network-policies-namespaces
compliancecheckresult.compliance.openshift.io "corp-ocp4-cis-configure-network-policies-namespaces" deleted
$ oc -n openshift-compliance  annotate compliancescans/corp-ocp4-cis compliance.openshift.io/rescan=
compliancescan.compliance.openshift.io/corp-ocp4-cis annotated
[... a couple minutes later ...]
$ oc get compliancecheckresults.compliance.openshift.io -w |grep -i fail
corp-ocp4-cis-api-server-no-adm-ctrl-plugins-disabled                   FAIL     medium
$ 


(ignore the cis-api-server-no-adm-ctrl-plugins-disabled result, it's unrelated)
so to me this looks strongly related to the number of namespaces present. Would it be a possibility to raise the pagination limit to i.e. 1000 for the time being, until a proper fix is found?

Comment 3 Vincent Shen 2022-02-05 07:42:19 UTC

I can try to take out the 500 limit, see if that fixes the issue

Comment 4 Jakub Hrozek 2022-02-11 12:32:34 UTC

Fix: https://github.com/ComplianceAsCode/content/commit/e2e7e9fb59b54d2f4d3bdb9ef8c9f79ac8ffa870

Comment 7 Prashant Dhamdhere 2022-03-31 09:39:28 UTC

[Bug Verification]


Looks good. Now, The rule ocp4-cis-configure-network-policies-namespaces 
is getting passed on cluster with >1000 namespaces.


Verified on:
4.10.0-0.nightly-2022-03-29-163038 + compliance-operator.v0.1.49


$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-03-29-163038   True        False         6h53m   Cluster version is 4.10.0-0.nightly-2022-03-29-163038


$ oc get csv
NAME                               DISPLAY                            VERSION     REPLACES   PHASE
compliance-operator.v0.1.49        Compliance Operator                0.1.49                 Succeeded
elasticsearch-operator.5.4.0-127   OpenShift Elasticsearch Operator   5.4.0-127              Succeeded


$ oc get pods
NAME                                              READY   STATUS    RESTARTS        AGE
compliance-operator-9bf58698f-g7w9d               1/1     Running   1 (3h10m ago)   3h11m
ocp4-openshift-compliance-pp-59cd7665d6-w5czh     1/1     Running   0               3h10m
rhcos4-openshift-compliance-pp-5c85d4d5c8-8xtk6   1/1     Running   0               3h10m


$ oc get rule |grep policies
ocp4-configure-network-policies                                                     3h10m
ocp4-configure-network-policies-namespaces                                          3h10m


$ oc get rule ocp4-configure-network-policies-namespaces -o=jsonpath={.instructions}
Verify that the every non-control plane namespace has an appropriate
NetworkPolicy.

To get all the non-control plane namespaces, you can do the
following command oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]'

To get all the non-control plane namespaces with a NetworkPolicy, you can do the
following command oc get --all-namespaces networkpolicies -o json | jq '[.items[] | select((.metadata.namespace | startswith("openshift") | not) and (.metadata.namespace | startswith("kube-") | not) and .metadata.namespace != "default") | .metadata.namespace] | unique'

Make sure that the namespaces displayed in the commands of the commands match.


$ oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]'
[]


$ oc get --all-namespaces networkpolicies -o json | jq '[.items[] | select((.metadata.namespace | startswith("openshift") | not) and (.metadata.namespace | startswith("kube-") | not) and .metadata.namespace != "default") | .metadata.namespace] | unique'
[]


$ oc create -f - << EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: my-ssb-moderate
profiles:
  - name: ocp4-moderate
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default 
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
EOF
scansettingbinding.compliance.openshift.io/my-ssb-moderate created


$ oc get suite -w
NAME              PHASE       RESULT
my-ssb-moderate   LAUNCHING   NOT-AVAILABLE
my-ssb-moderate   RUNNING     NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT

$ oc get scan
NAME            PHASE   RESULT
ocp4-moderate   DONE    NON-COMPLIANT


$ oc get checkresult ocp4-moderate-configure-network-policies-namespaces
NAME                                                  STATUS   SEVERITY
ocp4-moderate-configure-network-policies-namespaces   FAIL     high

$ oc get ns --no-headers | wc -l
68
 

$ for i in {001..400}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy.yaml ; done
namespace/test-compliancecheck-001 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-002 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-003 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-004 created
networkpolicy.networking.k8s.io/allow-same-namespace created
....
....
namespace/test-compliancecheck-398 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-399 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-400 created
networkpolicy.networking.k8s.io/allow-same-namespace created

$ oc get ns --no-headers | wc -l
468
 

$ oc get suite
NAME              PHASE   RESULT
my-ssb-moderate   DONE    NON-COMPLIANT


$ oc-compliance rerun-now compliancesuite/my-ssb-moderate -nopenshift-compliance                                                                     
Rerunning scans from 'my-ssb-moderate': ocp4-moderate
Re-running scan 'openshift-compliance/ocp4-moderate'

$ oc get suite -w
NAME              PHASE     RESULT
my-ssb-moderate   RUNNING   NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT

$ oc get checkresult ocp4-moderate-configure-network-policies-namespaces
NAME                                                  STATUS   SEVERITY
ocp4-moderate-configure-network-policies-namespaces   PASS     high

$ oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |head
[
  "test-compliancecheck-001",
  "test-compliancecheck-002",
  "test-compliancecheck-003",
  "test-compliancecheck-004",
  "test-compliancecheck-005",
  "test-compliancecheck-006",
  "test-compliancecheck-007",
  "test-compliancecheck-008",
  "test-compliancecheck-009",

$ oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |tail
  "test-compliancecheck-392",
  "test-compliancecheck-393",
  "test-compliancecheck-394",
  "test-compliancecheck-395",
  "test-compliancecheck-396",
  "test-compliancecheck-397",
  "test-compliancecheck-398",
  "test-compliancecheck-399",
  "test-compliancecheck-400"
]


$ for i in {401..600}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy.yaml ; done
namespace/test-compliancecheck-401 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-402 created
networkpolicy.networking.k8s.io/allow-same-namespace created
....
namespace/test-compliancecheck-598 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-599 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-600 created
networkpolicy.networking.k8s.io/allow-same-namespace created

$ oc get ns --no-headers | wc -l
668

$ oc-compliance rerun-now compliancesuite/my-ssb-moderate -nopenshift-compliance
Rerunning scans from 'my-ssb-moderate': ocp4-moderate
Re-running scan 'openshift-compliance/ocp4-moderate'

$ oc get suite -w
NAME              PHASE       RESULT
my-ssb-moderate   LAUNCHING   NOT-AVAILABLE
my-ssb-moderate   RUNNING     NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT

$ oc get checkresult ocp4-moderate-configure-network-policies-namespaces
NAME                                                  STATUS   SEVERITY
ocp4-moderate-configure-network-policies-namespaces   PASS     high
 
$ oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |tail
  "test-compliancecheck-592",
  "test-compliancecheck-593",
  "test-compliancecheck-594",
  "test-compliancecheck-595",
  "test-compliancecheck-596",
  "test-compliancecheck-597",
  "test-compliancecheck-598",
  "test-compliancecheck-599",
  "test-compliancecheck-600"
]


$ for i in {601..999}; do oc create ns test-compliancecheck-${i}; oc -n test-compliancecheck-${i} create -f network-policy.yaml ; done
namespace/test-compliancecheck-601 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-602 created
networkpolicy.networking.k8s.io/allow-same-namespace created
....
....
namespace/test-compliancecheck-998 created
networkpolicy.networking.k8s.io/allow-same-namespace created
namespace/test-compliancecheck-999 created
networkpolicy.networking.k8s.io/allow-same-namespace created

$ oc get ns --no-headers | wc -l
1067

$ oc-compliance rerun-now compliancesuite/my-ssb-moderate -nopenshift-compliance
Rerunning scans from 'my-ssb-moderate': ocp4-moderate
Re-running scan 'openshift-compliance/ocp4-moderate'

$ oc get suite -w
NAME              PHASE     RESULT
my-ssb-moderate   RUNNING   NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT

$ oc get scan
NAME            PHASE   RESULT
ocp4-moderate   DONE    NON-COMPLIANT

$ oc get checkresult ocp4-moderate-configure-network-policies-namespaces
NAME                                                  STATUS   SEVERITY
ocp4-moderate-configure-network-policies-namespaces   PASS     high

$ oc get  namespaces -o json | jq '[.items[] | select((.metadata.name | startswith("openshift") | not) and (.metadata.name | startswith("kube-") | not) and .metadata.name != "default") | .metadata.name ]' |tail
  "test-compliancecheck-991",
  "test-compliancecheck-992",
  "test-compliancecheck-993",
  "test-compliancecheck-994",
  "test-compliancecheck-995",
  "test-compliancecheck-996",
  "test-compliancecheck-997",
  "test-compliancecheck-998",
  "test-compliancecheck-999"
]

Comment 9 errata-xmlrpc 2022-04-18 07:54:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1148