Bug 2088201

Summary:	File integrity operator should comply to restricted pod security level
Product:	OpenShift Container Platform	Reporter:	xiyuan
Component:	File Integrity Operator	Assignee:	Matt Rogers <mrogers>
Status:	CLOSED ERRATA	QA Contact:	xiyuan
Severity:	high	Docs Contact:
Priority:	high
Version:	4.11	CC:	jhrozek, lbragsta, wenshen
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The operator used some overly broad permissions coupled with missing labels to show it needed certain permissions to function correctly. Consequence: This caused log warnings about potential excessive permissions being granted to the operator. Fix: Some permissions where reduced where they were too excessive and appropriate labels were applied to the namespace. Result: The operator logs no longer warn about potential permission issues when running the operator on OpenShift 4.11	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-02 08:17:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description xiyuan 2022-05-19 02:05:42 UTC

Description:
File integrity operator workloads should comply to restricted pod security level


Version :
4.11.0-0.nightly-2022-05-11-054135 + compliance-operatorv0.1.51-1

How to reproduce it (as minimally and precisely as possible)?
Always.


Steps to Reproduce:
Install File integrity Operator operator from Operators->OperatorHub, create fileintegrity  for test.
$ cat test.sh
# All workloads creation is audited on masters with below annotation. Below cmd checks all workloads that would violate PodSecurity.
cat > cmd.txt << EOF
grep -hir 'would violate PodSecurity' /var/log/kube-apiserver/ | jq -r '.requestURI + " " + .annotations."pod-security.kubernetes.io/audit-violations"'
EOF

CMD="`cat cmd.txt`"
oc new-project xxia-test

# With admin, run above cmd on all masters:
MASTERS=`oc get no | grep master | grep -o '^[^ ]*'`
for i in $MASTERS
do
  oc debug -n xxia-test no/$i -- chroot /host bash -c "$CMD || true"
done > all-violations.txt

cat all-violations.txt | grep -E 'namespaces/(openshift|kube)-' | sort | uniq > all-violations_system_components.txt
cat all-violations_system_components.txt
 
In 4.11.0-0.nightly-2022-05-11-054135 env, run above script with admin: 
./test.sh
 
Got:
/apis/apps/v1/namespaces/openshift-file-integrity/daemonsets/aide-example-fileintegrity would violate PodSecurity "restricted:latest": hostPath volumes (volume "hostroot"), privileged (container "daemon" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "daemon" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "daemon" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "hostroot" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "daemon" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "daemon" must not set runAsUser=0), seccompProfile (pod or container "daemon" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-file-integrity/daemonsets would violate PodSecurity "restricted:latest": hostPath volumes (volume "hostroot"), privileged (container "daemon" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "daemon" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "daemon" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "hostroot" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "daemon" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "daemon" must not set runAsUser=0), seccompProfile (pod or container "daemon" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-file-integrity/daemonsets would violate PodSecurity "restricted:latest": hostPath volumes (volume "hostroot"), privileged (container "reinit-script" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "reinit-script", "pause-script" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "reinit-script", "pause-script" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "hostroot" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "reinit-script", "pause-script" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "reinit-script" must not set runAsUser=0), seccompProfile (pod or containers "reinit-script", "pause-script" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-file-integrity/deployments/file-integrity-operator would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "file-integrity-operator" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "file-integrity-operator" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "file-integrity-operator" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "file-integrity-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-file-integrity/deployments would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "file-integrity-operator" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "file-integrity-operator" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "file-integrity-operator" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "file-integrity-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/api/v1/namespaces/openshift-file-integrity/pods would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/api/v1/namespaces/openshift-file-integrity/pods would violate PodSecurity "restricted:latest": hostPath volumes (volume "hostroot"), privileged (container "daemon" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "daemon" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "daemon" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "hostroot" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "daemon" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "daemon" must not set runAsUser=0), seccompProfile (pod or container "daemon" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/api/v1/namespaces/openshift-file-integrity/pods would violate PodSecurity "restricted:latest": hostPath volumes (volume "hostroot"), privileged (container "reinit-script" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "reinit-script", "pause-script" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "reinit-script", "pause-script" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "hostroot" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "reinit-script", "pause-script" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "reinit-script" must not set runAsUser=0), seccompProfile (pod or containers "reinit-script", "pause-script" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Expected results:
File integrity operator workload should comply to restricted pod security level

Additional info:

Comment 2 Jakub Hrozek 2022-06-02 17:59:11 UTC

Not a blocker anymore because the feature will not be enabled in enforcing mode in 4.11

Comment 4 Jakub Hrozek 2022-06-27 12:23:55 UTC

Partial fix before OLM allows us to set the proper labels: https://github.com/openshift/file-integrity-operator/pull/239
Leaving this ASSIGNED until then, though.

Comment 5 Matt Rogers 2022-07-14 16:05:25 UTC

Let's track the OLM label fix (required for 4.12) in https://issues.redhat.com/browse/CMP-1447. Otherwise this bug will be blocked by OLM requirements for awhile.

I suggest that with https://github.com/openshift/file-integrity-operator/pull/239 in, we verify that the warnings are squashed when deploying directly from manifest, and resolve this bug.

Comment 6 Lance Bragstad 2022-07-14 17:07:24 UTC

Moving this to post since the patch in comment #5 landed upstream.

Comment 8 Lance Bragstad 2022-07-15 13:00:10 UTC

Setting this to ON_QA since we have a build.

Comment 9 xiyuan 2022-07-20 03:54:29 UTC

verification pass with 4.11.0-rc.3 and file-integrity-operator.v0.1.28.
When creating namespace, remember to create namespace with labels:
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-file-integrity
  labels:
    openshift.io/cluster-monitoring: "true"
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
Tried two scenarios:
1. install file-integrity-operator.v01.1.24 and upgrade to file-integrity-operator.v0.1.28.
2. install file-integrity-operator.v0.1.28
For both scenarios, it won't report "violate PodSecurity" for file integrity operator.

$ ./test.sh
Now using project "xxia-test" on server "https://api.xiyuan20-1.alicloud-qe.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname

Warning: would violate PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/xiyuan20-1-s92vw-master-0-debug ...
To use host binaries, run `chroot /host`

Removing debug pod ...
Warning: would violate PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/xiyuan20-1-s92vw-master-1-debug ...
To use host binaries, run `chroot /host`

Removing debug pod ...
Warning: would violate PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/xiyuan20-1-s92vw-master-2-debug ...
To use host binaries, run `chroot /host`

Removing debug pod ...
/apis/apps/v1/namespaces/openshift-logging/deployments/cluster-logging-operator would violate PodSecurity "restricted:latest": unrestricted capabilities (container "cluster-logging-operator" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "cluster-logging-operator" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "cluster-logging-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-logging/deployments would violate PodSecurity "restricted:latest": unrestricted capabilities (container "cluster-logging-operator" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "cluster-logging-operator" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "cluster-logging-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-logging/replicasets would violate PodSecurity "restricted:latest": unrestricted capabilities (container "cluster-logging-operator" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "cluster-logging-operator" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "cluster-logging-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-operators-redhat/deployments/elasticsearch-operator would violate PodSecurity "restricted:latest": seccompProfile (pod or containers "kube-rbac-proxy", "elasticsearch-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-operators-redhat/deployments would violate PodSecurity "restricted:latest": seccompProfile (pod or containers "kube-rbac-proxy", "elasticsearch-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
/apis/apps/v1/namespaces/openshift-operators-redhat/replicasets would violate PodSecurity "restricted:latest": seccompProfile (pod or containers "kube-rbac-proxy", "elasticsearch-operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Comment 12 errata-xmlrpc 2022-08-02 08:17:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift File Integrity Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5538