Description of problem ====================== When I install ODF 4.11 Operator via OCP Console, alert PodSecurityViolation starts to fire. Version-Release number of selected component ============================================ OCP 4.11.0-0.nightly-2022-07-19-104004 ODF 4.11.0-124 How reproducible ================ 3/3 Steps to Reproduce ================== 1. Install minimal OCP cluster 2. Install ODF Operator via OCP console 3. Check alerts raised in OCP Console Actual results ============== Alert PodSecurityViolation is firing. See attached screenshot. Expected results ================ Alert PodSecurityViolation is not firing. Additional info =============== The description of the alert is: > A workload (pod, deployment, deamonset, ...) was created somewhere in the > cluster but it did not match the PodSecurity "restricted" profile defined by > its namespace either via the cluster-wide configuration (which triggers on a > "restricted" profile violations) or by the namespace local Pod Security > labels. Refer to Kubernetes documentation on Pod Security Admission to learn > more about these violations. Must gather data will be referenced in a comment. When I check alert details, I see that the moment when the started firing corelates with the moment when the ODF operator was being installed (see attache screenshot #2): I made screenshot at the beggining of the operator installation at 18:55 CEST, and the alert started firing at 18:56 CEST. Quick check of events shows one possible culprit in ODF namespace: ``` $ oc get events --all-namespaces | grep -i security openshift-storage 33m Warning FailedCreate replicaset/ocs-metrics-exporter-84676558b Error creating: pods "ocs-metrics-exporter-84676558b-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, pro vider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "no de-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] ``` But I don't claim that neither this is the root cause nor that it's the only possible reason for the problem. I'm not familiar with debugging pod security violations. This is flagged as a regression because this problem didn't exist in ODF 4.10 clusters.
Created attachment 1899212 [details] screenshot #1: PodSecurityViolation alert firign after ODF operator installation
Created attachment 1899213 [details] screenshot #2: PodSecurityViolation alert details
Quick overview of pods running in openshift-storage namespace (after ODF operator installation, while the alert is firing): ``` $ oc get pods -n openshift-storage NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-565845d567-hxpdk 2/2 Running 2 (12m ago) 61m noobaa-operator-787bc89f64-j47mk 1/1 Running 0 61m ocs-metrics-exporter-f9687ddb9-dgh8c 1/1 Running 0 60m ocs-operator-6b49685945-5xvrx 1/1 Running 4 (12m ago) 61m odf-console-6cbd88d546-8fwgt 1/1 Running 0 61m odf-operator-controller-manager-5ffcdb76cc-qwvcm 2/2 Running 4 (7m14s ago) 61m rook-ceph-operator-845f8c9558-x5hvq 1/1 Running 0 61m ```
added required doc text. @Orit, please review.
I am on ocp 4.12 nightly build and I can't see this alert anymore in the UI, while installing odf. This is still there on ocp 4.11 clusters though. I think some security setting has changed on ocp side for 4.12. My ocp version info- mparida@mparida-mac ~ % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0-0.nightly-2022-10-20-104328 True False 16m Cluster version is 4.12.0-0.nightly-2022-10-20-104328 Somebody from QE can verify this, And if verified we can think of what to do next.
I read comment 22 from Malay and Mudit's move to ON_QA from ASSIGNED state as an decision to verify this without any changes in ODF, assuming chagnges in OCP are enough not just from perspective misleading alert, but also from best openshift security practices. If that is not the case, I'm asking Mudit to create a new bug for tracking updates on ODF side. Verifying on vSphere platform with: OCP 4.12.0-0.nightly-2022-10-25-210451 ODF 4.12.0-82 And I see: - no PodSecurityViolation alert firing after installation of ODF operator. - no PodSecurityViolation alert firing after creating StorageSystem. And last but not least, there are no FailedCreate events: ``` $ oc get events --all-namespaces | grep -i security | grep FailedCreate $ ``` >>> VERIFIED
I see the pod security violation warning started firing from 4th April 10:33 UTC/ 16:03 IST, But I see the odf operator being installed(its pods coming up) on 3rd April at 11:53 UTC/ 17:23 IST. For more testing, I tried on another cluster that had odf installed & had also pod security violation warnings. I uninstalled odf & deleted the openshift-storage ns entirely, and waited for an hour or so, But I still see the warning firing. So we can conclude it's not due to ODF.