Bug 2082254

Summary: OCP 4.11 - Install fails because of: pods "management-ingress-63029-5cf6789dd6-" is forbidden: unable to validate against any security context constraint
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Constantin Vultur <cvultur>
Component: Core Services / ObservabilityAssignee: Subbarao Meduri <smeduri>
Status: CLOSED ERRATA QA Contact: Xiang Yin <xiyin>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhacm-2.5.zCC: akandath, amagrawa, cqu, dbewley, jagray, keyoung, mbukatov, mcornea, ngangadh, njean, nmanos, smeduri, vboulos
Target Milestone: ---Flags: cqu: qe_test_coverage-
bot-tracker-sync: rhacm-2.5.z+
Target Release: rhacm-2.5.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-13 20:06:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Constantin Vultur 2022-05-05 17:03:40 UTC
Description of the problem:
Cannot get ACM 2.5 installed on  4.11.0-0.nightly-2022-05-05-015322

Release version:
2.5.0-DOWNSTREAM-2022-04-29-18-21-49

Operator snapshot version:
2.5.0-DOWNSTREAM-2022-04-29-18-21-49

OCP version:
4.11.0-0.nightly-2022-05-05-015322


Browser Info:

Steps to reproduce:
1. Install snapshot on 4.11 IPv6/disconnected cluster
2. Install ACM
3.

Actual results:
Installation hangs on replicaset management-ingress timeout

# oc get mch multiclusterhub -n rhacm -o json | jq .status
{
  "components": {
    "cluster-lifecycle-sub": {
      "lastTransitionTime": "2022-05-05T15:27:11Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "console-chart-sub": {
      "lastTransitionTime": "2022-05-05T15:27:11Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "grc-sub": {
      "lastTransitionTime": "2022-05-05T15:27:12Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "local-cluster": {
      "lastTransitionTime": "2022-05-05T15:24:32Z",
      "message": "No conditions available",
      "reason": "No conditions available",
      "status": "Unknown",
      "type": "Unknown"
    },
    "management-ingress-sub": {
      "lastTransitionTime": "2022-05-05T15:38:15Z",
      "message": "ReplicaSet \"management-ingress-63029-5cf6789dd6\" has timed out progressing.",
      "reason": "ProgressDeadlineExceeded",
      "status": "False",
      "type": "Progressing"
    },
    "multicluster-engine": {
      "lastTransitionTime": "2022-05-05T16:55:09Z",
      "reason": "ComponentsAvailable",
      "status": "True",
      "type": "Available"
    },
    "multicluster-engine-csv": {
      "lastTransitionTime": "2022-05-05T16:55:09Z",
      "message": "install strategy completed with no errors",
      "reason": "InstallSucceeded",
      "status": "True",
      "type": "Available"
    },
    "multicluster-engine-sub": {
      "lastTransitionTime": "2022-05-05T16:55:09Z",
      "message": "installPlanApproval: Automatic. installPlan: multicluster-engine/install-t5w2w",
      "reason": "AtLatestKnown",
      "status": "True",
      "type": "Available"
    },
    "multiclusterhub-repo": {
      "lastTransitionTime": "2022-05-05T15:25:13Z",
      "reason": "MinimumReplicasAvailable",
      "status": "True",
      "type": "Available"
    },
    "policyreport-sub": {
      "lastTransitionTime": "2022-05-05T15:27:11Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "search-prod-sub": {
      "lastTransitionTime": "2022-05-05T15:28:15Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "volsync-addon-controller-sub": {
      "lastTransitionTime": "2022-05-05T15:27:11Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    }
  },
  "conditions": [
    {
      "lastTransitionTime": "2022-05-05T15:25:01Z",
      "lastUpdateTime": "2022-05-05T15:25:07Z",
      "message": "Created new resource",
      "reason": "NewResourceCreated",
      "status": "True",
      "type": "Progressing"
    }
  ],
  "desiredVersion": "2.5.0",
  "phase": "Installing"
}

Checking the replicaset, the error shows up:

# oc describe replicaset management-ingress-63029-5cf6789dd6 -n rhacm
......
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  5m40s (x27 over 68m)  replicaset-controller  Error creating: pods "management-ingress-63029-5cf6789dd6-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, spec.containers[1].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]




Expected results:

Installation to succeed

Additional info:

Comment 1 Jakob 2022-05-11 16:43:06 UTC
*** Bug 2083723 has been marked as a duplicate of this bug. ***

Comment 2 Jakob 2022-05-24 22:46:21 UTC
I was able to get around the management ingress issue and complete the install by providing the ingress serviceaccount with the access it needs. I ran `oc adm policy add-scc-to-user privileged -z management-ingress-622b1-sa  -n open-cluster-management` where `management-ingress-622b1-sa` was the name of the service account used in the deployment (The suffix is generated and subject to change on any given install). This should help get around the problem until the team can identify a more concrete solution to dealing with OCP's security changes.

Comment 3 Vincent Boulos 2022-06-07 18:47:11 UTC
The workaround looks ok, however it seems we have another issue preventing ACM to get successfully installed, the MCH isn't ready because ManagedClusterLeaseUpdateStopped.

# oc get mch multiclusterhub -n ocm -o json | jq .status
{
  "components": {
    "cluster-backup-chart-sub": {
      "lastTransitionTime": "2022-06-07T14:31:49Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "cluster-lifecycle-sub": {
      "lastTransitionTime": "2022-06-07T14:32:52Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "console-chart-sub": {
      "lastTransitionTime": "2022-06-07T14:31:50Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "grc-sub": {
      "lastTransitionTime": "2022-06-07T14:31:50Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "local-cluster": {
      "lastTransitionTime": "2022-06-07T18:02:32Z",
      "message": "Registration agent stopped updating its lease.",
      "reason": "ManagedClusterLeaseUpdateStopped",
      "status": "Unknown",
      "type": "ManagedClusterConditionAvailable"
    },
    "management-ingress-sub": {
      "lastTransitionTime": "2022-06-07T14:31:49Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "multicluster-engine": {
      "lastTransitionTime": "2022-06-07T18:02:32Z",
      "reason": "ComponentsAvailable",
      "status": "True",
      "type": "Available"
    },
    "multicluster-engine-csv": {
      "lastTransitionTime": "2022-06-07T18:02:32Z",
      "message": "install strategy completed with no errors",
      "reason": "InstallSucceeded",
      "status": "True",
      "type": "Available"
    },
    "multicluster-engine-sub": {
      "lastTransitionTime": "2022-06-07T18:02:32Z",
      "message": "installPlanApproval: Automatic. installPlan: multicluster-engine/install-jk7tg",
      "reason": "AtLatestKnown",
      "status": "True",
      "type": "Available"
    },
    "multiclusterhub-repo": {
      "lastTransitionTime": "2022-06-07T14:30:37Z",
      "reason": "MinimumReplicasAvailable",
      "status": "True",
      "type": "Available"
    },
    "policyreport-sub": {
      "lastTransitionTime": "2022-06-07T14:31:49Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "search-prod-sub": {
      "lastTransitionTime": "2022-06-07T14:31:50Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    },
    "volsync-addon-controller-sub": {
      "lastTransitionTime": "2022-06-07T14:31:49Z",
      "reason": "InstallSuccessful",
      "status": "True",
      "type": "Deployed"
    }
  },
  "conditions": [
    {
      "lastTransitionTime": "2022-06-07T14:30:31Z",
      "lastUpdateTime": "2022-06-07T14:30:32Z",
      "message": "created new resource: CustomResourceDefinition managedclusteractions.action.open-cluster-management.io",
      "reason": "NewResourceCreated",
      "status": "True",
      "type": "Progressing"
    }
  ],
  "desiredVersion": "2.5.0",
  "phase": "Installing"
}

Comment 4 Vincent Boulos 2022-06-14 13:48:16 UTC
*** Bug 2083723 has been marked as a duplicate of this bug. ***

Comment 5 Kevin Cormier 2022-06-15 13:28:38 UTC
*** Bug 2097304 has been marked as a duplicate of this bug. ***

Comment 9 Vincent Boulos 2022-09-01 13:00:54 UTC
Verified on 2.5.2-DOWNSTREAM-2022-08-02-15-08-54 and 2.5.2-RC4 builds

Comment 10 Vincent Boulos 2022-09-02 15:04:31 UTC
Correction: 
Verified on 2.5.2-DOWNSTREAM-2022-08-02-15-08-54 and 2.5.2-FC3 builds

Comment 15 errata-xmlrpc 2022-09-13 20:06:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Critical: Red Hat Advanced Cluster Management 2.5.2 security fixes and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6507