Bug 1829101 - OCP issues with AWS Organizations SCPs
Summary: OCP issues with AWS Organizations SCPs
Keywords:
Status: ON_QA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.3.z
Hardware: All
OS: All
unspecified
high
Target Milestone: ---
: 4.6.0
Assignee: Joel Diaz
QA Contact: wang lin
URL:
Whiteboard:
: 1832640 (view as bug list)
Depends On:
Blocks: 1757244
TreeView+ depends on / blocked
 
Reported: 2020-04-28 21:31 UTC by Juanjo Floristan
Modified: 2020-08-03 19:47 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift api pull 692 None closed create config for cloud-credential-operator 2020-08-05 06:29:57 UTC
Github openshift cloud-credential-operator pull 227 None closed start using the CCO config object 2020-08-05 06:29:57 UTC
Github openshift cloud-credential-operator pull 228 None closed handle bootstrap user-defined mode 2020-08-05 06:29:57 UTC
Github openshift installer pull 3919 None closed types: add field to InstallConfig to force credentials mode 2020-08-05 06:29:57 UTC
Github openshift installer pull 3968 None closed types: capitalize CredentialsMode values 2020-08-05 06:29:57 UTC

Description Juanjo Floristan 2020-04-28 21:31:42 UTC
Description of problem:

 OpenShift does not support AWS IPI or UPI installations when using AWS 
 Organizations and Service Control Policies if there is a rule that deny all 
 actions or a specific required permission using a global condition, i.e. “for 
 all regions except us-east-1 and us-west-2” or “for all roles except role 
 A”.. even if the credentials have actually the permission to perform that 
 action.

 The reason this happens is because OpenShift depends on an AWS policy 
 simulator API that fails to validate conditions in AWS Organizations SCPs, 
 providing false negatives, and the installation cannot proceed if the 
 validation fails.

 The BZ 1750338 addresses the problem of passing the region when policies 
 include global conditions based on the region, but this only works with IAM 
 policies, it does not work in AWS Organizations SCPs, for the reason 
 explained, the API does not evaluate global conditions against SCPs. Therefore 
 it only sees the rule, applied unconditionally.

 This problem is hit by the openshift-installer and the cloud-credential- 
 operator


How reproducible:

 Implement an environment in AWS using AWS Organizations and SCPs to control 
 permissions, in any of the SCPs applied to the user or account whose 
 credentials will be provided to OCP, make sure you include a statement like 
 this:

      {
         "Sid":"DenyOtherRegions",
         "Effect":"Deny",
         "Resource":"*",
         "Action":"*",
         "Condition":{
            "StringNotEquals":{
               "aws:RequestedRegion":[
                  "us-east-1",
                  "us-west-2"
               ]
            }
         }
      }

 Note: The AWS credentials provided have all the permissions needed by Openshift 

 Try to install Openshift IPI in AWS following the official documentation

 $ openshift-install create cluster --dir=ocp

Actual results:

 Installer fails to install due to validation permissions even when the user 
 has all the permission to perform the actual actions. AWS validation fails 
 because it ignores the region condition and sees only "Deny: *" statement 
 (false negative)

 $ openshift-install create cluster --dir=ocp
 ...
 WARNING Action not allowed with tested creds action="ec2:CreateDhcpOptions"
 WARNING Action not allowed with tested creds 
 action="ec2:CreateInternetGateway"
 WARNING Action not allowed with tested creds action="ec2:CreateNatGateway"
 WARNING Action not allowed with tested creds action="ec2:CreateRoute"
 WARNING Action not allowed with tested creds action="ec2:CreateRouteTable"
 WARNING Action not allowed with tested creds action="ec2:CreateSecurityGroup"
 WARNING Action not allowed with tested creds action="ec2:CreateSubnet"
 WARNING Action not allowed with tested creds action="ec2:CreateTags"
 
 I patched the installer following the workaround suggested in 
 https://bugzilla.redhat.com/show_bug.cgi?id=1750338#c25:

 The installer was still failing with the error:
 
 “AWS credentials cannot be used to either create new creds or use as-is”
 
 The I had to manually patch pkg/asset/installconfig/aws/permissions.go to 
 prevent the mint and  passthrough credentials errors (they do not work 
 either) and force a successful return add the end of the module:

        # return nil added after failing to verify mint or passthrough checks
        return nil
        return errors.New("AWS credentials cannot be used to either create new 
                           creds or use as-is")

 I could made the installer progress but then is the cloud-credential- 
 operator who fails to validate permissions too, preventing any 
 CredentialRequest from being granted and therefore blocking the cluster 
 deployment completion

 $ oc logs cloud-credential-operator-69479545fc-mlcn7 -n openshift-cloud- 
 credential-operator -f

 time="2020-04-16T00:52:03Z" level=info msg="calculating metrics for all 
 CredentialsRequests" controller=metrics
 time="2020-04-16T00:52:03Z" level=info msg="reconcile complete" 
 controller=metrics elapsed=1.660646ms
 time="2020-04-16T00:52:08Z" level=info msg="validating cloud cred secret" 
 controller=secretannotator
 time="2020-04-16T00:52:08Z" level=debug msg="Loading infrastructure name: 
 oc4poc-fw6q6" controller=secretannotator
 time="2020-04-16T00:52:08Z" level=warning msg="Action not allowed with tested 
 creds" action="iam:CreateAccessKey" controller=secretannotator
 time="2020-04-16T00:52:08Z" level=warning msg="Action not allowed with tested 
 creds" action="iam:CreateUser" controller=secretannotator
 time="2020-04-16T00:52:08Z" level=warning msg="Action not allowed with tested 
 creds" action="iam:DeleteAccessKey" controller=secretannotator

 After this worker nodes are not deployed and there are no pending CSRs. The 
 cluster stops deploying with operators depending on cloud credentials blocked

Expected results:

 Openshift to install normally, maybe adding a mechanism to manually allow 
 ignoring the validation of permissions for the provided AWS credentials

Comment 2 Abhinav Dahiya 2020-04-29 22:30:36 UTC
To moving to credm-minter. I think we need to fix this in the cluster before we fix it in the installer.

Comment 3 James Harrington 2020-04-30 12:55:41 UTC
This is the same issue that I raised here https://bugzilla.redhat.com/show_bug.cgi?id=1815331

Comment 4 Joel Diaz 2020-05-05 14:40:21 UTC
We are presently pursuing access to an SCP-enabled environment to investigate/replicate.

Comment 5 Joel Diaz 2020-05-14 19:17:42 UTC
For interested parties, the enhancement proposal to allow bypassing pre-flight permissions checks posted at https://github.com/openshift/enhancements/pull/324

Comment 6 Devan Goodwin 2020-05-15 17:46:49 UTC
*** Bug 1832640 has been marked as a duplicate of this bug. ***

Comment 8 Joel Diaz 2020-05-22 17:50:49 UTC
Summary of investigation was that the permissions simulation is indeed unreliable when SCP is being used in an account. The enhancement proposal https://github.com/openshift/enhancements/pull/324 is written to allow instructing the OpenShift installer to not perform permissions checks before the installation, and to indicate to the in-cluster cloud-credential-operator that it too should not perform permissions simulations (CCO must be told whether to run in 'mint' or 'passthrough' mode through).

Comment 9 Greg Sheremeta 2020-06-18 18:21:09 UTC
waiting on enhancement to be approved

Comment 12 Devan Goodwin 2020-07-16 14:21:14 UTC
This bugfix is actively being worked on and we expect to complete this sprint.


Note You need to log in before you can comment on or make changes to this bug.