Description of problem: The customer has issues with the permissions checks that the openshift installer runs during the installation process. Version-Release number of the following components: 4.1 How reproducible: Here is what the customer says: The Openshift installer validates whether the user performing the installation has enough permissions to do the installation.To do so, it uses the IAM Policy Simulator API. However, when using AWS Organizations and Service control policies (SCP), the Policy Simulator may deny a permission when in reality the user is authorized. The following statement denies most actions outside of the us-east-1 region. { "Effect": "Deny", "Action": [ "ec2:Create*", "ec2:Run*", "eks:Create*", "rds:Create*", "es:Create*", "lambda:Create*" ], "Resource": [ "*" ], "Condition": { "StringNotEquals": Unknown macro: { "aws} } } When attempting to create a cluster with access keys with administrative access in the account, the validation fails with the ec2 permissions WARNING Action not allowed with tested creds action="ec2:CreateDhcpOptions" WARNING Action not allowed with tested creds action="ec2:CreateInternetGateway" WARNING Action not allowed with tested creds action="ec2:CreateNatGateway" WARNING Action not allowed with tested creds action="ec2:CreateRoute" WARNING Action not allowed with tested creds action="ec2:CreateRouteTable" WARNING Action not allowed with tested creds action="ec2:CreateSecurityGroup" WARNING Action not allowed with tested creds action="ec2:CreateSubnet" WARNING Action not allowed with tested creds action="ec2:CreateTags" WARNING Action not allowed with tested creds action="ec2:CreateVpc" WARNING Action not allowed with tested creds action="ec2:CreateVpcEndpoint" WARNING Action not allowed with tested creds action="ec2:CreateVolume" WARNING Action not allowed with tested creds action="ec2:RunInstances" WARNING Tested creds not able to perform all requested actions FATAL failed to fetch Terraform Variables: failed to fetch dependency of "Terraform Variables": failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation In this case, this happens because the simulation doesn’t consider the region to perform the simulation. Additional info: The customer suggests these checks should be optional.
I don't have an account where I can enable/manipulate AWS SCP at this moment, but I would like to understand this better. And I'll likely have to reach out to AWS to verify my findings. I've set up a new IAM user, and assigned group membership that allows ec2:DescribeInstances (with no restrictions). I then set up a Permissions Boundary and attached it to the user that denies ec2:DescribeInstances if the region is not us-east-1 (this was as close as I could get to simulate the AWS SCP configuration). When I run the policy simulator, it tells me that I can ec2:DescribeInstances in us-east-2, but when I go to actually perform the action it is denied: jdiaz@minigoomba bin (cco-bootstrap %=)]$ aws iam simulate-principal-policy --action-names '["ec2:DescribeInstances"]' --policy-source-arn "arn:aws:iam::125931421481:user/jdiazregiontest" --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyType=string,ContextKeyValues=us-east-2 | jq -r '.EvaluationResults[].EvalDecision' allowed [jdiaz@minigoomba bin (cco-bootstrap %=)]$ aws ec2 describe-instances --region=us-east-2 An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation. [jdiaz@minigoomba bin (cco-bootstrap %=)]$ Now when I create an inline policy for the same user with the same denial of ec2:DescribeInstances if region is not us-east-1, then the IAM policy simulation does work. [jdiaz@minigoomba bin (cco-bootstrap %=)]$ aws iam simulate-principal-policy --action-names '["ec2:DescribeInstances"]' --policy-source-arn "arn:aws:iam::125931421481:user/jdiazregiontest" --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyType=string,ContextKeyValues=us-east-2 | jq -r '.EvaluationResults[].EvalDecision' explicitDeny While it appears that the IAM SimulatePolicy API call does allow us to pass in a region when doing simulations, it does not appear to sufficiently simulate things when things like Permissions Boundaries (which are similar to AWS SCP as far as I can tell) are in effect.
When setting things up with Permissions Boundaries (in an effort to more closely mimic the AWS SCP setup as I understand it) I've only been able to set up an environment where I get false passing, but not false failing. The only way I've been able to get false failing conditions is when attaching policy denials/conditions directly attached to an AWS user. But this is far removed from AWS SCP/Permissions Boundaries. Can you provide the details on how the exact permissions are set up in the scenario originally reported in the BZ?
Also got this problem with 4.3.0 installer. Trying to install a cluster in us-east-2 with this policy that restricts all actions to us-east-2 region only: { "Version": "2012-10-17", "Statement": [ { "Sid": "DenyAllRegions", "Effect": "Deny", "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "config:*", "guardduty:*", "devicefarm:*", "directconnect:*" ], "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": [ "us-east-2" ] } } } ] } Using IPI installer results in: DEBUG Generating Platform Credentials Check... WARNING Action not allowed with tested creds action="ec2:AllocateAddress" [...] WARNING Action not allowed with tested creds action="ec2:DisassociateRouteTable" WARNING Action not allowed with tested creds action="ec2:ReplaceRouteTableAssociation" WARNING Tested creds not able to perform all requested actions FATAL failed to fetch Terraform Variables: failed to fetch dependency of "Terraform Variables": failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation Using the same credentials, I manually tried many of the actions and they worked. Example: $ aws ec2 describe-internet-gateways --region us-east-2 { "InternetGateways": [ { [...] Any workarounds for this?
Created attachment 1656528 [details] skip aws checks patch I got a workaround on this by patching the branch "release-4.3" from the installer and forcing the installer to continue even if it detects that you don't have the proper AWS permissions. Please note that this is completely unofficial and not supported, it's just a proof of concept that the installation can happen even if the installer complains about lack of permissions (IMHO, wrongly). $ git clone <openshift-installer-url> $ cd installer $ git checkout release-4.3 $ patch -p1 < /tmp/openshift-installer-43-skip-aws-checks.patch $ hack/build.sh $ sudo cp bin/openshift-installer /usr/local/bin/openshift-installer-custom Checking version: $ openshift-install-custom version openshift-install-custom unreleased-master-2206-ge8a128e459722896c9c099a7710aa60b30bf0cba-dirty built from commit e8a128e459722896c9c099a7710aa60b30bf0cba release image registry.svc.ci.openshift.org/origin/release:4.3 *Important*: the custom built installer by default uses OKD images, which are not compatible with OpenShift, so before installing, you should also define the proper release image. For 4.3: $ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release@sha256:3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d Then you can install: $ openshift-install-custom create cluster --dir=ocp4-cluster/ --log-level=debug
This bug is only targeted at tracking the fact that we don't properly scope our permissions checks to the appropriate region. Any additional issues outside the scope of that change should be filed as an independent bug.
I have set the customer escalation flag => on, based on Customer escalation EN-26619, case 02570853. The case owner is Novonil Choudhuri. Thank you, Dwayne Hellard Escalation Management Team Red Hat - Customer Experience & Engagement
Reproduce steps: # openshift-install version openshift-install v4.1.34-202002031224-dirty built from commit 1fd119d156f9dca16d6eaa9c888c512071c83c34 release image registry.svc.ci.openshift.org/ocp/release@sha256:570628002b6229010bac119be2ae0f76a431cea392a64bbc74efb0773f6da57b create a policy using the following json, that means most actions not in us-east-2 region is denied, everything in us-east-2 region is allowed for an IAM user with this policy attached. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" }, { "Effect": "Deny", "Action": [ "ec2:Create*", "ec2:Run*", "eks:Create*", "rds:Create*", "es:Create*", "lambda:Create*" ], "Resource": [ "*" ], "Condition": { "StringNotEquals": { "aws:RequestedRegion": "us-east-2" } } }, { "Effect": "Deny", "Action": "s3:CreateBucket", "Resource": "arn:aws:s3:::*", "Condition": { "StringLike": { "s3:LocationConstraint": "*" } } } ] } Trying to install a cluster in us-east-2 with this policy that restricts all actions to us-east-2 region only: # openshift-install create ignition-configs --dir demo3 INFO Consuming "Install Config" from target directory WARNING Action not allowed with tested creds action="ec2:CreateDhcpOptions" WARNING Action not allowed with tested creds action="ec2:CreateInternetGateway" WARNING Action not allowed with tested creds action="ec2:CreateNatGateway" WARNING Action not allowed with tested creds action="ec2:CreateRoute" WARNING Action not allowed with tested creds action="ec2:CreateRouteTable" WARNING Action not allowed with tested creds action="ec2:CreateSecurityGroup" WARNING Action not allowed with tested creds action="ec2:CreateSubnet" WARNING Action not allowed with tested creds action="ec2:CreateTags" WARNING Action not allowed with tested creds action="ec2:CreateVpc" WARNING Action not allowed with tested creds action="ec2:CreateVpcEndpoint" WARNING Action not allowed with tested creds action="ec2:CreateVolume" WARNING Action not allowed with tested creds action="ec2:RunInstances" WARNING Tested creds not able to perform all requested actions FATAL failed to fetch Bootstrap Ignition Config: failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation Return a false negative result. The same IAM user can launch an instance successfully in us-east-2 region. # aws ec2 run-instances --image-id ami-075b7fe10944d8748 --instance-type t2.medium { "Instances": [ { "Monitoring": { "State": "disabled" }, "PublicDnsName": "", "StateReason": { "Message": "pending", "Code": "pending" }, "State": { "Code": 0, "Name": "pending" }, "EbsOptimized": false, "LaunchTime": "2020-02-11T03:39:02.000Z", "PrivateIpAddress": "172.31.44.14", "ProductCodes": [], "VpcId": "vpc-0171369946eada62a", "CpuOptions": { "CoreCount": 2, "ThreadsPerCore": 1 }, "StateTransitionReason": "", "InstanceId": "i-0e87e067de7f1e62c", "ImageId": "ami-075b7fe10944d8748", "PrivateDnsName": "ip-172-31-44-14.us-east-2.compute.internal", "SecurityGroups": [ { "GroupName": "default", "GroupId": "sg-0c3ae688493a18161" } ], "ClientToken": "", "SubnetId": "subnet-0e6f2f221b0ea5381", "InstanceType": "t2.medium", "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "NetworkInterfaces": [ { "Status": "in-use", "MacAddress": "0a:32:9c:ae:40:5e", "SourceDestCheck": true, "VpcId": "vpc-0171369946eada62a", "Description": "", "NetworkInterfaceId": "eni-0bbaab9ae793dea2f", "PrivateIpAddresses": [ { "PrivateDnsName": "ip-172-31-44-14.us-east-2.compute.internal", "Primary": true, "PrivateIpAddress": "172.31.44.14" } ], "PrivateDnsName": "ip-172-31-44-14.us-east-2.compute.internal", "InterfaceType": "interface", "Attachment": { "Status": "attaching", "DeviceIndex": 0, "DeleteOnTermination": true, "AttachmentId": "eni-attach-0e01a958bc337408f", "AttachTime": "2020-02-11T03:39:02.000Z" }, "Groups": [ { "GroupName": "default", "GroupId": "sg-0c3ae688493a18161" } ], "Ipv6Addresses": [], "OwnerId": "301721915996", "SubnetId": "subnet-0e6f2f221b0ea5381", "PrivateIpAddress": "172.31.44.14" } ], "SourceDestCheck": true, "Placement": { "Tenancy": "default", "GroupName": "", "AvailabilityZone": "us-east-2c" }, "Hypervisor": "xen", "BlockDeviceMappings": [], "Architecture": "x86_64", "RootDeviceType": "ebs", "RootDeviceName": "/dev/sda1", "VirtualizationType": "hvm", "AmiLaunchIndex": 0 } ], "ReservationId": "r-08f6b7e33dc0aa0dd", "Groups": [], "OwnerId": "301721915996" } Extract installer from 4.4.0-0.nightly-2020-02-09-220310, run verification, passed. # openshift-install version openshift-install 4.4.0-0.nightly-2020-02-09-220310 built from commit 7289dd40ec78df4aa18e98fc4fbcac88d3d5a598 release image registry.svc.ci.openshift.org/ocp/release@sha256:edcdbd54d65f244182d11340a71682b959bb31e19bbc0c359ce53dc6abd728dc # cp ipi_template/install-config.yaml.aws demo3/install-config.yaml # openshift-install create ignition-configs --dir demo3 INFO Consuming Install Config from target directory INFO Credentials loaded from the "default" profile in file "/root/.aws/credentials"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581