Description of problem: Install fails because AWS Organization SCP policy restricts API calls to a specific region eg. eu-central-1 Version-Release number of selected component (if applicable): 4.3 release - SHA 637eaddb8031a33c8b95b667bc28bb0457007c2f54ab9aaeb0a7fe36d1eb4ea9 How reproducible: Steps to Reproduce: 1.Create SCP deny policy that restricts API calls to a specific region eg. eu-central-1 { "Version": "2012-10-17", "Statement": [ { "Sid": "DenyNotEnabledRegions", "Effect": "Deny", "NotAction": [ "iam:*" ], "Resource": [ "*" ], "Condition": { "StringNotEqualsIfExists": { "aws:RequestedRegion": [ "eu-central-1" ] } } }, { "Sid": "AllowEC2", "Effect": "Allow", "Action": [ "ec2:*" ], "Resource": [ "*" ] }, { "Sid": "AllowIAM", "Effect": "Allow", "Action": [ "iam:*" ], "Resource": [ "*" ] } ] } 2. Run installer with specified region eu-central-1 Actual results: Installer fails to validate credentials time="2020-03-19T23:34:10Z" level=fatal msg="failed to fetch Master Machines: failed to fetch dependency of \"Master Machines\": failed to generate asset \"Platform Credentials Check\": validate AWS credentials: current credentials insufficient for performing cluster installation" time="2020-03-19T23:34:11Z" level=error msg="error after waiting for command completion" error="exit status 1" installID=c4zfdd8z time="2020-03-19T23:34:11Z" level=error msg="error generating installer assets" error="exit status 1" installID=c4zfdd8z time="2020-03-19T23:34:11Z" level=info msg="reading installer log" installID=c4zfdd8z time="2020-03-19T23:34:11Z" level=info msg="saving installer output" installID=c4zfdd8z time="2020-03-19T23:34:11Z" level=debug msg="installer console log: level=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AllocateAddress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AssociateAddress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AuthorizeSecurityGroupEgress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AuthorizeSecurityGroupIngress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CopyImage\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateNetworkInterface\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateSecurityGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateVolume\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteSecurityGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteSnapshot\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeregisterImage\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeAccountAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeAddresses\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeAvailabilityZones\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeImages\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInstanceAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInstanceCreditSpecifications\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInstances\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInternetGateways\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeKeyPairs\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeNatGateways\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeNetworkAcls\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeNetworkInterfaces\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribePrefixLists\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeRegions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeRouteTables\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeSecurityGroups\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeSubnets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVolumes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcClassicLink\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcClassicLinkDnsSupport\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcEndpoints\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcs\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifyInstanceAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifyNetworkInterfaceAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ReleaseAddress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:RevokeSecurityGroupEgress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:RevokeSecurityGroupIngress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:RunInstances\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:TerminateInstances\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:AddTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:AttachLoadBalancerToSubnets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ConfigureHealthCheck\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateListener\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateLoadBalancerListeners\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateTargetGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeleteLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeregisterInstancesFromLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeregisterTargets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeInstanceHealth\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeListeners\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeLoadBalancerAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeLoadBalancers\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTargetGroupAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTargetHealth\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ModifyLoadBalancerAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ModifyTargetGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ModifyTargetGroupAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:RegisterInstancesWithLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:RegisterTargets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:SetLoadBalancerPoliciesOfListener\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ChangeResourceRecordSets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ChangeTagsForResource\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:CreateHostedZone\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:DeleteHostedZone\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:GetChange\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:GetHostedZone\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListHostedZones\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListHostedZonesByName\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListResourceRecordSets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListTagsForResource\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:UpdateHostedZoneComment\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:CreateBucket\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:DeleteBucket\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetAccelerateConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketCors\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketLocation\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketLogging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketObjectLockConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketReplication\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketRequestPayment\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketVersioning\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketWebsite\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetEncryptionConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetLifecycleConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetReplicationConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:ListBucket\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutBucketAcl\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutBucketTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutEncryptionConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:DeleteObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObjectAcl\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObjectTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObjectVersion\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutObjectAcl\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutObjectTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"autoscaling:DescribeAutoScalingGroups\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteNetworkInterface\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteVolume\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeleteTargetGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTargetGroups\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:DeleteObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"tag:GetResources\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AssociateDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AssociateRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AttachInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateNatGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateRoute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateSubnet\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateVpc\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateVpcEndpoint\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifySubnetAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifyVpcAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteNatGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteRoute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteSubnet\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteVpc\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteVpcEndpoints\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DetachInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DisassociateRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ReplaceRouteTableAssociation\"\nlevel=warning msg=\"Tested creds not able to perform all requested actions\"\nlevel=fatal msg=\"failed to fetch Master Machines: failed to fetch dependency of \\\"Master Machines\\\": failed to generate asset \\\"Platform Credentials Check\\\": validate AWS credentials: current credentials insufficient for performing cluster installation\"\n" installID=c4zfdd8z time="2020-03-19T23:34:11Z" level=info msg="updating clusterprovision" installID=c4zfdd8z time="2020-03-19T23:34:11Z" level=fatal msg="runtime error" error="exit status 1" Expected results: Installer successfully installs in allowed region Additional info: While the installer cannot validate the permissions for the region eu-central-1 we can make API calls to the resources returning "ImplicitDeny" by the policy simulator in the installer in region eu-central-1 $ aws sts get-caller-identity --profile=xxxxxxxxx { "Account": "xxxxxxxxx", "UserId": "xxxxxxxxx", "Arn": "arn:aws:iam::xxxxxxxxx:user/osdCcsAdmin" } $ aws --profile=xxxxxxxxx --region=eu-central-1 iam simulate-principal-policy --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyValues=eu-central-1,ContextKeyType=string --policy-source-arn=arn:aws:iam::xxxxxxxxx:user/osdCcsAdmin --action-name ec2:DescribeAvailabilityZones { "EvaluationResults": [ { "EvalDecision": "implicitDeny", "MissingContextValues": [], "EvalActionName": "ec2:DescribeAvailabilityZones", "MatchedStatements": [], "EvalResourceName": "*" } ] } $ aws ec2 describe-availability-zones --profile=xxxxxxxxx --region=eu-central-1 { "AvailabilityZones": [ { "State": "available", "RegionName": "eu-central-1", "Messages": [], "ZoneName": "eu-central-1a" }, { "State": "available", "RegionName": "eu-central-1", "Messages": [], "ZoneName": "eu-central-1b" }, { "State": "available", "RegionName": "eu-central-1", "Messages": [], "ZoneName": "eu-central-1c" } ] } $ aws ec2 describe-availability-zones --profile=xxxxxxxxx --region=us-east-1 An error occurred (UnauthorizedOperation) when calling the DescribeAvailabilityZones operation: You are not authorized to perform this operation. Its possible that this error is because AWS are restricting the endpoint URL access to only eu-central-1. You can see that us-east-1 is denied in the above CLI call but not eu-central-1 indicating that ec2:DescribeAvailabilityZones should be allowed although the installers (CCO) policy simulator is returning an implicit deny. Maybe we need to pass an endpoint URL however IAM is global?
*** This bug has been marked as a duplicate of bug 1757244 ***