Bug 1815331 - AWS simulate principal policy fails when AWS organization SCP policy restricts region
Summary: AWS simulate principal policy fails when AWS organization SCP policy restrict...
Keywords:
Status: CLOSED DUPLICATE of bug 1757244
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Abhinav Dahiya
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-20 02:59 UTC by James Harrington
Modified: 2023-05-15 07:49 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-20 03:13:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description James Harrington 2020-03-20 02:59:07 UTC
Description of problem:
Install fails because AWS Organization SCP policy restricts API calls to a specific region eg. eu-central-1

Version-Release number of selected component (if applicable):
4.3 release - SHA 637eaddb8031a33c8b95b667bc28bb0457007c2f54ab9aaeb0a7fe36d1eb4ea9

How reproducible:

Steps to Reproduce:
1.Create SCP deny policy that restricts API calls to a specific region eg. eu-central-1

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "DenyNotEnabledRegions",
			"Effect": "Deny",
			"NotAction": [
				"iam:*"
			],
			"Resource": [
				"*"
			],
			"Condition": {
				"StringNotEqualsIfExists": {
					"aws:RequestedRegion": [
						"eu-central-1"
					]
				}
			}
		},
		{
			"Sid": "AllowEC2",
			"Effect": "Allow",
			"Action": [
				"ec2:*"
			],
			"Resource": [
				"*"
			]
		},
		{
			"Sid": "AllowIAM",
			"Effect": "Allow",
			"Action": [
				"iam:*"
			],
			"Resource": [
				"*"
			]
		}
	]
}

2. Run installer with specified region eu-central-1

Actual results:
Installer fails to validate credentials

time="2020-03-19T23:34:10Z" level=fatal msg="failed to fetch Master Machines: failed to fetch dependency of \"Master Machines\": failed to generate asset \"Platform Credentials Check\": validate AWS credentials: current credentials insufficient for performing cluster installation"
time="2020-03-19T23:34:11Z" level=error msg="error after waiting for command completion" error="exit status 1" installID=c4zfdd8z
time="2020-03-19T23:34:11Z" level=error msg="error generating installer assets" error="exit status 1" installID=c4zfdd8z
time="2020-03-19T23:34:11Z" level=info msg="reading installer log" installID=c4zfdd8z
time="2020-03-19T23:34:11Z" level=info msg="saving installer output" installID=c4zfdd8z
time="2020-03-19T23:34:11Z" level=debug msg="installer console log: level=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AllocateAddress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AssociateAddress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AuthorizeSecurityGroupEgress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AuthorizeSecurityGroupIngress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CopyImage\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateNetworkInterface\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateSecurityGroup\"\nlevel=warning
msg=\"Action not allowed with tested creds\" action=\"ec2:CreateTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateVolume\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteSecurityGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteSnapshot\"\nlevel=warning msg=\"Action not
allowed with tested creds\" action=\"ec2:DeregisterImage\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeAccountAttributes\"\nlevel=warning msg=\"Action
not allowed with tested creds\" action=\"ec2:DescribeAddresses\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeAvailabilityZones\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeImages\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInstanceAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInstanceCreditSpecifications\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInstances\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeInternetGateways\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeKeyPairs\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeNatGateways\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeNetworkAcls\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeNetworkInterfaces\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribePrefixLists\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeRegions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeRouteTables\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeSecurityGroups\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeSubnets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVolumes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcClassicLink\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcClassicLinkDnsSupport\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcEndpoints\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DescribeVpcs\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifyInstanceAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifyNetworkInterfaceAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ReleaseAddress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:RevokeSecurityGroupEgress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:RevokeSecurityGroupIngress\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:RunInstances\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:TerminateInstances\"\nlevel=warning msg=\"Action not allowed with tested
creds\" action=\"elasticloadbalancing:AddTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:AttachLoadBalancerToSubnets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ConfigureHealthCheck\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateListener\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateLoadBalancerListeners\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:CreateTargetGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeleteLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeregisterInstancesFromLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeregisterTargets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeInstanceHealth\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeListeners\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeLoadBalancerAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeLoadBalancers\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTags\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTargetGroupAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTargetHealth\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ModifyLoadBalancerAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ModifyTargetGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:ModifyTargetGroupAttributes\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:RegisterInstancesWithLoadBalancer\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:RegisterTargets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:SetLoadBalancerPoliciesOfListener\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ChangeResourceRecordSets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ChangeTagsForResource\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:CreateHostedZone\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:DeleteHostedZone\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:GetChange\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:GetHostedZone\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListHostedZones\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListHostedZonesByName\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListResourceRecordSets\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:ListTagsForResource\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"route53:UpdateHostedZoneComment\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:CreateBucket\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:DeleteBucket\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetAccelerateConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketCors\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketLocation\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketLogging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketObjectLockConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketReplication\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketRequestPayment\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketVersioning\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetBucketWebsite\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetEncryptionConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetLifecycleConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetReplicationConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:ListBucket\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutBucketAcl\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutBucketTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutEncryptionConfiguration\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:DeleteObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObjectAcl\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObjectTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:GetObjectVersion\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutObjectAcl\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:PutObjectTagging\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"autoscaling:DescribeAutoScalingGroups\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteNetworkInterface\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteVolume\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DeleteTargetGroup\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"elasticloadbalancing:DescribeTargetGroups\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"s3:DeleteObject\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"tag:GetResources\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AssociateDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AssociateRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:AttachInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateNatGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateRoute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateSubnet\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateVpc\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:CreateVpcEndpoint\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifySubnetAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ModifyVpcAttribute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteDhcpOptions\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteNatGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteRoute\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteSubnet\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteVpc\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DeleteVpcEndpoints\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DetachInternetGateway\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:DisassociateRouteTable\"\nlevel=warning msg=\"Action not allowed with tested creds\" action=\"ec2:ReplaceRouteTableAssociation\"\nlevel=warning msg=\"Tested creds not able to perform all requested actions\"\nlevel=fatal msg=\"failed to fetch Master Machines: failed to fetch dependency of \\\"Master Machines\\\": failed to generate asset \\\"Platform Credentials Check\\\": validate AWS credentials: current credentials insufficient for performing cluster installation\"\n" installID=c4zfdd8z          
time="2020-03-19T23:34:11Z" level=info msg="updating clusterprovision" installID=c4zfdd8z
time="2020-03-19T23:34:11Z" level=fatal msg="runtime error" error="exit status 1"


Expected results:
Installer successfully installs in allowed region

Additional info:
While the installer cannot validate the permissions for the region eu-central-1 we can make API calls to the resources returning "ImplicitDeny" by the policy simulator in the installer in region eu-central-1

$ aws sts get-caller-identity --profile=xxxxxxxxx                                                                                                                                                                   
{
    "Account": "xxxxxxxxx",
    "UserId": "xxxxxxxxx",
    "Arn": "arn:aws:iam::xxxxxxxxx:user/osdCcsAdmin"
}
$ aws --profile=xxxxxxxxx --region=eu-central-1 iam simulate-principal-policy --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyValues=eu-central-1,ContextKeyType=string --policy-source-arn=arn:aws:iam::xxxxxxxxx:user/osdCcsAdmin --action-name ec2:DescribeAvailabilityZones
{
    "EvaluationResults": [
        {
            "EvalDecision": "implicitDeny",
            "MissingContextValues": [],
            "EvalActionName": "ec2:DescribeAvailabilityZones",
            "MatchedStatements": [],
            "EvalResourceName": "*"
        }
    ]
}                                                                                
$ aws ec2 describe-availability-zones --profile=xxxxxxxxx --region=eu-central-1
{
    "AvailabilityZones": [
        {
            "State": "available",
            "RegionName": "eu-central-1",
            "Messages": [],
            "ZoneName": "eu-central-1a"
        },
        {
            "State": "available",
            "RegionName": "eu-central-1",
            "Messages": [],
            "ZoneName": "eu-central-1b"
        },
        {
            "State": "available",
            "RegionName": "eu-central-1",
            "Messages": [],
            "ZoneName": "eu-central-1c"
        }
    ]
}
$ aws ec2 describe-availability-zones --profile=xxxxxxxxx --region=us-east-1                                                                                                                                        
An error occurred (UnauthorizedOperation) when calling the DescribeAvailabilityZones operation: You are not authorized to perform this operation.    

Its possible that this error is because AWS are restricting the endpoint URL access to only eu-central-1. You can see that us-east-1 is denied in the above CLI call but not eu-central-1 indicating that ec2:DescribeAvailabilityZones should be allowed although the installers (CCO) policy simulator is returning an implicit deny. Maybe we need to pass an endpoint URL however IAM is global?

Comment 1 Abhinav Dahiya 2020-03-20 03:13:51 UTC

*** This bug has been marked as a duplicate of bug 1757244 ***


Note You need to log in before you can comment on or make changes to this bug.