Bug 1750338 - Permission validation optional in OpenShift AWS installer incorrectly denies permission
Summary: Permission validation optional in OpenShift AWS installer incorrectly denies ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.4.0
Assignee: Joel Diaz
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks: 1757244 1832640
TreeView+ depends on / blocked
 
Reported: 2019-09-09 11:00 UTC by David Caldwell
Modified: 2020-05-21 09:35 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Use Region info when simulating permissions. Reason: Some permissions setups allow/deny permissions based on what region the API call is being made against. Result: Include region information during permissions simulation to more accurately determine permissions.
Clone Of:
: 1757244 1803221 1832640 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:13:32 UTC
Target Upstream Version:


Attachments (Terms of Use)
skip aws checks patch (717 bytes, patch)
2020-01-30 15:26 UTC, Hugo Cisneiros (Eitch)
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 124 None closed Bug 1750338: improve permissions simulation by adding region info 2020-07-27 02:36:08 UTC
Github openshift installer pull 3056 None closed Bug 1750338: use region info when simulating permissions 2020-07-27 02:36:08 UTC
Red Hat Knowledge Base (Solution) 4775281 None None None 2020-01-29 18:49:20 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:14:08 UTC

Description David Caldwell 2019-09-09 11:00:16 UTC
Description of problem:

The customer has issues with the permissions checks that the openshift installer runs during the installation process. 


Version-Release number of the following components:

4.1


How reproducible:

Here is what the customer says:

The Openshift installer validates whether the user performing the installation has enough permissions to do the installation.To do so, it uses the IAM Policy Simulator API.

However, when using AWS Organizations and Service control policies (SCP), the Policy Simulator may deny a permission when in reality the user is authorized.
The following statement denies most actions outside of the us-east-1 region.

{
"Effect": "Deny",
"Action": [
"ec2:Create*",
"ec2:Run*",
"eks:Create*",
"rds:Create*",
"es:Create*",
"lambda:Create*"
],
"Resource": [
"*"
],
"Condition": {
"StringNotEquals":

Unknown macro:

{ "aws}
}
}

When attempting to create a cluster with access keys with administrative access in the account, the validation fails with the ec2 permissions

WARNING Action not allowed with tested creds action="ec2:CreateDhcpOptions"
WARNING Action not allowed with tested creds action="ec2:CreateInternetGateway"
WARNING Action not allowed with tested creds action="ec2:CreateNatGateway"
WARNING Action not allowed with tested creds action="ec2:CreateRoute"
WARNING Action not allowed with tested creds action="ec2:CreateRouteTable"
WARNING Action not allowed with tested creds action="ec2:CreateSecurityGroup"
WARNING Action not allowed with tested creds action="ec2:CreateSubnet"
WARNING Action not allowed with tested creds action="ec2:CreateTags"
WARNING Action not allowed with tested creds action="ec2:CreateVpc"
WARNING Action not allowed with tested creds action="ec2:CreateVpcEndpoint"
WARNING Action not allowed with tested creds action="ec2:CreateVolume"
WARNING Action not allowed with tested creds action="ec2:RunInstances"
WARNING Tested creds not able to perform all requested actions
FATAL failed to fetch Terraform Variables: failed to fetch dependency of "Terraform Variables": failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation

In this case, this happens because the simulation doesn’t consider the region to perform the simulation.


Additional info:

The customer suggests these checks should be optional.

Comment 3 Joel Diaz 2019-10-02 01:09:58 UTC
I don't have an account where I can enable/manipulate AWS SCP at this moment, but I would like to understand this better. And I'll likely have to reach out to AWS to verify my findings.

I've set up a new IAM user, and assigned group membership that allows ec2:DescribeInstances (with no restrictions).

I then set up a Permissions Boundary and attached it to the user that denies ec2:DescribeInstances if the region is not us-east-1 (this was as close as I could get to simulate the AWS SCP configuration).

When I run the policy simulator, it tells me that I can ec2:DescribeInstances in us-east-2, but when I go to actually perform the action it is denied:

jdiaz@minigoomba bin (cco-bootstrap %=)]$ aws iam simulate-principal-policy --action-names '["ec2:DescribeInstances"]' --policy-source-arn "arn:aws:iam::125931421481:user/jdiazregiontest" --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyType=string,ContextKeyValues=us-east-2 | jq -r '.EvaluationResults[].EvalDecision'
allowed
[jdiaz@minigoomba bin (cco-bootstrap %=)]$ aws ec2 describe-instances --region=us-east-2

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.
[jdiaz@minigoomba bin (cco-bootstrap %=)]$ 




Now when I create an inline policy for the same user with the same denial of ec2:DescribeInstances if region is not us-east-1, then the IAM policy simulation does work.

[jdiaz@minigoomba bin (cco-bootstrap %=)]$ aws iam simulate-principal-policy --action-names '["ec2:DescribeInstances"]' --policy-source-arn "arn:aws:iam::125931421481:user/jdiazregiontest" --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyType=string,ContextKeyValues=us-east-2 | jq -r '.EvaluationResults[].EvalDecision'
explicitDeny


While it appears that the IAM SimulatePolicy API call does allow us to pass in a region when doing simulations, it does not appear to sufficiently simulate things when things like Permissions Boundaries (which are similar to AWS SCP as far as I can tell) are in effect.

Comment 4 Joel Diaz 2019-10-02 22:12:12 UTC
When setting things up with Permissions Boundaries (in an effort to more closely mimic the AWS SCP setup as I understand it) I've only been able to set up an environment where I get false passing, but not false failing.

The only way I've been able to get false failing conditions is when attaching policy denials/conditions directly attached to an AWS user. But this is far removed from AWS SCP/Permissions Boundaries.

Can you provide the details on how the exact permissions are set up in the scenario originally reported in the BZ?

Comment 8 Hugo Cisneiros (Eitch) 2020-01-28 18:03:11 UTC
Also got this problem with 4.3.0 installer. Trying to install a cluster in us-east-2 with this policy that restricts all actions to us-east-2 region only:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyAllRegions",
            "Effect": "Deny",
            "NotAction": [
                "iam:*",
                "organizations:*",
                "route53:*",
                "budgets:*",
                "waf:*",
                "cloudfront:*",
                "globalaccelerator:*",
                "importexport:*",
                "support:*",
                "config:*",
                "guardduty:*",
                "devicefarm:*",
                "directconnect:*"
            ],
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:RequestedRegion": [
                        "us-east-2"
                    ]
                }
            }
        }
    ]
}

Using IPI installer results in:

DEBUG       Generating Platform Credentials Check... 
WARNING Action not allowed with tested creds          action="ec2:AllocateAddress"
[...]
WARNING Action not allowed with tested creds          action="ec2:DisassociateRouteTable"
WARNING Action not allowed with tested creds          action="ec2:ReplaceRouteTableAssociation"
WARNING Tested creds not able to perform all requested actions 
FATAL failed to fetch Terraform Variables: failed to fetch dependency of "Terraform Variables": failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation

Using the same credentials, I manually tried many of the actions and they worked. Example:

$ aws ec2 describe-internet-gateways --region us-east-2
{
    "InternetGateways": [
        {
[...]

Any workarounds for this?

Comment 25 Hugo Cisneiros (Eitch) 2020-01-30 15:26:53 UTC
Created attachment 1656528 [details]
skip aws checks patch

I got a workaround on this by patching the branch "release-4.3" from the installer and forcing the installer to continue even if it detects that you don't have the proper AWS permissions.

Please note that this is completely unofficial  and not supported, it's just a proof of concept that the installation can happen even if the installer complains about lack of permissions (IMHO, wrongly).

$ git clone <openshift-installer-url>
$ cd installer
$ git checkout release-4.3
$ patch -p1 < /tmp/openshift-installer-43-skip-aws-checks.patch
$ hack/build.sh
$ sudo cp bin/openshift-installer /usr/local/bin/openshift-installer-custom

Checking version:

$ openshift-install-custom version
openshift-install-custom unreleased-master-2206-ge8a128e459722896c9c099a7710aa60b30bf0cba-dirty
built from commit e8a128e459722896c9c099a7710aa60b30bf0cba
release image registry.svc.ci.openshift.org/origin/release:4.3

*Important*: the custom built installer by default uses OKD images, which are not compatible with OpenShift, so before installing, you should also define the proper release image. For 4.3:

$ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release@sha256:3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d

Then you can install:

$ openshift-install-custom create cluster --dir=ocp4-cluster/ --log-level=debug

Comment 26 Scott Dodson 2020-02-04 15:16:48 UTC
This bug is only targeted at tracking the fact that we don't properly scope our permissions checks to the appropriate region. Any additional issues outside the scope of that change should be filed as an independent bug.

Comment 27 Dwayne 2020-02-04 19:44:16 UTC
I have set the customer escalation flag => on, based on Customer escalation EN-26619, case 02570853.
The case owner is Novonil Choudhuri.

Thank you,
Dwayne Hellard
Escalation Management Team
Red Hat - Customer Experience & Engagement

Comment 30 Johnny Liu 2020-02-11 03:49:36 UTC
Reproduce steps:
# openshift-install version
openshift-install v4.1.34-202002031224-dirty
built from commit 1fd119d156f9dca16d6eaa9c888c512071c83c34
release image registry.svc.ci.openshift.org/ocp/release@sha256:570628002b6229010bac119be2ae0f76a431cea392a64bbc74efb0773f6da57b

create a policy using the following json, that means most actions not in us-east-2 region is denied, everything in us-east-2 region is allowed for an IAM user with this policy attached.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        },
        {
            "Effect": "Deny",
            "Action": [
                "ec2:Create*",
                "ec2:Run*",
                "eks:Create*",
                "rds:Create*",
                "es:Create*",
                "lambda:Create*"
            ],
            "Resource": [
                "*"
            ],
            "Condition": {
                "StringNotEquals": {
                    "aws:RequestedRegion": "us-east-2"
                }
            }
        },
        {
            "Effect": "Deny",
            "Action": "s3:CreateBucket",
            "Resource": "arn:aws:s3:::*",
            "Condition": {
                "StringLike": {
                    "s3:LocationConstraint": "*"
                }
            }
        }
    ]
}

Trying to install a cluster in us-east-2 with this policy that restricts all actions to us-east-2 region only:
# openshift-install create ignition-configs --dir demo3
INFO Consuming "Install Config" from target directory 
WARNING Action not allowed with tested creds          action="ec2:CreateDhcpOptions"
WARNING Action not allowed with tested creds          action="ec2:CreateInternetGateway"
WARNING Action not allowed with tested creds          action="ec2:CreateNatGateway"
WARNING Action not allowed with tested creds          action="ec2:CreateRoute"
WARNING Action not allowed with tested creds          action="ec2:CreateRouteTable"
WARNING Action not allowed with tested creds          action="ec2:CreateSecurityGroup"
WARNING Action not allowed with tested creds          action="ec2:CreateSubnet"
WARNING Action not allowed with tested creds          action="ec2:CreateTags"
WARNING Action not allowed with tested creds          action="ec2:CreateVpc"
WARNING Action not allowed with tested creds          action="ec2:CreateVpcEndpoint"
WARNING Action not allowed with tested creds          action="ec2:CreateVolume"
WARNING Action not allowed with tested creds          action="ec2:RunInstances"
WARNING Tested creds not able to perform all requested actions 
FATAL failed to fetch Bootstrap Ignition Config: failed to fetch dependency of "Bootstrap Ignition Config": failed to fetch dependency of "Master Machines": failed to generate asset "Platform Credentials Check": validate AWS credentials: current credentials insufficient for performing cluster installation 


Return a false negative result.

The same IAM user can launch an instance successfully in us-east-2 region.
# aws ec2 run-instances --image-id ami-075b7fe10944d8748 --instance-type t2.medium
{
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            }, 
            "PublicDnsName": "", 
            "StateReason": {
                "Message": "pending", 
                "Code": "pending"
            }, 
            "State": {
                "Code": 0, 
                "Name": "pending"
            }, 
            "EbsOptimized": false, 
            "LaunchTime": "2020-02-11T03:39:02.000Z", 
            "PrivateIpAddress": "172.31.44.14", 
            "ProductCodes": [], 
            "VpcId": "vpc-0171369946eada62a", 
            "CpuOptions": {
                "CoreCount": 2, 
                "ThreadsPerCore": 1
            }, 
            "StateTransitionReason": "", 
            "InstanceId": "i-0e87e067de7f1e62c", 
            "ImageId": "ami-075b7fe10944d8748", 
            "PrivateDnsName": "ip-172-31-44-14.us-east-2.compute.internal", 
            "SecurityGroups": [
                {
                    "GroupName": "default", 
                    "GroupId": "sg-0c3ae688493a18161"
                }
            ], 
            "ClientToken": "", 
            "SubnetId": "subnet-0e6f2f221b0ea5381", 
            "InstanceType": "t2.medium", 
            "CapacityReservationSpecification": {
                "CapacityReservationPreference": "open"
            }, 
            "NetworkInterfaces": [
                {
                    "Status": "in-use", 
                    "MacAddress": "0a:32:9c:ae:40:5e", 
                    "SourceDestCheck": true, 
                    "VpcId": "vpc-0171369946eada62a", 
                    "Description": "", 
                    "NetworkInterfaceId": "eni-0bbaab9ae793dea2f", 
                    "PrivateIpAddresses": [
                        {
                            "PrivateDnsName": "ip-172-31-44-14.us-east-2.compute.internal", 
                            "Primary": true, 
                            "PrivateIpAddress": "172.31.44.14"
                        }
                    ], 
                    "PrivateDnsName": "ip-172-31-44-14.us-east-2.compute.internal", 
                    "InterfaceType": "interface", 
                    "Attachment": {
                        "Status": "attaching", 
                        "DeviceIndex": 0, 
                        "DeleteOnTermination": true, 
                        "AttachmentId": "eni-attach-0e01a958bc337408f", 
                        "AttachTime": "2020-02-11T03:39:02.000Z"
                    }, 
                    "Groups": [
                        {
                            "GroupName": "default", 
                            "GroupId": "sg-0c3ae688493a18161"
                        }
                    ], 
                    "Ipv6Addresses": [], 
                    "OwnerId": "301721915996", 
                    "SubnetId": "subnet-0e6f2f221b0ea5381", 
                    "PrivateIpAddress": "172.31.44.14"
                }
            ], 
            "SourceDestCheck": true, 
            "Placement": {
                "Tenancy": "default", 
                "GroupName": "", 
                "AvailabilityZone": "us-east-2c"
            }, 
            "Hypervisor": "xen", 
            "BlockDeviceMappings": [], 
            "Architecture": "x86_64", 
            "RootDeviceType": "ebs", 
            "RootDeviceName": "/dev/sda1", 
            "VirtualizationType": "hvm", 
            "AmiLaunchIndex": 0
        }
    ], 
    "ReservationId": "r-08f6b7e33dc0aa0dd", 
    "Groups": [], 
    "OwnerId": "301721915996"
}


Extract installer from 4.4.0-0.nightly-2020-02-09-220310, run verification, passed.
# openshift-install version
openshift-install 4.4.0-0.nightly-2020-02-09-220310
built from commit 7289dd40ec78df4aa18e98fc4fbcac88d3d5a598
release image registry.svc.ci.openshift.org/ocp/release@sha256:edcdbd54d65f244182d11340a71682b959bb31e19bbc0c359ce53dc6abd728dc

# cp ipi_template/install-config.yaml.aws demo3/install-config.yaml

# openshift-install create ignition-configs --dir demo3
INFO Consuming Install Config from target directory 
INFO Credentials loaded from the "default" profile in file "/root/.aws/credentials"

Comment 33 errata-xmlrpc 2020-05-04 11:13:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.