Bug 1970062 - ccoctl does not work with STS authentication
Summary: ccoctl does not work with STS authentication
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.8.0
Assignee: Joel Diaz
QA Contact: wang lin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-09 17:35 UTC by Dale Bewley
Modified: 2021-07-27 23:12 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:12:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 352 0 None open Bug 1970062: enable shared config files for AWS SDK 2021-06-09 18:34:29 UTC
Github openshift cloud-credential-operator pull 353 0 None open Bug 1970062: delete client should also use shared session function 2021-06-10 12:06:47 UTC
Github openshift cloud-credential-operator pull 354 0 None open Bug 1970062: use shared session setup in ccoctl create-all 2021-06-11 11:29:10 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:12:42 UTC

Description Dale Bewley 2021-06-09 17:35:38 UTC
Description of problem:

Local AWS authentication using STS is not supported by ccoctl because it is not loading the aws config.


Version-Release number of selected component (if applicable):


How reproducible:

 every time

Steps to Reproduce:

 
$ cat ~/.aws/config
[profile dev]
credential_process = custom_exe -u dale_bewley -a <account_number> -r developer_role
region = us-west-2

$ env | grep AWS_
AWS_PROFILE=dev

$ aws sts get-caller-identity
{
    "UserId": "xxx",
    "Account": "<account_number>",
    "Arn": "arn:aws:sts::<account_number>:assumed-role/developer_role/xxx"
}


Rebuild ccoctl with debugging as ccoctl-debug per diff:

diff --git a/pkg/cmd/provisioning/aws/create_identity_provider.go b/pkg/cmd/provisioning/aws/create_identity_provider.go
index f0c1b701..865b3adf 100644
--- a/pkg/cmd/provisioning/aws/create_identity_provider.go
+++ b/pkg/cmd/provisioning/aws/create_identity_provider.go
@@ -460,6 +460,7 @@ spec:
 func createIdentityProviderCmd(cmd *cobra.Command, args []string) {
        cfg := &awssdk.Config{
                Region: awssdk.String(CreateIdentityProviderOpts.Region),
+               CredentialsChainVerboseErrors: awssdk.Bool(true),
        }

        s, err := session.NewSession(cfg)



Actual results:

$ ccoctl-debug aws create-identity-provider \
    --name=debug-oidc \
    --region=us-west-2 \
    --public-key-file=cco/serviceaccount-signer.public
2021/06/09 09:31:54 failed to create a bucket to store OpenID Connect configuration: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, apple_dev.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: no route to host


Expected results:

$ AWS_SDK_LOAD_CONFIG=true ccoctl-debug aws create-identity-provider \
    --name=debug-oidc \
    --region=us-west-2 \
    --public-key-file=cco/serviceaccount-signer.public
2021/06/09 10:15:09 Bucket debug-oidc-oidc created
2021/06/09 10:15:09 failed to upload discovery document in the S3 bucket debug-oidc-oidc: AccessDenied: Access Denied
        status code: 403, request id: P8ZK2FK7YN6E7JAD, host id: xxx=

Additional info:

* AWS CLI works fine by default.
* ccoctl requires AWS_SDK_LOAD_CONFIG=true variable to utilize STS authentication.
* Requirement for this variable is not in documentation https://docs.openshift.com/container-platform/4.8/authentication/managing_cloud_provider_credentials/cco-mode-sts.html

Comment 1 Dale Bewley 2021-06-09 19:54:57 UTC
PR352 resolved the issue for me.

Comment 3 wang lin 2021-06-10 05:53:11 UTC
joel, The deletion part haven't fixed yet.

$ ./ccoctl aws delete --name lwan-test-2 --region us-east-2
2021/06/10 13:46:14 failed to fetch list of Identity Provider objects in the bucket lwan-test-2-oidc: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2021/06/10 13:46:17 failed to fetch tags of the bucket lwan-test-2-oidc: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2021/06/10 13:46:21 failed to fetch a list of IAM roles: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2021/06/10 13:46:24 failed to fetch list of Identity Providers: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Comment 4 Joel Diaz 2021-06-10 12:08:29 UTC
@lwan yes, i missed converting the client setup for the delete path. Fixed in https://github.com/openshift/cloud-credential-operator/pull/353

Comment 6 wang lin 2021-06-11 03:19:23 UTC
Sorry joel, my bad, we still miss one for create-all path. this time, we should convert all(create-iam-roles,create_identity_provider,deleteand create-all)

Comment 7 Joel Diaz 2021-06-11 11:30:31 UTC
Hopefully the last PR for this issue...
https://github.com/openshift/cloud-credential-operator/pull/354

Comment 9 wang lin 2021-06-15 03:16:07 UTC
The issue has fixed on 4.8.0-0.nightly-2021-06-14-145150

Now all subcommands(create-iam-roles,create_identity_provider,delete and create-all) can work in AWS authentication using STS.

Comment 12 errata-xmlrpc 2021-07-27 23:12:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.