Bug 1970062

Summary: ccoctl does not work with STS authentication
Product: OpenShift Container Platform Reporter: Dale Bewley <dbewley>
Component: Cloud Credential OperatorAssignee: Joel Diaz <jdiaz>
Status: CLOSED ERRATA QA Contact: wang lin <lwan>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.8CC: jdiaz, lwan
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:12:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dale Bewley 2021-06-09 17:35:38 UTC
Description of problem:

Local AWS authentication using STS is not supported by ccoctl because it is not loading the aws config.


Version-Release number of selected component (if applicable):


How reproducible:

 every time

Steps to Reproduce:

 
$ cat ~/.aws/config
[profile dev]
credential_process = custom_exe -u dale_bewley -a <account_number> -r developer_role
region = us-west-2

$ env | grep AWS_
AWS_PROFILE=dev

$ aws sts get-caller-identity
{
    "UserId": "xxx",
    "Account": "<account_number>",
    "Arn": "arn:aws:sts::<account_number>:assumed-role/developer_role/xxx"
}


Rebuild ccoctl with debugging as ccoctl-debug per diff:

diff --git a/pkg/cmd/provisioning/aws/create_identity_provider.go b/pkg/cmd/provisioning/aws/create_identity_provider.go
index f0c1b701..865b3adf 100644
--- a/pkg/cmd/provisioning/aws/create_identity_provider.go
+++ b/pkg/cmd/provisioning/aws/create_identity_provider.go
@@ -460,6 +460,7 @@ spec:
 func createIdentityProviderCmd(cmd *cobra.Command, args []string) {
        cfg := &awssdk.Config{
                Region: awssdk.String(CreateIdentityProviderOpts.Region),
+               CredentialsChainVerboseErrors: awssdk.Bool(true),
        }

        s, err := session.NewSession(cfg)



Actual results:

$ ccoctl-debug aws create-identity-provider \
    --name=debug-oidc \
    --region=us-west-2 \
    --public-key-file=cco/serviceaccount-signer.public
2021/06/09 09:31:54 failed to create a bucket to store OpenID Connect configuration: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, apple_dev.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: no route to host


Expected results:

$ AWS_SDK_LOAD_CONFIG=true ccoctl-debug aws create-identity-provider \
    --name=debug-oidc \
    --region=us-west-2 \
    --public-key-file=cco/serviceaccount-signer.public
2021/06/09 10:15:09 Bucket debug-oidc-oidc created
2021/06/09 10:15:09 failed to upload discovery document in the S3 bucket debug-oidc-oidc: AccessDenied: Access Denied
        status code: 403, request id: P8ZK2FK7YN6E7JAD, host id: xxx=

Additional info:

* AWS CLI works fine by default.
* ccoctl requires AWS_SDK_LOAD_CONFIG=true variable to utilize STS authentication.
* Requirement for this variable is not in documentation https://docs.openshift.com/container-platform/4.8/authentication/managing_cloud_provider_credentials/cco-mode-sts.html

Comment 1 Dale Bewley 2021-06-09 19:54:57 UTC
PR352 resolved the issue for me.

Comment 3 wang lin 2021-06-10 05:53:11 UTC
joel, The deletion part haven't fixed yet.

$ ./ccoctl aws delete --name lwan-test-2 --region us-east-2
2021/06/10 13:46:14 failed to fetch list of Identity Provider objects in the bucket lwan-test-2-oidc: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2021/06/10 13:46:17 failed to fetch tags of the bucket lwan-test-2-oidc: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2021/06/10 13:46:21 failed to fetch a list of IAM roles: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2021/06/10 13:46:24 failed to fetch list of Identity Providers: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Comment 4 Joel Diaz 2021-06-10 12:08:29 UTC
@lwan yes, i missed converting the client setup for the delete path. Fixed in https://github.com/openshift/cloud-credential-operator/pull/353

Comment 6 wang lin 2021-06-11 03:19:23 UTC
Sorry joel, my bad, we still miss one for create-all path. this time, we should convert all(create-iam-roles,create_identity_provider,deleteand create-all)

Comment 7 Joel Diaz 2021-06-11 11:30:31 UTC
Hopefully the last PR for this issue...
https://github.com/openshift/cloud-credential-operator/pull/354

Comment 9 wang lin 2021-06-15 03:16:07 UTC
The issue has fixed on 4.8.0-0.nightly-2021-06-14-145150

Now all subcommands(create-iam-roles,create_identity_provider,delete and create-all) can work in AWS authentication using STS.

Comment 12 errata-xmlrpc 2021-07-27 23:12:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438