Bug 1770344

Summary: Access key is deleted via AWS, credentials secret does not get recreated.
Product: OpenShift Container Platform Reporter: Ryan Howe <rhowe>
Component: Cloud Credential OperatorAssignee: Joel Diaz <jdiaz>
Status: CLOSED CURRENTRELEASE QA Contact: Xiaoli Tian <xtian>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.1.z   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-05 16:00:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ryan Howe 2019-11-08 19:24:04 UTC
Description of problem:

If aws access key is removed manually then the secret with the creditials the cloud-credentials operator does not recreate the secret or a new access key.  

Version-Release number of selected component (if applicable):
4.x

How reproducible:
100%

Steps to Reproduce:
1. 
 # aws iam  --region ${region} delete-access-key --user-name  mycluster-z8grw-openshift-image-registry-s4jlz --access-key-id ABCD1234

2. delete secret 

 # oc delete secret -n openshift-image-registry installer-cloud-credentials

Actual results:

 Secret never gets recreated or updated (if not deleted). New access-key never gets created. Credentials operator never logs anything regarding an issue. 

Expected results:

 New access key gets created and secret is recreated or updated.

Comment 1 Devan Goodwin 2019-11-08 19:30:30 UTC
Can you clarify how long you waited after deleting the key in AWS?

I thought we repaired missing credentials but because the operator has no event indicating anything happened, we only check periodically when we reconcile again.

Comment 2 Ryan Howe 2019-11-08 19:41:01 UTC
I waited a 2-3 hours. 
To fix this I updated the credentialsrequests.cloudcredential.openshift.io CR adding an action. 

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.21    True        False         8d      Cluster version is 4.1.21

Comment 4 Joel Diaz 2019-11-08 21:04:49 UTC
This was recently addressed in https://github.com/openshift/cloud-credential-operator/pull/131

Well the part about the secret being removed. Removing the access key from AWS directly is not something cloud-cred-operator would ever know about.