Bug 1896918

Hi, Joel. For this issue, if we edit the secret created by cco and only remove .data.credentials field, the expected result is that cco will immeditately update secret (like the logic when deleting the secret, cco will immedately create a new one), or need to wait at most 1h10m until next reconciling? 
in my test, the result is the latter. if it's the expected result, then the bug is fixed.


test payload: registry.svc.ci.openshift.org/ocp/release:4.7.0-0.nightly-2020-11-11-220947

####now cco has add the credentials field in secrets.
$ oc -n openshift-cloud-credential-operator get CredentialsRequest -o json | jq -r '.items[] | select (.spec[].kind=="AWSProviderSpec") | .spec.secretRef'
{
  "name": "ebs-cloud-credentials",
  "namespace": "openshift-cluster-csi-drivers"
}
{
  "name": "cloud-credential-operator-iam-ro-creds",
  "namespace": "openshift-cloud-credential-operator"
}
{
  "name": "cloud-credential-operator-s3-creds",
  "namespace": "openshift-cloud-credential-operator"
}
{
  "name": "installer-cloud-credentials",
  "namespace": "openshift-image-registry"
}
{
  "name": "cloud-credentials",
  "namespace": "openshift-ingress-operator"
}
{
  "name": "aws-cloud-credentials",
  "namespace": "openshift-machine-api"
}

####the below info doesn't contain an aws root credential.########
[lwan@lwan Downloads]$ oc get secret ebs-cloud-credentials -n openshift-cluster-csi-drivers -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVNUUFIQ0pPRTVWTVdHV0Y=",
  "aws_secret_access_key": "d3NIUm10dWhmR3lRTGxzaUp4ZHMvL0oxS290aUxFNEV3bHN3UWhvSg==",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPRTVWTVdHV0YKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gd3NIUm10dWhmR3lRTGxzaUp4ZHMvL0oxS290aUxFNEV3bHN3UWhvSg=="
}
[lwan@lwan Downloads]$ oc get secret cloud-credential-operator-iam-ro-creds -n openshift-cloud-credential-operator -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVNUUFIQ0pPR0xRTUFPMk4=",
  "aws_secret_access_key": "TTBVOHBJblcxZndLZFdoYlcydWlTUk1vMy9ySGNXcm9TdzlvY2didw==",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPR0xRTUFPMk4KYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gTTBVOHBJblcxZndLZFdoYlcydWlTUk1vMy9ySGNXcm9TdzlvY2didw=="
}
[lwan@lwan Downloads]$ oc get secret cloud-credential-operator-s3-creds -n openshift-cloud-credential-operator -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVNUUFIQ0pPQk41QTNRVDM=",
  "aws_secret_access_key": "ZWlDamVIdXNIWHNFNFl0UzhPcHFrdWczTzRSZFV5RXFoREhNK0xibg==",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPQk41QTNRVDMKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gZWlDamVIdXNIWHNFNFl0UzhPcHFrdWczTzRSZFV5RXFoREhNK0xibg=="
}

[lwan@lwan Downloads]$ oc get secret installer-cloud-credentials -n openshift-image-registry -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVNUUFIQ0pPSERQT0laSk4=",
  "aws_secret_access_key": "UXM4SkdRSjFHRXZMWXdoRVFvU1ZRYjlWdGk3Rlc3NlRUN3BsbnNsaA==",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPSERQT0laSk4KYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gUXM4SkdRSjFHRXZMWXdoRVFvU1ZRYjlWdGk3Rlc3NlRUN3BsbnNsaA=="
}
[lwan@lwan Downloads]$ oc get secret cloud-credentials -n openshift-ingress-operator -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVNUUFIQ0pPRExUNVRRRjY=",
  "aws_secret_access_key": "Q2UwRGhzTHBXWXdrd3Uxd0lwNVd0Z0hpSi9xS2hsK0hDRWYrYkRYcw==",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPRExUNVRRRjYKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gQ2UwRGhzTHBXWXdrd3Uxd0lwNVd0Z0hpSi9xS2hsK0hDRWYrYkRYcw=="
}
[lwan@lwan Downloads]$ oc get secret aws-cloud-credentials -n openshift-machine-api -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVNUUFIQ0pPSUdQTEVLTTc=",
  "aws_secret_access_key": "MUlhNUNsR2k2RjA2dDBmQTZuby8rUzd3ZGF0WXBoRzg4NkFxRElHUw==",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPSUdQTEVLTTcKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gMUlhNUNsR2k2RjA2dDBmQTZuby8rUzd3ZGF0WXBoRzg4NkFxRElHUw=="
}

Lin, we do not do any platform-specific checking when the contents of the Secret changes (as we are trying to limit the number of cloud API calls that we would make), so you are correct that we need to wait for up to 1h10m for a full reconcile to happen for the Secret to have any missing fields restored.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633