Bug 1838718

Summary:	the read-only CR cannot be reconciled after removing the root cloud creds
Product:	OpenShift Container Platform	Reporter:	Joel Diaz <jdiaz>
Component:	Cloud Credential Operator	Assignee:	Joel Diaz <jdiaz>
Status:	CLOSED ERRATA	QA Contact:	wang lin <lwan>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	knewcome, lwan, ngreen, sandeepredhat
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The cloud-credential-operator had special-case handling for the CCO's own read-only CredentialsRequest CR that required the existence of the cloud root creds. Consequence: If the cloud root creds are missing, CCO would be unable to reconcile CCO's own read-only CredentialsRequest CR. Fix: Use the read-only CredentialsRequest credentials to validate the read-only CredentialsRequest just like every other CredentialsRequest. Result: Removing the cloud root creds doesn't put CCO into a degraded state.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:40:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1838810

Description Joel Diaz 2020-05-21 16:17:52 UTC

Description of problem:
After removing the root cloud credentials secret (eg kube-system/aws-creds), when the cloud-credential-operator goes to re-reconcile the read-only creds (cloud-credential-operator-iam-ro), it fails b/c it cannot find the root credentials (which of course were previously deleted).

Version-Release number of selected component (if applicable):

4.4.4


How reproducible: 100%


Steps to Reproduce:
1. Install OpenShift 4.4.4 on AWS
2. Delete the secret kube-system/aws-creds
3. Wait for CCO to reconcile the read-only CR

Actual results:

CCO will fail to reconcile the read-only credentials.


Expected results:

CCO doesn't enter an error state (where not all credentials have been successfully reconciled) when the root creds are removed.


Additional info:

Comment 3 wang lin 2020-05-25 11:43:50 UTC

The bug has been fixed.
The test payload is 4.5.0-0.nightly-2020-05-24-223848

The result is as below:
1.Install OpenShift on AWS
2.wait for cco reconcile (it is about 6 hours)
3.The cco successes to reconcile the read-only credentials and don't Degraded


time="2020-05-25T09:18:47Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="target secret exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="running sync" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="Loading infrastructure name: lwanjk525-1-c9m26" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="target secret exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="found access key ID in target secret" accessKeyID=XXXX actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="loading AWS credentials from secret" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2020-05-25T09:18:47Z" level=debug msg="creating read AWS client" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2020-05-25T09:18:47Z" level=debug msg="loading cluster version to read clusterID" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="found cluster ID" actuator=aws clusterID=ed845e25-94f6-41c0-8fec-e49f4483d0ff cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro


status:
  conditions:
  - lastTransitionTime: "2020-05-25T03:11:23Z"
    message: No credentials requests reporting errors.
    reason: NoCredentialsFailing
    status: "False"
    type: Degraded
  - lastTransitionTime: "2020-05-25T03:21:05Z"
    message: 4 of 4 credentials requests provisioned and reconciled.
    reason: ReconcilingComplete
    status: "False"
    type: Progressing
  - lastTransitionTime: "2020-05-25T03:11:23Z"
    status: "True"
    type: Available
  - lastTransitionTime: "2020-05-25T03:11:23Z"
    status: "True"
    type: Upgradeable
  extension: null
  relatedObjects:

Comment 6 errata-xmlrpc 2020-07-13 17:40:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409