Bug 1838718 - the read-only CR cannot be reconciled after removing the root cloud creds
Summary: the read-only CR cannot be reconciled after removing the root cloud creds
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.0
Assignee: Joel Diaz
QA Contact: wang lin
URL:
Whiteboard:
Depends On:
Blocks: 1838810
TreeView+ depends on / blocked
 
Reported: 2020-05-21 16:17 UTC by Joel Diaz
Modified: 2020-07-13 17:41 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The cloud-credential-operator had special-case handling for the CCO's own read-only CredentialsRequest CR that required the existence of the cloud root creds. Consequence: If the cloud root creds are missing, CCO would be unable to reconcile CCO's own read-only CredentialsRequest CR. Fix: Use the read-only CredentialsRequest credentials to validate the read-only CredentialsRequest just like every other CredentialsRequest. Result: Removing the cloud root creds doesn't put CCO into a degraded state.
Clone Of:
Environment:
Last Closed: 2020-07-13 17:40:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 161 0 None closed Bug 1838718: do not special-case handling read-only credentialsRequest 2021-01-06 02:15:36 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:41:14 UTC

Description Joel Diaz 2020-05-21 16:17:52 UTC
Description of problem:
After removing the root cloud credentials secret (eg kube-system/aws-creds), when the cloud-credential-operator goes to re-reconcile the read-only creds (cloud-credential-operator-iam-ro), it fails b/c it cannot find the root credentials (which of course were previously deleted).

Version-Release number of selected component (if applicable):

4.4.4


How reproducible: 100%


Steps to Reproduce:
1. Install OpenShift 4.4.4 on AWS
2. Delete the secret kube-system/aws-creds
3. Wait for CCO to reconcile the read-only CR

Actual results:

CCO will fail to reconcile the read-only credentials.


Expected results:

CCO doesn't enter an error state (where not all credentials have been successfully reconciled) when the root creds are removed.


Additional info:

Comment 3 wang lin 2020-05-25 11:43:50 UTC
The bug has been fixed.
The test payload is 4.5.0-0.nightly-2020-05-24-223848

The result is as below:
1.Install OpenShift on AWS
2.wait for cco reconcile (it is about 6 hours)
3.The cco successes to reconcile the read-only credentials and don't Degraded


time="2020-05-25T09:18:47Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="target secret exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="running sync" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="Loading infrastructure name: lwanjk525-1-c9m26" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="target secret exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="found access key ID in target secret" accessKeyID=XXXX actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="loading AWS credentials from secret" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2020-05-25T09:18:47Z" level=debug msg="creating read AWS client" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2020-05-25T09:18:47Z" level=debug msg="loading cluster version to read clusterID" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2020-05-25T09:18:47Z" level=debug msg="found cluster ID" actuator=aws clusterID=ed845e25-94f6-41c0-8fec-e49f4483d0ff cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro


status:
  conditions:
  - lastTransitionTime: "2020-05-25T03:11:23Z"
    message: No credentials requests reporting errors.
    reason: NoCredentialsFailing
    status: "False"
    type: Degraded
  - lastTransitionTime: "2020-05-25T03:21:05Z"
    message: 4 of 4 credentials requests provisioned and reconciled.
    reason: ReconcilingComplete
    status: "False"
    type: Progressing
  - lastTransitionTime: "2020-05-25T03:11:23Z"
    status: "True"
    type: Available
  - lastTransitionTime: "2020-05-25T03:11:23Z"
    status: "True"
    type: Upgradeable
  extension: null
  relatedObjects:

Comment 6 errata-xmlrpc 2020-07-13 17:40:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.