Bug 1883412 - cloud-credential upgradeable status didn't immediately respond when we remove or recreate the root creds in mint/passthrough mode
Summary: cloud-credential upgradeable status didn't immediately respond when we remov...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.z
Assignee: Devan Goodwin
QA Contact: wang lin
URL:
Whiteboard:
Depends On: 1883644
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-29 06:27 UTC by wang lin
Modified: 2020-12-01 10:49 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1883644 (view as bug list)
Environment:
Last Closed: 2020-12-01 10:48:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 253 0 None closed Bug 1883412: Update upgradeable condition on admin cred changes. 2020-11-30 14:17:15 UTC
Red Hat Product Errata RHSA-2020:5194 0 None None None 2020-12-01 10:49:34 UTC

Description wang lin 2020-09-29 06:27:10 UTC
Description of problem:
A backport PR to 4.5 involves a new issue. This PR adds a function that if cluster is in mint/passthrough mode, and at the same time root creds being removed, it will set cloud-credential upgradeable=false. But the issue now is the state won't change immediately when we remove or re-add the root creds, it maybe need to wait for several hours.
The same issue occurred when re-creating root creds while upgradeable=false, the state won't change immediately to upgradeable=true.


Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-09-28-124031

How reproducible:
always

Steps to Reproduce:
1. Launch an IPI with latest 4.5
2. Check the root creds is in mint mode
$oc get secret aws-creds -n kube-system  -o json | jq .metadata.annotations
{
 "cloudcredential.openshift.io/mode": "mint"
}
3.Check the cloud-credential upgradeable=true
$oc get co cloud-credential -o json | jq -r ".status.conditions"
[
{
    "lastTransitionTime": "2020-09-28T04:24:19Z",
    "status": "True",
    "type": "Upgradeable"
  }
]
4. Remove the root creds ,check the cloud-credential upgradeable status again,the status is still upgradeable=true
5. The same situation when we re-add root creds, check the upgradeable=false now, and recreate root creds, verify the cloud-credential condiftion is still upgradeable=false

Actual results:
1. Launch an IPI with latest 4.5
2. Check the root creds is in mint mode
$oc get secret aws-creds -n kube-system  -o json | jq .metadata.annotations
{
 "cloudcredential.openshift.io/mode": "mint"
}
3.Check the cloud-credential upgradeable=true
$oc get co cloud-credential -o json | jq -r ".status.conditions"
[
{
    "lastTransitionTime": "2020-09-28T04:24:19Z",
    "status": "True",
    "type": "Upgradeable"
  }
]
4. Remove the root creds ,check the cloud-credential upgradeable status again,the status is still upgradeable=true
5. The same situation when we re-add root creds, check the upgradeable=false now, and recreate root creds, verify the cloud-credential condiftion is still upgradeable=false

Expected results:
It can immediately update the upgradeable status. 

Additional info:
[1] in version 4.6, it works well
[2] related bug https://bugzilla.redhat.com/show_bug.cgi?id=1871713

Comment 1 wang lin 2020-09-29 06:34:33 UTC
Sorry for that I put a wrong description info in "Actual results" above

Actual results:
Cloud-credential doesn’t immediately update the upgradeable status when we remove or re-add the root creds until we force a cco reconcile via adding an annotations in CloudCredential object or deleting the cco pod

Comment 5 wang lin 2020-11-23 03:47:54 UTC
The issue has fixed.
test payload:4.5.0-0.nightly-2020-11-22-160319


1. Run "oc edit cloudcredential cluster" and modify .spec.credentialsMode to "Mint" or "Passthrough"
2. remove root creds
   oc delete secret aws-creds -n kube-system
3. check co cloud-credential status,  Upgradeable=False
   $ oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2020-11-23T02:31:48Z",
    "message": "No credentials requests reporting errors.",
    "reason": "NoCredentialsFailing",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2020-11-23T02:50:15Z",
    "message": "4 of 4 credentials requests provisioned and reconciled.",
    "reason": "ReconcilingComplete",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2020-11-23T02:31:48Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2020-11-23T03:35:37Z",
    "message": "Parent credential secret must be restored prior to upgrade: kube-system/aws-creds",
    "reason": "MissingRootCredential",
    "status": "False",
    "type": "Upgradeable"
  }
]
4. re-create root credential
5. check co cloud-credential status, Upgradeable=True
$ oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2020-11-23T02:31:48Z",
    "message": "No credentials requests reporting errors.",
    "reason": "NoCredentialsFailing",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2020-11-23T02:50:15Z",
    "message": "4 of 4 credentials requests provisioned and reconciled.",
    "reason": "ReconcilingComplete",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2020-11-23T02:31:48Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2020-11-23T03:36:22Z",
    "status": "True",
    "type": "Upgradeable"
  }
]

Comment 8 errata-xmlrpc 2020-12-01 10:48:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.5.21 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5194


Note You need to log in before you can comment on or make changes to this bug.