Description of problem: A backport PR to 4.5 involves a new issue. This PR adds a function that if cluster is in mint/passthrough mode, and at the same time root creds being removed, it will set cloud-credential upgradeable=false. But the issue now is the state won't change immediately when we remove or re-add the root creds, it maybe need to wait for several hours. The same issue occurred when re-creating root creds while upgradeable=false, the state won't change immediately to upgradeable=true. Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-09-28-124031 How reproducible: always Steps to Reproduce: 1. Launch an IPI with latest 4.5 2. Check the root creds is in mint mode $oc get secret aws-creds -n kube-system -o json | jq .metadata.annotations { "cloudcredential.openshift.io/mode": "mint" } 3.Check the cloud-credential upgradeable=true $oc get co cloud-credential -o json | jq -r ".status.conditions" [ { "lastTransitionTime": "2020-09-28T04:24:19Z", "status": "True", "type": "Upgradeable" } ] 4. Remove the root creds ,check the cloud-credential upgradeable status again,the status is still upgradeable=true 5. The same situation when we re-add root creds, check the upgradeable=false now, and recreate root creds, verify the cloud-credential condiftion is still upgradeable=false Actual results: 1. Launch an IPI with latest 4.5 2. Check the root creds is in mint mode $oc get secret aws-creds -n kube-system -o json | jq .metadata.annotations { "cloudcredential.openshift.io/mode": "mint" } 3.Check the cloud-credential upgradeable=true $oc get co cloud-credential -o json | jq -r ".status.conditions" [ { "lastTransitionTime": "2020-09-28T04:24:19Z", "status": "True", "type": "Upgradeable" } ] 4. Remove the root creds ,check the cloud-credential upgradeable status again,the status is still upgradeable=true 5. The same situation when we re-add root creds, check the upgradeable=false now, and recreate root creds, verify the cloud-credential condiftion is still upgradeable=false Expected results: It can immediately update the upgradeable status. Additional info: [1] in version 4.6, it works well [2] related bug https://bugzilla.redhat.com/show_bug.cgi?id=1871713
Sorry for that I put a wrong description info in "Actual results" above Actual results: Cloud-credential doesn’t immediately update the upgradeable status when we remove or re-add the root creds until we force a cco reconcile via adding an annotations in CloudCredential object or deleting the cco pod
The issue has fixed. test payload:4.5.0-0.nightly-2020-11-22-160319 1. Run "oc edit cloudcredential cluster" and modify .spec.credentialsMode to "Mint" or "Passthrough" 2. remove root creds oc delete secret aws-creds -n kube-system 3. check co cloud-credential status, Upgradeable=False $ oc get co cloud-credential -o json | jq -r ".status.conditions" [ { "lastTransitionTime": "2020-11-23T02:31:48Z", "message": "No credentials requests reporting errors.", "reason": "NoCredentialsFailing", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2020-11-23T02:50:15Z", "message": "4 of 4 credentials requests provisioned and reconciled.", "reason": "ReconcilingComplete", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2020-11-23T02:31:48Z", "status": "True", "type": "Available" }, { "lastTransitionTime": "2020-11-23T03:35:37Z", "message": "Parent credential secret must be restored prior to upgrade: kube-system/aws-creds", "reason": "MissingRootCredential", "status": "False", "type": "Upgradeable" } ] 4. re-create root credential 5. check co cloud-credential status, Upgradeable=True $ oc get co cloud-credential -o json | jq -r ".status.conditions" [ { "lastTransitionTime": "2020-11-23T02:31:48Z", "message": "No credentials requests reporting errors.", "reason": "NoCredentialsFailing", "status": "False", "type": "Degraded" }, { "lastTransitionTime": "2020-11-23T02:50:15Z", "message": "4 of 4 credentials requests provisioned and reconciled.", "reason": "ReconcilingComplete", "status": "False", "type": "Progressing" }, { "lastTransitionTime": "2020-11-23T02:31:48Z", "status": "True", "type": "Available" }, { "lastTransitionTime": "2020-11-23T03:36:22Z", "status": "True", "type": "Upgradeable" } ]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.5.21 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5194