Description of problem: Some 4.3 installs and 4.3 nightly -> 4.3 nightly upgrades fail as cloud credential operator fail to sync credentials Version-Release number of selected component (if applicable): 4.3.0-0.nightly-2019-12-08-190955 How reproducible: Rare - 4 occurences in 14 days https://ci-search-ci-search-next.svc.ci.openshift.org/?search=Cluster+operator+cloud-credential+is+reporting+a+failure%3A&maxAge=336h&context=2&type=all Seems to be AWS specific. Additional info: Not much additional info in the logs: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/12313/artifacts/e2e-aws-upgrade/pods/openshift-cloud-credential-operator_cloud-credential-operator-684d9fcffb-kgmjd_manager.log Other prow tasks: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/12313 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-upi-4.3/464 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-upi-4.3/449 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2756/pull-ci-openshift-installer-master-e2e-aws-fips/251
I've tested this issue during upgrade from 4.4.0-0.nightly-2019-12-14-103510 to 4.4.0-0.nightly-2019-12-14-103510. Currently I don't observe reported failures by cco. We will leave cluster running for some days to check if failures are appeared for the some period
Happened 3 times over the weekend, mostly on upgrade jobs: * https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-osde2e-master-e2e-int-4.3/650 * https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/12666 * https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6396/rehearse-6396-pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-upgrade/2
As I can see the target release is 4.4 for this fix. Could you please check it on 4.4 too?
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6396/rehearse-6396-pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-upgrade/2 is 4.4 and so is https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/12666 (this is 4.4 nightly -> 4.4 nightly upgrade). However both ran on Dec 13 payloads, so the PR might not have merged by that time. Lets give it a few more days to run.
Verified on 4.4.0-0.nightly-2019-12-14-103510. I've checked logs on cco pod after two days after install and did not observe this issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581