Seth Joel and Maru would be better equipped to answer this. The PR to backport is open, I'm not sure how it could be worked around. Sounds a little like the customer hit the reason the controller had to be removed in the first place.

Seth / Maru / Joel, any idea how a customer who hit this could work past it during an upgrade?

I suppose the question is how the CredentialsRequest object was removed. Right now in the 4.6 branch, the CredentialsRequest for the S3 permissions still exists https://github.com/openshift/cloud-credential-operator/blob/release-4.6/manifests/05-s3-credentialsrequest.yaml, and should be part of any 4.6 release image payload.

And, even if we remove that file, it is orphaned and sticks around (we have a separate TODO to clean it up https://issues.redhat.com/browse/HIVE-1326 ).

Basically, why is this object missing? Is this cluster installed with CCO in "Manual" mode? The S3 CredentialsRequest object was not in 4.5, and upgrading to 4.6 with CCO in Manual mode would require creating the new objects before upgrade. And CCO in 4.5 has code to explicitly block the upgrade if the new Secret doesn't exist https://github.com/openshift/cloud-credential-operator/blob/release-4.5/pkg/aws/actuator/actuator.go#L1277 .

You should be able to work around this by just doing an 'oc create' to the cluster using the YAML as defined in https://github.com/openshift/cloud-credential-operator/blob/release-4.6/manifests/05-s3-credentialsrequest.yaml .

Haven't heard back in a week. Can we close this, or is there still an issue to investigate?

I was out of office, apologies. Closing this case. Re-creating the namespace has fixed the problem. The update process was restarted and ran successfully