Bug 1911812

Summary:	Workaround: oidcdiscoveryendpoint controller invalidates all TokenRequest API tokens during install
Product:	OpenShift Container Platform	Reporter:	David Hernández Fernández <dahernan>
Component:	Cloud Credential Operator	Assignee:	Joel Diaz <jdiaz>
Status:	CLOSED WORKSFORME	QA Contact:	wang lin <lwan>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.6.z	CC:	dgoodwin, jdiaz, lwan, mnewby, openshift-bugs-escalate, rcarrier, sjenning
Target Milestone:	---	Keywords:	UpcomingSprint
Target Release:	4.6.z
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1907613	Environment:
Last Closed:	2021-01-19 19:47:27 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1906880, 1907613
Bug Blocks:

Comment 3 Devan Goodwin 2021-01-05 14:57:59 UTC

Seth Joel and Maru would be better equipped to answer this. The PR to backport is open, I'm not sure how it could be worked around. Sounds a little like the customer hit the reason the controller had to be removed in the first place.

Seth / Maru / Joel, any idea how a customer who hit this could work past it during an upgrade?

Comment 4 Joel Diaz 2021-01-05 15:27:59 UTC

I suppose the question is how the CredentialsRequest object was removed. Right now in the 4.6 branch, the CredentialsRequest for the S3 permissions still exists https://github.com/openshift/cloud-credential-operator/blob/release-4.6/manifests/05-s3-credentialsrequest.yaml, and should be part of any 4.6 release image payload.

And, even if we remove that file, it is orphaned and sticks around (we have a separate TODO to clean it up https://issues.redhat.com/browse/HIVE-1326 ).

Basically, why is this object missing? Is this cluster installed with CCO in "Manual" mode? The S3 CredentialsRequest object was not in 4.5, and upgrading to 4.6 with CCO in Manual mode would require creating the new objects before upgrade. And CCO in 4.5 has code to explicitly block the upgrade if the new Secret doesn't exist https://github.com/openshift/cloud-credential-operator/blob/release-4.5/pkg/aws/actuator/actuator.go#L1277 .

You should be able to work around this by just doing an 'oc create' to the cluster using the YAML as defined in https://github.com/openshift/cloud-credential-operator/blob/release-4.6/manifests/05-s3-credentialsrequest.yaml .

Comment 15 Joel Diaz 2021-01-19 19:16:42 UTC

Haven't heard back in a week. Can we close this, or is there still an issue to investigate?

Comment 16 David Hernández Fernández 2021-01-19 19:47:27 UTC

I was out of office, apologies. Closing this case. Re-creating the namespace has fixed the problem. The update process was restarted and ran successfully