Bug 1889541

Summary:	[4.5 upgrade]console is not accessible during cluster upgrade
Product:	OpenShift Container Platform	Reporter:	Hongkai Liu <hongkliu>
Component:	oauth-apiserver	Assignee:	Standa Laznicka <slaznick>
Status:	CLOSED NOTABUG	QA Contact:	Xingxing Xia <xxia>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.5	CC:	aos-bugs, jokerman, mfojtik
Target Milestone:	---	Keywords:	Upgrades
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-14 08:52:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hongkai Liu 2020-10-19 23:31:40 UTC

Description of problem:
It happened during 4.5.14 to 4.6.0-rc.4 upgrade of build01, a cluster on CI build-farm.

The console UI showed
https://coreos.slack.com/archives/CHY2E1BL4/p1603137457228500

{"error":"server_error","error_description":"The authorization server encountered an unexpected condition that prevented it from fulfilling the request.","state":"7da15fbe"}

Not sure if it was a symptom of other issues.
It has been a while since the last time that upgrade requires logging in.

Eventually the console was back to normal after refreshing the page and re-logging in. The upgrade was successful too.

Not an urgent issue. Want the team to be aware of this.

Comment 2 Jakub Hadvig 2020-10-21 08:17:47 UTC

After some investigation I've found out a few errors in the console-operator logs with OAuthClient
```
2020-10-19T19:56:58.806997884Z W1019 19:56:58.804176       1 reflector.go:424] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: watch of *v1.OAuthClient ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 2381; INTERNAL_ERROR") has prevented the request from succeeding
2020-10-19T19:57:04.543936166Z E1019 19:57:04.543892       1 reflector.go:127] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to watch *v1.OAuthClient: failed to list *v1.OAuthClient: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io)
2020-10-19T19:57:05.052857714Z E1019 19:57:05.052818       1 status.go:78] ConfigMapSyncDegraded FailedGetOAuthClient the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
```

After checking the oauth-apiserver pod logs I've seen a lot of "SHOULD NOT HAPPEN" errors:
```
2020-10-19T20:08:48.962394748Z I1019 20:08:48.962350       1 client.go:360] parsed scheme: "passthrough"
2020-10-19T20:08:48.962443812Z I1019 20:08:48.962405       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.0.159.123:2379  <nil> 0 <nil>}] <nil> <nil>}
2020-10-19T20:08:48.962443812Z I1019 20:08:48.962416       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
2020-10-19T20:08:48.962763421Z I1019 20:08:48.962739       1 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000244f40, {CONNECTING <nil>}
2020-10-19T20:08:48.980362067Z I1019 20:08:48.980255       1 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000244f40, {READY <nil>}
2020-10-19T20:08:48.999857658Z I1019 20:08:48.998895       1 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-10-19T20:09:00.308037964Z E1019 20:09:00.307996       1 fieldmanager.go:175] [SHOULD NOT HAPPEN] failed to update managedFields for /, Kind=: failed to convert new object (oauth.openshift.io/v1, Kind=OAuthClient) to smd typed: no corresponding type for oauth.openshift.io/v1, Kind=OAuthClient
```

Due to that reason Im sending it to the oauth-apiserver team for further investigation.

Comment 6 Standa Laznicka 2020-12-07 15:18:32 UTC

Hongkai, have you observed this issue recently?

Comment 7 Hongkai Liu 2020-12-10 00:56:32 UTC

Standa,
No, not this one recently.

Comment 8 Standa Laznicka 2020-12-14 08:52:16 UTC

thanks for the information, I'm going to close as NOTABUG