Bug 1889541 - [4.5 upgrade]console is not accessible during cluster upgrade
Summary: [4.5 upgrade]console is not accessible during cluster upgrade
Keywords:
Status: NEW
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oauth-apiserver
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Standa Laznicka
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-19 23:31 UTC by Hongkai Liu
Modified: 2020-11-17 00:37 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Hongkai Liu 2020-10-19 23:31:40 UTC
Description of problem:
It happened during 4.5.14 to 4.6.0-rc.4 upgrade of build01, a cluster on CI build-farm.

The console UI showed
https://coreos.slack.com/archives/CHY2E1BL4/p1603137457228500

{"error":"server_error","error_description":"The authorization server encountered an unexpected condition that prevented it from fulfilling the request.","state":"7da15fbe"}

Not sure if it was a symptom of other issues.
It has been a while since the last time that upgrade requires logging in.

Eventually the console was back to normal after refreshing the page and re-logging in. The upgrade was successful too.

Not an urgent issue. Want the team to be aware of this.

Comment 2 Jakub Hadvig 2020-10-21 08:17:47 UTC
After some investigation I've found out a few errors in the console-operator logs with OAuthClient
```
2020-10-19T19:56:58.806997884Z W1019 19:56:58.804176       1 reflector.go:424] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: watch of *v1.OAuthClient ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 2381; INTERNAL_ERROR") has prevented the request from succeeding
2020-10-19T19:57:04.543936166Z E1019 19:57:04.543892       1 reflector.go:127] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to watch *v1.OAuthClient: failed to list *v1.OAuthClient: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io)
2020-10-19T19:57:05.052857714Z E1019 19:57:05.052818       1 status.go:78] ConfigMapSyncDegraded FailedGetOAuthClient the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
```

After checking the oauth-apiserver pod logs I've seen a lot of "SHOULD NOT HAPPEN" errors:
```
2020-10-19T20:08:48.962394748Z I1019 20:08:48.962350       1 client.go:360] parsed scheme: "passthrough"
2020-10-19T20:08:48.962443812Z I1019 20:08:48.962405       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.0.159.123:2379  <nil> 0 <nil>}] <nil> <nil>}
2020-10-19T20:08:48.962443812Z I1019 20:08:48.962416       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
2020-10-19T20:08:48.962763421Z I1019 20:08:48.962739       1 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000244f40, {CONNECTING <nil>}
2020-10-19T20:08:48.980362067Z I1019 20:08:48.980255       1 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000244f40, {READY <nil>}
2020-10-19T20:08:48.999857658Z I1019 20:08:48.998895       1 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-10-19T20:09:00.308037964Z E1019 20:09:00.307996       1 fieldmanager.go:175] [SHOULD NOT HAPPEN] failed to update managedFields for /, Kind=: failed to convert new object (oauth.openshift.io/v1, Kind=OAuthClient) to smd typed: no corresponding type for oauth.openshift.io/v1, Kind=OAuthClient
```

Due to that reason Im sending it to the oauth-apiserver team for further investigation.


Note You need to log in before you can comment on or make changes to this bug.