Bug 1905576

Summary: [4.5] Changing the bound token service account issuer invalids previously issued bound tokens
Product: OpenShift Container Platform Reporter: Maru Newby <mnewby>
Component: kube-apiserverAssignee: Maru Newby <mnewby>
Status: CLOSED ERRATA QA Contact: scheng
Severity: high Docs Contact:
Priority: low    
Version: 4.5CC: aos-bugs, kewang, mfojtik, scheng, xxia
Target Milestone: ---Keywords: UpcomingSprint
Target Release: 4.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: Changing the serviceAccountIssuer field of the authentication resource will update the kube-apiserver to validate tokens with the new issuer and reject tokens with the previous issuer. kube-apiserver does not support multiple issuers at this time, so a graceful transition is not possible. Consequence: Changing the serviceAccountIssuer has the potential to disrupt applications relying on bound tokens. Unless an application is coded to explicitly request a new token when their existing token starts receiving 401 responses from the apiserver, they will continue to use the invalid token until restarted or until their invalid token exceeds 80% of its duration (at which point the kubelet will request a new token for them). Workaround (if any): Only change the serviceAccountIssuer field if disruption is acceptable and restarting all pods is an option.
Story Points: ---
Clone Of: 1905573 Environment:
Last Closed: 2021-03-03 04:40:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1905573    
Bug Blocks:    

Description Maru Newby 2020-12-08 15:22:27 UTC
+++ This bug was initially created as a clone of Bug #1905573 +++

+++ This bug was initially created as a clone of Bug #1905328 +++

Changing the serviceAccountIssuer field of the authentication resource will update the kube-apiserver to validate tokens with the new issuer and reject tokens with the previous issuer. This has the potential to disrupt applications relying on bound tokens. Unless an application is coded to explicitly request a new token when their existing token starts receiving 401 responses from the apiserver, they will continue to use the invalid token until restarted or until their invalid token exceeds 80% of its duration (at which point the kubelet will request a new token for them).

A likely fix for 4.8 is ensuring the apiserver supports a graceful transition between 2 issuers. For 4.7, 4.6, and 4.5, the near-term remedy is to document the impact of a change in issuer and ensure the compatibility of control plane components like controller manager with an issuer change.

--- Additional comment from OpenShift Automated Release Tooling on 2020-12-08 15:02:55 UTC ---

Elliott changed bug status from MODIFIED to ON_QA.

Comment 1 Michal Fojtik 2021-01-07 15:24:34 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 2 Maru Newby 2021-01-07 18:06:04 UTC
I'm still working on this, but am blocked by the 4.6 fix being cherry-picked and verified.

Comment 3 Maru Newby 2021-02-01 16:10:41 UTC
Bumping to high. The risk of this change is zero - doc only - and changing the issuer has the potential to disrupt user workloads by invalidating all previously issued tokens.

Comment 8 errata-xmlrpc 2021-03-03 04:40:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.5.33 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0428