Bug 1661394

Summary: Unable to authenticate the request due to an error: [invalid bearer token errors in the master logs
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: apiserver-authAssignee: Stefan Schimanski <sttts>
Status: CLOSED DEFERRED QA Contact: Wei Sun <wsun>
Severity: high Docs Contact:
Priority: high    
Version: 3.9.0CC: aos-bugs, emahoney, evb, fshaikh, jokerman, mirollin, mmccomas, nagrawal, slaznick, somalley, ssadhale, szobair
Target Milestone: ---Keywords: Reopened
Target Release: 3.9.zFlags: slaznick: needinfo? (ssadhale)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-20 18:52:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jaspreet Kaur 2018-12-21 06:06:50 UTC
Description of problem: Master logs populated with below errors: 


> Dec 11 18:43:23 master.example.com atomic-openshift-master-api[94560]: E1211 18:43:23.469811   94560 authentication.go:64] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, oauthaccesstokens.oauth.openshift.io "YEx9fyVdKZHo-N3Q0fqtCRBJAa2EkpGlrbujYydRhXo" not found]]

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: Lots of error messages in the master logs


Expected results: These messages should not be there.


Additional info:

Comment 15 Standa Laznicka 2019-05-29 11:07:12 UTC
Neelesh never really got the answer to his question - what's broken? It's hard to tell what the root cause might be since the only thing we seem to have is the token which can be used by, well, anything.

Comment 16 Shah Zobair 2019-09-17 15:47:56 UTC
Is there a way to track down the requester? We are also facing a similar issue which is generating huge calls to one of the Master node:

Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.200563   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.205312   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.205338   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.205339   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.206694   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.206766   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Sep 17 11:34:22 ocbdllmcsmst002.devhcloud.XXXX.net atomic-openshift-master-api[68844]: E0917 11:34:22.544204   68844 authentication.go:58] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]

Comment 17 Standa Laznicka 2019-09-18 06:35:36 UTC
That's a completely different issue - one of your certificates expired. You may want to check the logs, specifically the audit log, and renew the certificate for whoever is the caller.

Comment 19 Shah Zobair 2019-09-18 13:55:47 UTC
I have checked all the certificates and those were recently renewed. Here is the certificate validity report: https://access.redhat.com/hydra/rest/cases/02463245/attachments/137694fe-70aa-4049-a1b1-67512b5a7a36 . There might be something else that we are missing, but can't identify the component from the logs.

Thanks

Comment 21 Stephen Cuppett 2019-11-20 18:52:12 UTC
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift