Bug 1736800
| Summary: | openshift-apiserver is down due to "x509: certificate signed by unknown authority" error | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||
| Component: | openshift-apiserver | Assignee: | Standa Laznicka <slaznick> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Xingxing Xia <xxia> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 4.2.0 | CC: | akamra, anusaxen, aos-bugs, decarr, dhellmann, dmoessne, eminguez, gklein, jhou, jiazha, mfojtik, mifiedle, mkarg, nagrawal, rsandu, scuppett, sejug, slaznick, wabouham, wking, yprokule | ||||
| Target Milestone: | --- | Keywords: | Regression, TestBlocker | ||||
| Target Release: | 4.2.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-10-16 06:34:29 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Junqi Zhao
2019-08-02 04:31:38 UTC
Encounter the same issue, it works well after re-running the apiserver pods. mac:~ jianzhang$ oc project Error from server (ServiceUnavailable): the server is currently unable to handle the request (get projects.project.openshift.io openshift-operator-lifecycle-manager) mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.2.0-0.nightly-2019-07-31-162901 True False 27h Cluster version is 4.2.0-0.nightly-2019-07-31-162901 mac:~ jianzhang$ oc delete pods --all -n openshift-apiserver pod "apiserver-2gs2q" deleted pod "apiserver-5qxlk" deleted pod "apiserver-m5cbs" deleted mac:~ jianzhang$ oc get pods -n openshift-apiserver NAME READY STATUS RESTARTS AGE apiserver-jg6mp 1/1 Running 0 18s apiserver-n2lwb 1/1 Running 0 15s apiserver-pbkgb 1/1 Running 0 15s mac:~ jianzhang$ oc project Using project "openshift-operator-lifecycle-manager" on server "https://api.zhsun7.qe.devcluster.openshift.com:6443". Adding TestBlocker - this blocks the long running reliability tests for 4.2. This seems related: https://bugzilla.redhat.com/show_bug.cgi?id=1737611 *** Bug 1737591 has been marked as a duplicate of this bug. *** Simple reproducer - force cert rotation in the openshift-kube-apiserver namespace:
oc get secret -n openshift-kube-apiserver -A -o json | jq -r '.items[] | select(.metadata.annotations."auth.openshift.io/certificate-not-after" | .!=null and fromdateiso8601<='$( date --date='+1year' +%s )') | "-n \(.metadata.namespace) \(.metadata.name)"' | xargs -n3 oc patch secret -p='{"metadata": {"annotations": {"auth.openshift.io/certificate-not-after": null}}}'
I'm seeing a similar issue on a new bare metal server running 4.1.8. After being installed and running for over 24 hours all `oc` commands return: ``` Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer") ``` (In reply to Sebastian Jug from comment #10) > I'm seeing a similar issue on a new bare metal server running 4.1.8. > > After being installed and running for over 24 hours all `oc` commands return: > ``` > Unable to connect to the server: x509: certificate signed by unknown > authority (possibly because of "crypto/rsa: verification error" while trying > to verify candidate authority certificate "kube-apiserver-lb-signer") > ``` This issue is different. Verified in 4.2.0-0.nightly-2019-08-09-000333: after keeping 35h watch, the issue still does not occur. (In reply to Michal Fojtik from comment #13) > (In reply to Sebastian Jug from comment #10) > > I'm seeing a similar issue on a new bare metal server running 4.1.8. > > > > After being installed and running for over 24 hours all `oc` commands return: > > ``` > > Unable to connect to the server: x509: certificate signed by unknown > > authority (possibly because of "crypto/rsa: verification error" while trying > > to verify candidate authority certificate "kube-apiserver-lb-signer") > > ``` > > This issue is different. Correct, thank you Michal *** Bug 1736168 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922 |