1688503 – openshift-apiserver goes Available=False after 4hrs

Bug 1688503 - openshift-apiserver goes Available=False after 4hrs

Summary: openshift-apiserver goes Available=False after 4hrs

Keywords:
Status:	CLOSED DUPLICATE of bug 1684547
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Master
Sub Component:
Version:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	4.1.0
Assignee:	Michal Fojtik
QA Contact:	Xingxing Xia
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-03-13 21:06 UTC by Seth Jennings
Modified:	2019-03-18 14:35 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-03-18 14:35:53 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1688147	0	urgent	CLOSED	Service discovery is broken for a several hours cluster	2021-02-22 00:41:40 UTC

Description Seth Jennings 2019-03-13 21:06:10 UTC

$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.alpha-2019-03-13-010143   True        False         5h58m   Cluster version is 4.0.0-0.alpha-2019-03-13-010143

$ oc get clusteroperators openshift-apiserver
NAME                  VERSION                           AVAILABLE   PROGRESSING   FAILING   SINCE
openshift-apiserver   4.0.0-0.alpha-2019-03-13-010143   False       False         False     133m

$ oc get clusteroperators openshift-apiserver -oyaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-13T14:59:59Z
  generation: 1
  name: openshift-apiserver
  resourceVersion: "158276"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-apiserver
  uid: ac303a76-45a0-11e9-8640-0651e81f5f5c
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-03-13T16:50:42Z
    reason: AsExpected
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-13T15:02:19Z
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-13T18:52:06Z
    message: |-
      Available: v1.apps.openshift.io is not ready: 401
      Available: v1.authorization.openshift.io is not ready: 401
      Available: v1.build.openshift.io is not ready: 401
      Available: v1.image.openshift.io is not ready: 401
      Available: v1.oauth.openshift.io is not ready: 401
      Available: v1.project.openshift.io is not ready: 401
      Available: v1.quota.openshift.io is not ready: 401
      Available: v1.route.openshift.io is not ready: 401
      Available: v1.security.openshift.io is not ready: 401
      Available: v1.template.openshift.io is not ready: 401
      Available: v1.user.openshift.io is not ready: 401
    reason: Available
    status: "False"
    type: Available
  - lastTransitionTime: 2019-03-13T14:59:59Z
    reason: NoData
    status: Unknown
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: openshiftapiservers
  - group: ""
    name: openshift-config
    resource: namespaces
  - group: ""
    name: openshift-config-managed
    resource: namespaces
  - group: ""
    name: openshift-apiserver-operator
    resource: namespaces
  - group: ""
    name: openshift-apiserver
    resource: namespaces
  - group: apiregistration.k8s.io
    name: v1.apps.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.authorization.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.build.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.image.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.oauth.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.project.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.quota.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.route.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.security.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.template.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.user.openshift.io
    resource: apiservices
  versions:
  - name: operator
    version: 4.0.0-0.alpha-2019-03-13-010143
  - name: openshift-apiserver
    version: 4.0.0-0.alpha-2019-03-13-010143_openshift

$ oc logs -n openshift-apiserver apiserver-7qphx | tail -n20
E0313 21:05:18.574628       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:18.581683       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:18.592547       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.367177       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.399546       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.437282       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.439531       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.488737       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.543609       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.545969       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.550761       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.212368       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.219074       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.401941       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.406456       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.490426       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.546941       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.547837       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.551435       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.552280       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]

Comment 1 Seth Jennings 2019-03-13 21:08:51 UTC

I think this is because the apiserver no longer trusted the CA that signed the cert that openshift-authentication is using.

Comment 2 Xingxing Xia 2019-03-14 07:51:42 UTC

Same as bug 1688147

Comment 3 Radek Vokál 2019-03-18 14:35:53 UTC


*** This bug has been marked as a duplicate of bug 1684547 ***

Note You need to log in before you can comment on or make changes to this bug.