Bug 1916902 - [sig-api-machinery] OpenShift APIs remain available
Summary: [sig-api-machinery] OpenShift APIs remain available
Keywords:
Status: CLOSED DUPLICATE of bug 1845414
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Stefan Schimanski
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-15 19:24 UTC by rvanderp
Modified: 2021-01-18 12:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
[sig-api-machinery] OpenShift APIs remain available
Last Closed: 2021-01-18 12:44:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description rvanderp 2021-01-15 19:24:33 UTC
test:
[sig-api-machinery] OpenShift APIs remain available 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?search=+API+%22kubernetes-api-available%22+was+unreachable+during+disruption+for+at+leas&maxAge=48h&context=1&type=build-log&name=&maxMatches=5&maxBytes=20971520&groupBy=job

It appears that the kube-apiserver is experiencing an outage of some form and this is having an impact on other tests in ci jobs such as [sig-api-machinery] OAuth APIs remain available

Its immediately not apparent if there is some underlying infrastructure issue impacting the ability to interact with the api server.

failing job: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.6/1350120935289524224

log snippet:
Jan 15 18:19:44.660: API "kubernetes-api-available" was unreachable during disruption for at least 4m8s of 50m1s (8%):

Jan 15 17:53:24.426 E kube-apiserver Kube API started failing: Get "https://api.ci-op-dlbc4pcq-d89b2.ci.azure.devcluster.openshift.com:6443/api/v1/namespaces/kube-system?timeout=15s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 15 17:53:25.426 - 56s   E kube-apiserver Kube API is not responding to GET requests
Jan 15 17:54:21.948 I kube-apiserver Kube API started responding to GET requests
Jan 15 17:55:05.426 E kube-apiserver Kube API started failing: Get "https://api.ci-op-dlbc4pcq-d89b2.ci.azure.devcluster.openshift.com:6443/api/v1/namespaces/kube-system?timeout=15s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 15 17:55:06.426 - 56s   E kube-apiserver Kube API is not responding to GET requests
Jan 15 17:56:02.928 I kube-apiserver Kube API started responding to GET requests
Jan 15 18:06:17.426 E kube-apiserver Kube API started failing: Get "https://api.ci-op-dlbc4pcq-d89b2.ci.azure.devcluster.openshift.com:6443/api/v1/namespaces/kube-system?timeout=15s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 15 18:06:18.426 - 130s  E kube-apiserver Kube API is not responding to GET requests
Jan 15 18:08:29.601 I kube-apiserver Kube API started responding to GET requests
Jan 15 18:15:52.505 E kube-apiserver Kube API started failing: Get "https://api.ci-op-dlbc4pcq-d89b2.ci.azure.devcluster.openshift.com:6443/api/v1/namespaces/kube-system?timeout=15s": dial tcp 13.87.135.144:6443: connect: connection refused
Jan 15 18:15:53.426 E kube-apiserver Kube API is not responding to GET requests
Jan 15 18:15:53.506 I kube-apiserver Kube API started responding to GET requests

github.com/openshift/origin/test/extended/util/disruption.ExpectNoDisruption(0xc001491340, 0x3fb47ae147ae147b, 0x2bac77dd5a7, 0xc001b4e000, 0xc, 0x1f6, 0xc001b46440, 0x40)
	github.com/openshift/origin/test/extended/util/disruption/disruption.go:296 +0x665
github.com/openshift/origin/test/extended/util/disruption/controlplane.(*availableTest).test(0xc001501410, 0xc001491340, 0xc0014eac00, 0x2, 0x5e184a8)
	github.com/openshift/origin/test/extended/util/disruption/controlplane/controlplane.go:99 +0x397
github.com/openshift/origin/test/extended/util/disruption/controlplane.(*kubeAvailableTest).Test(0xc001501410, 0xc001491340, 0xc0014eac00, 0x2)
	github.com/openshift/origin/test/extended/util/disruption/controlplane/controlplane.go:31 +0x57
github.com/openshift/origin/test/extended/util/disruption.(*chaosMonkeyAdapter).Test(0xc001b442d0, 0xc00119bdc0)
	github.com/openshift/origin/test/extended/util/disruption/disruption.go:175 +0x3be
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do.func1(0xc00119bdc0, 0xc000fd7f90)
	k8s.io/kubernetes.0/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x6d
created by k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do
	k8s.io/kubernetes.0/test/e2e/chaosmonkey/chaosmonkey.go:87 +0xc9

Comment 1 Stefan Schimanski 2021-01-18 12:44:39 UTC
Disruption of the main kube-apiserver endpoints (/api/v1/namespaces) suggests that it is actually neither oauth- nor openshift-apiserver related. Hence, duplicate of Azure API outage umbrella bug.

*** This bug has been marked as a duplicate of bug 1845414 ***


Note You need to log in before you can comment on or make changes to this bug.