Bug 1818071
| Summary: | upgrade failed on 'controller-operator" cannot get resource "configmaps" in API group "" in the namespace "kube-system"' | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Hongkai Liu <hongkliu> |
| Component: | kube-apiserver | Assignee: | Abu Kashem <akashem> |
| Status: | CLOSED DUPLICATE | QA Contact: | Xingxing Xia <xxia> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.4 | CC: | akashem, aos-bugs, bparees, eparis, jokerman, mfojtik, sttts, wking |
| Target Milestone: | --- | ||
| Target Release: | 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | buildcop | ||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-28 16:01:26 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Hongkai Liu
2020-03-27 15:26:18 UTC
Actual error for this job: error waiting for deployment "dp" status to match expectation: Get https://api.ci-op-vj6dl884-77109.origin-ci-int-aws.dev.rhcloud.com:6443/apis/apps/v1/namespaces/e2e-k8s-sig-apps-deployment-upgrade-3212/deployments/dp: unexpected EOF EOF suggests this is a networking thing, so might be something for the SDN team. There are two issues here:
Mar 27 11:29:43.558: INFO: API was unreachable during disruption for at least 15s of 43m21s (1%):
We should not see this on a 4.4 -> 4.5 upgrade on AWS. Since we have put in a fix for graceful shutdown. So both kube-apiserver and openshift-apiserver should be able to serve requests on flight and gracefully terminate.
fail [k8s.io/kubernetes/test/e2e/upgrades/apps/deployments.go:67]: Unexpected error:
<*errors.errorString | 0xc000f1d610>: {
s: "error waiting for deployment \"dp\" status to match expectation: Get https://api.ci-op-vj6dl884-77109.origin-ci-int-aws.dev.rhcloud.com:6443/apis/apps/v1/namespaces/e2e-k8s-sig-apps-deployment-upgrade-3212/deployments/dp: unexpected EOF",
}
error waiting for deployment "dp" status to match expectation: Get https://api.ci-op-vj6dl884-77109.origin-ci-int-aws.dev.rhcloud.com:6443/apis/apps/v1/namespaces/e2e-k8s-sig-apps-deployment-upgrade-3212/deployments/dp: unexpected EOF
occurred
This also relates to kube-apiserver not responding - unexpected EOF
Given this, we need to keep it in 4.4 and investigate further.
- clusteroperator objects seem to be reporting ok. - I have gone through kube-apiserver logs, didn't see anything relevant that could be an issue. - Checked the sdn logs, nothing pops out given my limited knowledge. From the test log, I can see the following Mar 27 10:46:42.746 - 3s E kube-apiserver Kube API is not responding to GET requests ... Mar 27 10:46:47.165 I kube-apiserver Kube API started responding to GET requests Mar 27 10:54:05.746 E kube-apiserver Kube API is not responding to GET requests ... Mar 27 10:54:06.036 I kube-apiserver Kube API started responding to GET requests Mar 27 11:15:51.746 E kube-apiserver Kube API is not responding to GET requests ... Mar 27 11:15:51.919 I kube-apiserver Kube API started responding to GET requests And the "unexpected EOF" error the test encounters coincide Mar 27 10:46:46.803: INFO: Get pod "pod-secrets-cd6fdddb-3bd7-487a-bb46-06dc85de2591" in namespace "e2e-k8s-sig-storage-sig-api-machinery-secret-upgrade-1475" failed, ignoring for 2s. Error: Get https://api.ci-op-vj6dl884-77109.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/e2e-k8s-sig-storage-sig-api-machinery-secret-upgrade-1475/pods/pod-secrets-cd6fdddb-3bd7-487a-bb46-06dc85de2591: unexpected EOF Mar 27 10:46:46.803: INFO: Get pod "pod-configmap-9c585800-9bd7-4e75-98b1-f44c4bc41341" in namespace "e2e-k8s-sig-storage-sig-api-machinery-configmap-upgrade-4568" failed, ignoring for 2s. Error: Get https://api.ci-op-vj6dl884-77109.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/e2e-k8s-sig-storage-sig-api-machinery-configmap-upgrade-4568/pods/pod-configmap-9c585800-9bd7-4e75-98b1-f44c4bc41341: unexpected EOF kube-apiserver was NOT responding to request from 10:46:42 to 10:46:47 and the above "unexpected EOF" occurred at 10:46:46. But I expected the test to keep trying and pass eventually, test wait poll time is 2s and it times out after 5m. https://github.com/openshift/kubernetes/blob/d6035f3e0d79dd05628ef42231beae97806a06ad/test/e2e/framework/deployment/wait.go#L34 I also see the following in the test log: "Your test failed. Ginkgo panics to prevent subsequent assertions from running. Normally Ginkgo rescues this panic so you shouldn't see it. But, if you make an assertion in a goroutine, Ginkgo can't capture the panic. To circumvent this, you should call defer GinkgoRecover()" Does this mean we have a test running in a go-routine that does not have "defer GinkgoRecover()"? The test in question is here https://github.com/openshift/kubernetes/blob/master/test/e2e/upgrades/apps/deployments.go#L67. It's supposed to poll every 2s but I don't see enough poll attempts Mar 27 10:46:22.639: INFO: deployment status: v1.DeploymentStatus{...} Mar 27 10:46:24.667: INFO: deployment status: v1.DeploymentStatus{...} and then the Ginkgo panic follows. Could it be that the panic (from a different test) caused this test to abort and fail? I also did a search in CI, apparently there is only 3 incidents like this in the last 14 days. https://search.svc.ci.openshift.org/?search=error+waiting+for+deployment.*status+to+match+expectation.*unexpected+EOF&maxAge=336h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520 My findings don't point to a root cause yet. *** This bug has been marked as a duplicate of bug 1817588 *** |