Bug 1583079
| Summary: | Project keeps in Terminating status | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wang Haoran <haowang> |
| Component: | Service Catalog | Assignee: | David Eads <deads> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Xingxing Xia <xxia> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.10.0 | CC: | aos-bugs, chezhang, haowang, jaboyd, jiazha, jokerman, jpeeler, mfojtik, mmccomas, pmorie, sdodson, xtian, zhsun, zitang |
| Target Milestone: | --- | ||
| Target Release: | 3.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-12-20 21:36:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Captured some logs:
from controller logs (last 6 hours):
W0529 09:59:03.400268 1 reflector.go:341] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: watch of <nil> ended with: unexpected object: &{map[message:The resourceVersion for the provided watch is too old. reason:Expired code:410 kind:Status apiVersion:v1 metadata:map[] status:Failure]}
W0529 09:59:20.933720 1 garbagecollector.go:598] failed to discover some groups: map[servicecatalog.k8s.io/v1beta1:the server is currently unable to handle the request]
W0529 09:59:51.022132 1 garbagecollector.go:598] failed to discover some groups: map[servicecatalog.k8s.io/v1beta1:the server is currently unable to handle the request]
W0529 10:00:21.108070 1 garbagecollector.go:598] failed to discover some groups: map[servicecatalog.k8s.io/v1beta1:the server is currently unable to handle the request]
from the api logs (last 6 hours):
I0529 03:58:41.249216 1 get.go:238] Starting watch for /api/v1/endpoints, rv=924307 labels= fields= timeout=7m55s
E0529 03:58:41.504425 1 handler_proxy.go:123] error resolving kube-service-catalog/apiserver: no service port "443" found for service "apiserver"
E0529 03:58:41.505416 1 handler_proxy.go:123] error resolving kube-service-catalog/apiserver: no service port "443" found for service "apiserver"
E0529 03:58:41.506266 1 handler_proxy.go:123] error resolving kube-service-catalog/apiserver: no service port "443" found for service "apiserver"
E0529 03:58:41.507108 1 handler_proxy.go:123] error resolving kube-service-catalog/apiserver: no service port "443" found for service "apiserver"
$ cat api.log | grep 'error resolving kube-service-catalog' | wc -l
116887
[root@preserved-ui-xxia-310-master-etcd-1 ~]# oc get svc -n kube-service-catalog
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
apiserver ClusterIP 172.30.217.221 <none> 8443/TCP 6d
controller-manager ClusterIP 172.30.132.230 <none> 443/TCP 6d
It seems like the service catalog is misconfigured and trying to connect to apiserver using wrong port?
issues: 1. the apiserver pod is running with --secure-port=6443, and the apiserver service is set 8443->443,this is not correct, should mapping to 6443. 2. the kube-apiserver is proxyed talking to 443 port of the service instead of 8443 setup in the apiservice, this maybe a api-aggregator problem? After I manually correct service using mapping 443-6443, all works as expected. Proposed fix upstream: https://github.com/kubernetes/kubernetes/pull/64443 origin pick https://github.com/openshift/origin/pull/19866 Fix was merged a month ago, will be in 3.10.0 release. Tested in OCP v3.10.10, the issue is fixed, projects can be deleted. Please move to ON_QA |
Description of problem: The project cannot be deleted, and keeps "Terminating" status Version-Release number of selected component (if applicable): How reproducible: Not sure how to reproduce, this does not happen in a new installed cluster. Steps to Reproduce: 1.oc get ns NAME STATUS AGE 322hm Terminating 2h 8uja8 Terminating 4h 94ab4 Terminating 3h a2mrm Terminating 2h default Active 5d 2.the namespace leaves the "kubernete" finalizer their oc get ns 322hm -o json { "apiVersion": "v1", "kind": "Namespace", "metadata": { "annotations": { "openshift.io/description": "", "openshift.io/display-name": "", "openshift.io/requester": "zyp2", "openshift.io/sa.scc.mcs": "s0:c18,c12", "openshift.io/sa.scc.supplemental-groups": "1000330000/10000", "openshift.io/sa.scc.uid-range": "1000330000/10000" }, "creationTimestamp": "2018-05-28T06:03:36Z", "deletionTimestamp": "2018-05-28T06:29:15Z", "name": "322hm", "resourceVersion": "913529", "selfLink": "/api/v1/namespaces/322hm", "uid": "dc60d238-623c-11e8-bcb3-fa163e73dad0" }, "spec": { "finalizers": [ "kubernetes" ] }, "status": { "phase": "Terminating" } } 3. Actual results: Expected results: Additional info: