Bug 1277912
| Summary: | race condition problem when deleting lots of projects | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Meng Bo <bmeng> |
| Component: | Node | Assignee: | Andy Goldstein <agoldste> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jianwei Hou <jhou> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.1.0 | CC: | aos-bugs, ccoleman, decarr, eparis, gpei, jokerman, mmccomas |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-23 14:44:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This looks similar to the other namespace controller hot looping thing. What was the role of the user that did the delete of all projects? Here is a related issue on why its bad to ever do oc delete project --all as cluster admin https://github.com/openshift/origin/issues/3274 I did the following to reproduce:
$ oc login --username=test --password=mypass
$ for i in {1..40} ; do oc new-project pro-$i ; done
$ oc delete projects --all
The first 34 projects deleted smoothly, but then something stalled on the final 6.
I then saw messages like following repeated in log for a few minutes until the project was ultimately deleted:
E1104 14:47:13.554209 18982 namespace_controller.go:98] namespaces "pro-32" not found
Investigating more.
Fix for issue is pending 3.1 https://github.com/openshift/origin/pull/5720 Issue fixed on openshift v1.0.7-291-g9bd5fb9 with Origin env, will wait for the changes merge into latest OSE puddle. Checked with build: openshift v3.1.0.3 Issue has been fixed. Move bug to verified. |
Description of problem: Delete all the projects (more than 40) with `oc delete project --all`, there will be tons of "namespace_controller.go:98] namespaces "<ns_name>" not found" logs in master log. And the projects are hard to be deleted. When deleting one project, there are 3 or 4 such logs found, When deleting a few projects (less than 10), there are dozens of such logs found, When deleting large number of projects (about 40), there are tons of such logs found, and the projects are hard to be deleted(In Terminating status for long time). Version-Release number of selected component (if applicable): openshift v3.0.2.905 kubernetes v1.2.0-alpha.1-1107-g4c8e6f4 etcd 2.1.2 How reproducible: always Steps to Reproduce: 1. Create about 40 projects $ for i in {1..40} ; do oc new-project pro-$i ; done 2. Delete them all via oc $ oc delete project --all 3. Check the master log at the same time. Actual results: Tons of "namespace_controller.go:98] namespaces "<ns_name>" not found" logs found in master.log. And the projects are hard to be deleted. Expected results: The projects should be deleted smoothly. Additional info: Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.351443 27114 namespace_controller.go:98] namespaces "u1pp-43" not found Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.489693 27114 tokens_controller.go:185] Error deleting secret u1pp-45/deployer-token-8h4rd: secrets "deployer-token-8h4rd" not found Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.534707 27114 namespace_controller.go:98] namespaces "u1pp-45" not found Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.540589 27114 namespace_controller.go:98] namespaces "u1pp-4" not found Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.071649 27114 namespace_controller.go:98] namespaces "u1pp-34" not found Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.271790 27114 namespace_controller.go:98] namespaces "u1pp-34" not found Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.541892 27114 cache.go:188] error synchronizing: Internal error occurred: [namespace u1pp-34 does not exist, couldn't find ObjectMeta field in api.LocalResourceAccessReview{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, Action:api.AuthorizationAttributes{Namespace:"", Verb:"get", Resource:"namespaces", ResourceName:"u1pp-34", Content:runtime.EmbeddedObject{Object:runtime.Object(nil)}}}] Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.882300 27114 namespace_controller.go:98] namespaces "u1pp-42" not found Nov 04 16:46:16 master.bmeng.local openshift-master[27114]: E1104 16:46:16.002200 27114 namespace_controller.go:98] namespaces "u1pp-42" not found Nov 04 16:46:16 master.bmeng.local openshift-master[27114]: E1104 16:46:16.121566 27114 namespace_controller.go:98] namespaces "u1pp-42" not found Nov 04 16:46:18 master.bmeng.local openshift-master[27114]: E1104 16:46:18.541648 27114 namespace_controller.go:98] namespaces "u1pp-48" not found Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.131805 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.311636 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.481781 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.643805 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.781814 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.951690 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:20 master.bmeng.local openshift-master[27114]: E1104 16:46:20.104844 27114 namespace_controller.go:98] namespaces "u1pp-1" not found Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.321843 27114 namespace_controller.go:98] namespaces "u1pp-40" not found Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.451819 27114 namespace_controller.go:98] namespaces "u1pp-40" not found Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.611834 27114 namespace_controller.go:98] namespaces "u1pp-40" not found Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.771835 27114 namespace_controller.go:98] namespaces "u1pp-40" not found Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.021761 27114 namespace_controller.go:98] namespaces "u1pp-12" not found Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.181896 27114 cache.go:188] error synchronizing: Internal error occurred: [namespace u1pp-12 does not exist, couldn't find ObjectMeta field in api.LocalResourceAccessReview{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, Action:api.AuthorizationAttributes{Namespace:"", Verb:"get", Resource:"namespaces", ResourceName:"u1pp-12", Content:runtime.EmbeddedObject{Object:runtime.Object(nil)}}}] Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.191894 27114 namespace_controller.go:98] namespaces "u1pp-12" not found Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.351877 27114 namespace_controller.go:98] namespaces "u1pp-12" not found Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.551629 27114 namespace_controller.go:98] namespaces "u1pp-12" not found Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.711673 27114 namespace_controller.go:98] namespaces "u1pp-12" not found Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.881834 27114 namespace_controller.go:98] namespaces "u1pp-12" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.142734 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.241798 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.341649 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.442237 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.541718 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.641807 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.741682 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.881723 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.032021 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.155210 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.262370 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.361820 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.462356 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.562296 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.661806 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.761868 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.861777 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.961868 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.082369 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.191763 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.356316 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.501672 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.601683 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.710124 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.802245 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.902648 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.002341 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.101743 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.201779 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.302386 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.401774 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.551877 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.721706 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.831905 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.931822 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.031964 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.132360 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.231730 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.333263 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.431677 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.532286 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.632386 27114 namespace_controller.go:98] namespaces "u1pp-37" not found Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.741578 27114 namespace_controller.go:98] namespaces "u1pp-37" not found