This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1277912 - race condition problem when deleting lots of projects
race condition problem when deleting lots of projects
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Andy Goldstein
Jianwei Hou
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-04 05:33 EST by Meng Bo
Modified: 2015-11-23 09:44 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 09:44:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Meng Bo 2015-11-04 05:33:09 EST
Description of problem:
Delete all the projects (more than 40) with `oc delete project --all`, there will be tons of "namespace_controller.go:98] namespaces "<ns_name>" not found" logs in master log. And the projects are hard to be deleted.

When deleting one project, there are 3 or 4 such logs found,
When deleting a few projects (less than 10), there are dozens of such logs found,
When deleting large number of projects (about 40), there are tons of such logs found, and the projects are hard to be deleted(In Terminating status for long time).

Version-Release number of selected component (if applicable):
openshift v3.0.2.905
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4
etcd 2.1.2


How reproducible:
always

Steps to Reproduce:
1. Create about 40 projects 
$ for i in {1..40} ; do oc new-project pro-$i ; done
2. Delete them all via oc
$ oc delete project --all
3. Check the master log at the same time. 

Actual results:
Tons of "namespace_controller.go:98] namespaces "<ns_name>" not found" logs found in master.log. And the projects are hard to be deleted.

Expected results:
The projects should be deleted smoothly.

Additional info:
Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.351443   27114 namespace_controller.go:98] namespaces "u1pp-43" not found
Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.489693   27114 tokens_controller.go:185] Error deleting secret u1pp-45/deployer-token-8h4rd: secrets "deployer-token-8h4rd" not found
Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.534707   27114 namespace_controller.go:98] namespaces "u1pp-45" not found
Nov 04 16:46:12 master.bmeng.local openshift-master[27114]: E1104 16:46:12.540589   27114 namespace_controller.go:98] namespaces "u1pp-4" not found
Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.071649   27114 namespace_controller.go:98] namespaces "u1pp-34" not found
Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.271790   27114 namespace_controller.go:98] namespaces "u1pp-34" not found
Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.541892   27114 cache.go:188] error synchronizing: Internal error occurred: [namespace u1pp-34 does not exist, couldn't find ObjectMeta field in api.LocalResourceAccessReview{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, Action:api.AuthorizationAttributes{Namespace:"", Verb:"get", Resource:"namespaces", ResourceName:"u1pp-34", Content:runtime.EmbeddedObject{Object:runtime.Object(nil)}}}]
Nov 04 16:46:15 master.bmeng.local openshift-master[27114]: E1104 16:46:15.882300   27114 namespace_controller.go:98] namespaces "u1pp-42" not found
Nov 04 16:46:16 master.bmeng.local openshift-master[27114]: E1104 16:46:16.002200   27114 namespace_controller.go:98] namespaces "u1pp-42" not found
Nov 04 16:46:16 master.bmeng.local openshift-master[27114]: E1104 16:46:16.121566   27114 namespace_controller.go:98] namespaces "u1pp-42" not found
Nov 04 16:46:18 master.bmeng.local openshift-master[27114]: E1104 16:46:18.541648   27114 namespace_controller.go:98] namespaces "u1pp-48" not found
Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.131805   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.311636   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.481781   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.643805   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.781814   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:19 master.bmeng.local openshift-master[27114]: E1104 16:46:19.951690   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:20 master.bmeng.local openshift-master[27114]: E1104 16:46:20.104844   27114 namespace_controller.go:98] namespaces "u1pp-1" not found
Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.321843   27114 namespace_controller.go:98] namespaces "u1pp-40" not found
Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.451819   27114 namespace_controller.go:98] namespaces "u1pp-40" not found
Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.611834   27114 namespace_controller.go:98] namespaces "u1pp-40" not found
Nov 04 16:46:21 master.bmeng.local openshift-master[27114]: E1104 16:46:21.771835   27114 namespace_controller.go:98] namespaces "u1pp-40" not found
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.021761   27114 namespace_controller.go:98] namespaces "u1pp-12" not found
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.181896   27114 cache.go:188] error synchronizing: Internal error occurred: [namespace u1pp-12 does not exist, couldn't find ObjectMeta field in api.LocalResourceAccessReview{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, Action:api.AuthorizationAttributes{Namespace:"", Verb:"get", Resource:"namespaces", ResourceName:"u1pp-12", Content:runtime.EmbeddedObject{Object:runtime.Object(nil)}}}]
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.191894   27114 namespace_controller.go:98] namespaces "u1pp-12" not found
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.351877   27114 namespace_controller.go:98] namespaces "u1pp-12" not found
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.551629   27114 namespace_controller.go:98] namespaces "u1pp-12" not found
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.711673   27114 namespace_controller.go:98] namespaces "u1pp-12" not found
Nov 04 16:46:26 master.bmeng.local openshift-master[27114]: E1104 16:46:26.881834   27114 namespace_controller.go:98] namespaces "u1pp-12" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.142734   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.241798   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.341649   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.442237   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.541718   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.641807   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.741682   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:52 master.bmeng.local openshift-master[27114]: E1104 16:59:52.881723   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.032021   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.155210   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.262370   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.361820   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.462356   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.562296   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.661806   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.761868   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.861777   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:53 master.bmeng.local openshift-master[27114]: E1104 16:59:53.961868   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.082369   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.191763   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.356316   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.501672   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.601683   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.710124   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.802245   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:54 master.bmeng.local openshift-master[27114]: E1104 16:59:54.902648   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.002341   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.101743   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.201779   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.302386   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.401774   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.551877   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.721706   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.831905   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:55 master.bmeng.local openshift-master[27114]: E1104 16:59:55.931822   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.031964   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.132360   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.231730   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.333263   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.431677   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.532286   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.632386   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Nov 04 16:59:56 master.bmeng.local openshift-master[27114]: E1104 16:59:56.741578   27114 namespace_controller.go:98] namespaces "u1pp-37" not found
Comment 2 Clayton Coleman 2015-11-04 09:30:38 EST
This looks similar to the other namespace controller hot looping thing.
Comment 3 Derek Carr 2015-11-04 14:21:17 EST
What was the role of the user that did the delete of all projects?
Comment 4 Derek Carr 2015-11-04 14:26:24 EST
Here is a related issue on why its bad to ever do 

oc delete project --all

as cluster admin

https://github.com/openshift/origin/issues/3274
Comment 5 Derek Carr 2015-11-04 14:48:30 EST
I did the following to reproduce:

$ oc login --username=test --password=mypass
$ for i in {1..40} ; do oc new-project pro-$i ; done
$ oc delete projects --all

The first 34 projects deleted smoothly, but then something stalled on the final 6.

I then saw messages like following repeated in log for a few minutes until the project was ultimately deleted:

E1104 14:47:13.554209   18982 namespace_controller.go:98] namespaces "pro-32" not found

Investigating more.
Comment 6 Derek Carr 2015-11-04 18:35:49 EST
Fix for issue is pending 3.1

https://github.com/openshift/origin/pull/5720
Comment 7 Meng Bo 2015-11-06 02:58:09 EST
Issue fixed on openshift v1.0.7-291-g9bd5fb9 with Origin env, will wait for the changes merge into latest OSE puddle.
Comment 8 Meng Bo 2015-11-10 00:29:48 EST
Checked with build: openshift v3.1.0.3

Issue has been fixed.

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.