Bug 1367432 - limitation of project number
Summary: limitation of project number
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Master
Version: 3.x
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Maciej Szulik
QA Contact: weiwei jiang
URL:
Whiteboard:
: 1367308 1368098 1369789 1369823 1370988 1371402 1371839 1380922 1381038 (view as bug list)
Depends On:
Blocks: OSOPS_V3
TreeView+ depends on / blocked
 
Reported: 2016-08-16 11:51 UTC by Behnam Loghmani
Modified: 2016-11-09 21:31 UTC (History)
32 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-20 03:32:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Behnam Loghmani 2016-08-16 11:51:31 UTC
I don't have any projects but when I want to create a project, openshift show a "projectrequests "*****" is forbidden: user BehnamLoghmani cannot create more than 1 project(s)."

Comment 1 Corey Daley 2016-08-16 18:25:13 UTC
I am also experiencing this issue on https://console.preview.openshift.com

Comment 2 Jordan Liggitt 2016-08-16 20:47:46 UTC
The cause of the error is an existing project/namespace, with a requester annotation matching the user, in Active state, without any rolebindings that give the user permission to see/delete the project/namespace

There are apparently two ways to reach this state:

1. An error occurs during the projectrequest API call, after the project is created with the requester annotation, but before the rolebinding granting the user permission to access the project completes (or the error occurs while creating that rolebinding). 

This is reproducible in an environment that is generally experiencing API call failures (timeouts, etc)


2. Deleting a project (DELETE /oapi/v1/projects/myproject) triggered removing the rolebinding that gave the user access, but somehow(?) still left the project in an Active state, with no deletionTimestamp set, so the finalizers did not remove themselves and delete the project.

This is not yet reproducible

Comment 3 David Eads 2016-08-17 12:00:41 UTC
*** Bug 1367308 has been marked as a duplicate of this bug. ***

Comment 5 Jordan Liggitt 2016-08-19 14:31:06 UTC
*** Bug 1368098 has been marked as a duplicate of this bug. ***

Comment 8 Jordan Liggitt 2016-08-23 07:04:16 UTC
After auditing the project deletion code, I'm reasonably sure this is only being caused by failures during project creation, which is leaving projects in existence, but without any permissions granted to the requesting user

https://github.com/openshift/origin/pull/10577 adds error handling to the projectrequest API that cleans up a partially created project template if a failure occurs during the creation.

Comment 13 Xiaoli Tian 2016-08-25 07:48:27 UTC
*** Bug 1369823 has been marked as a duplicate of this bug. ***

Comment 14 Jordan Liggitt 2016-08-26 12:44:25 UTC
*** Bug 1369789 has been marked as a duplicate of this bug. ***

Comment 15 Jordan Liggitt 2016-08-30 16:44:29 UTC
*** Bug 1370988 has been marked as a duplicate of this bug. ***

Comment 16 Abhishek Gupta 2016-08-30 23:17:24 UTC
*** Bug 1371402 has been marked as a duplicate of this bug. ***

Comment 17 Steve Speicher 2016-08-31 02:09:23 UTC
Any movement on this? I'm currently totally blocked on dev preview (prod)

Comment 18 Steve Speicher 2016-08-31 12:20:50 UTC
To be clear, my scenario was;

1. I had a project I used for a number of days
2. Deleted the project
3. Created a project
   did a little work
4. Deleted the project (no error)
5. tried to create a project and got the error

I can't find any existence of the project doing oc get project --loglevel=7, etc.

I'm blocked. Is there any script I or an admin can run to get me back to a good place?

Comment 19 Jessica Forrester 2016-08-31 12:34:08 UTC
*** Bug 1371839 has been marked as a duplicate of this bug. ***

Comment 22 Peter Ruan 2016-09-02 06:12:41 UTC
following the steps given in comment #11, QE was able to verify the system can now detect the invalid template and roll back the project creation.

in addition to comment #11, we need to change the master-config.yaml by following this doc (step #4 is a must)
https://docs.openshift.com/enterprise/3.2/admin_guide/managing_projects.html#modifying-the-template-for-new-projects

Expectded output:

[root@ip-172-18-10-247 rpms]# systemctl restart atomic-openshift-master.service
[root@ip-172-18-10-247 rpms]# oc new-project foofoo
Error from server: Internal error occurred: role "notexist" not found
[root@ip-172-18-10-247 rpms]# oc get project foofoo
Error from server: namespaces "foofoo" not found

Comment 26 Corey Daley 2016-09-27 14:34:29 UTC
We are still getting reports of this happening on the Dev Preview cluster.  Is this issue resolved?  Do we have a job that cleans up old projects without rolebindings that runs on a schedule?

Comment 28 Dylan Maassen van den Brink 2016-10-03 12:53:26 UTC
After contacting OpenShift support Jiří pointed out that that I may be experiencing this bug. I personally experienced the following on October 1st 2016. This was on https://console.preview.openshift.com/console/ (The dev preview)

1. I logged into the console at https://console.preview.openshift.com/
2. I clicked 'New Project' and filled in a Name and an Display Name
3. During generation it seemed to have frozen so I reloaded the page
4. The console popped up the way it did initially, just the "Welcome to OpenShift"
5. When I tried to create another new project OpenShift responded with the error the original reporter got ("projectrequests "*****" is forbidden: user Mozoa cannot create more than 1 project(s).")

I've since also tried the commandline application (oc.exe) to see if it would yield different results, here is a screenshot of the output http://i.imgur.com/Y9gVxib.png

Of course I can't speak from the developer's perspective, this is just how it worked for me as a client in the Dev Preview

Comment 29 Stefanie Forrester 2016-10-03 14:36:49 UTC
*** Bug 1380922 has been marked as a duplicate of this bug. ***

Comment 30 Abhishek Gupta 2016-10-03 17:58:01 UTC
Dylan: You should now be able to create a new project.

Comment 31 Henry 2016-10-04 09:05:35 UTC
Hi Team,

I faced the same issue as Dylan 

After contacting OpenShift support Jiří pointed out that that I may be experiencing this bug. I personally experienced the following on October 1st 2016. This was on https://console.preview.openshift.com/console/ (The dev preview)

1. I logged into the console at https://console.preview.openshift.com/
2. I clicked 'New Project' and filled in a Name and an Display Name
3. During generation it seemed to have frozen so I reloaded the page
4. The console popped up the way it did initially, just the "Welcome to OpenShift"
5. When I tried to create another new project OpenShift responded with the error the original reporter got ("projectrequests "*****" is forbidden: user Mozoa cannot create more than 1 project(s).")

I've since also tried the commandline application (oc.exe) to see if it would yield different results

Comment 32 Henry 2016-10-04 09:56:06 UTC
(In reply to Abhishek Gupta from comment #30)
> Dylan: You should now be able to create a new project.

Hi Abhishek, Please can you fix my account aswell. Thanks

Comment 33 Stefanie Forrester 2016-10-04 14:44:09 UTC
Henry, I just checked and I don't see a project owned by user 'Mozoa'. Is the issue resolved? Or did you create the project using a different username?

If the issue is still happening, let me know what your username is. Or if you can remember the name of the project you first tried to make, that would help too. Thanks,

Comment 34 Henry 2016-10-04 14:48:22 UTC
(In reply to Stefanie Forrester from comment #33)
> Henry, I just checked and I don't see a project owned by user 'Mozoa'. Is
> the issue resolved? Or did you create the project using a different username?
> 
> If the issue is still happening, let me know what your username is. Or if
> you can remember the name of the project you first tried to make, that would
> help too. Thanks,

I login account is henryfernandes_redhat; user henryfernandes

and the project I tried to create was henryfernandes_goapp

Comment 35 Abhishek Gupta 2016-10-04 16:20:12 UTC
Henry: You should be able to create a new project now

Comment 36 Henry 2016-10-04 16:23:55 UTC
Thanks Abhishek. It worked, I have created a new project

Comment 37 Abhishek Gupta 2016-10-04 19:25:06 UTC
Corey: We have cleared all projects that have no role bindings. The issue in the code has also been addressed. We will keep an eye out for any new projects that are not auto-cleared and use those for debugging the issue, if it still persists.

Stefanie: The 'mozoa' user account belonged to Dylan and was cleared earlier as well.

Comment 38 James Ng 2016-10-05 01:17:51 UTC
Hi,

Sorry for the trouble but my account 'nghianja' is also having this problem. Please help urgently as I am using the system for an app challenge with a close deadline. Thank you very much.

Best Regards,
James Ng

Comment 39 Martin Etmajer 2016-10-05 12:23:06 UTC
Hi, we are experiencing the same issue since yesterday. Our account: isvintegrate.

Comment 40 Stefanie Forrester 2016-10-05 14:03:18 UTC
James, Martin, I have deleted all the new stuck projects since yesterday. You should be able to create apps again. Apparently this issue is still occurring, so my team will have to watch for this until a permanent solution is reached.

Comment 45 Abhishek Gupta 2016-10-10 21:25:46 UTC
*** Bug 1381038 has been marked as a duplicate of this bug. ***

Comment 46 Abhishek Gupta 2016-10-13 22:36:32 UTC
The fix has been deployed to production and all stuck projects have been cleared.

Comment 47 Bing Li 2016-10-18 08:28:47 UTC
We've verified this bug in online as comment 23, and didn't meet this issue again in online INT/STG/PROD.
And in OCP 3.3, stuck projects would be cleared successfully in our tests. 
So I'll move this bug to verified. Thanks!

Comment 48 Corey Daley 2016-10-19 19:13:32 UTC
If this bug is fixed can we go ahead and close it?

Comment 49 Jessica Forrester 2016-11-09 21:31:49 UTC
*** Bug 1393559 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.