Bug 1329221

Summary:	Can't deploy due hitting quota of RCs (RCs not being pruned?)
Product:	OpenShift Online	Reporter:	Steve Speicher <sspeiche>
Component:	Deployments	Assignee:	Dan Mace <dmace>
Status:	CLOSED CURRENTRELEASE	QA Contact:	zhou ying <yinzhou>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.x	CC:	aos-bugs, sspeiche
Target Milestone:	---	Keywords:	Reopened
Target Release:	---	Flags:	sspeiche: needinfo+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-05-23 15:08:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Steve Speicher 2016-04-21 12:47:58 UTC

Description of problem:
Been running for about 24 hours, have hit quota of RCs and can't deploy an app now.

Version-Release number of selected component (if applicable):
dev-preview-int.openshift.com
OpenShift Master:
v3.2.0.16
Kubernetes Master:
v1.2.0-36-g4a3f9c5

How reproducible:
Just needs > 20 deployments


Steps to Reproduce:
1.Deploy an app
2.repeat 20 times

Actual results:
No deployment/RC is created and no error message. Have to go to events page to see the error.


Expected results:
Auto-pruner cleaning up RCs.
UI shows reason why deployment fails.

Additional info:

Error message received in the events page:

8:42:48 AM	rails-postgresql-example	Deployment Config	Warning	Deployment creation failed 	Couldn't deploy version 10: replicationcontrollers "rails-postgresql-example-10" is forbidden: Exceeded quota: object-counts, requested: replicationcontrollers=1, used: replicationcontrollers=20, limited: replicationcontrollers=20
4 times in the last

Comment 1 Dan Mace 2016-04-21 12:55:27 UTC


*** This bug has been marked as a duplicate of bug 1325796 ***

Comment 2 Dan Mace 2016-04-22 21:13:44 UTC

The pruning job for builds and deployments was broken. Ops made a fix and this can be re-tested after Monday (April 25).

Comment 3 Dan Mace 2016-04-22 21:16:21 UTC

FYI, the pruning interval for these resources is currently once a day at 1AM.

Comment 4 zhou ying 2016-04-26 01:32:00 UTC

Confirmed with the latest version on dev-preview-int.openshift.com, the  pruning job works fine now. 

[root@zhouy roottest]# oc get po
NAME             READY     STATUS    RESTARTS   AGE
hooks-23-28fvn   1/1       Running   1          7h
[root@zhouy roottest]# oc get rc
NAME       DESIRED   CURRENT   AGE
hooks-18   0         0         21h
hooks-19   0         0         21h
hooks-20   0         0         21h
hooks-23   1         1         7h