Bug 1038753

Summary:	Scaled app, deployments retained over configured limit on a single gear [Upstream]
Product:	OpenShift Container Platform	Reporter:	Brenton Leanhardt <bleanhar>
Component:	ImageStreams	Assignee:	Luke Meyer <lmeyer>
Status:	CLOSED ERRATA	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.0.0	CC:	agoldste, dave, gpei
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Windows
Whiteboard:
Fixed In Version:	rubygem-openshift-origin-node-1.17.5.3-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1038634	Environment:
Last Closed:	2014-01-13 14:55:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:	1038634
Bug Blocks:

Description Brenton Leanhardt 2013-12-05 18:21:29 UTC

+++ This bug was initially created as a clone of Bug #1038634 +++

Description of problem:
On a scaled application with 3 gears (2 NodeJS, 1 MongoDB), the number of deployments to keep is set to 1.  However, 1 of the 2 NodeJS gears is retaining multiple deployments.  Eventually this leads the app to reach the quota file limit (40,000), preventing further deployments.

So far it is always the same gear retaining deployments, the other is correctly retaining a single one.

Version-Release number of selected component (if applicable):
rhc 1.16.9

How reproducible:
Unsure how easily reproducible this is.  Steps below show my theory as to what I believe caused it.

Steps to Reproduce:
1. Create a scaled application.
2. Set scaling settings to 1 gear minimum.
3. Set configuration to retain 1 deployment.
4. Set scaling settings to 2 gears minimum.
5. Make some deployments, check "2nd" gear's "app-deployment" folder.

Actual results:
Contains multiple deployments on the 2nd gear.

Expected results:
Should only contain 1 deployment on both gears.

Additional info:
Currently resolving by ssh'ing to the gear at fault, changing to the app deployment folder and running "rm -rf 2*" to remove files.

--- Additional comment from Andy Goldstein on 2013-12-05 09:51:41 EST ---

David, are you saying that app-deployments has more than 1 directory named 2013-…?

--- Additional comment from David Boyer on 2013-12-05 09:53:38 EST ---

That's right. But only in my 2nd gear.

(In reply to Andy Goldstein from comment #1)
> David, are you saying that app-deployments has more than 1 directory named
> 2013-…?

--- Additional comment from Andy Goldstein on 2013-12-05 10:10:16 EST ---

Thanks David, I have been able to reproduce what you're seeing. I'm investigating the root cause now. Thanks for letting us know about the issue.

--- Additional comment from David Boyer on 2013-12-05 10:12:40 EST ---

You're more than welcome and I'm kind of glad it wasn't something I did ;-) Good luck with the bug hunt.

--- Additional comment from Andy Goldstein on 2013-12-05 11:06:47 EST ---

Root cause identified; working on a fix now.

--- Additional comment from Andy Goldstein on 2013-12-05 11:49:34 EST ---

https://github.com/openshift/origin-server/pull/4290

Comment 2 Luke Meyer 2013-12-23 16:51:48 UTC

next build of rubygem-openshift-origin-node
https://github.com/openshift/enterprise-server/pull/176

origin-server:
commit 754840e2b9dd32a1e30e05392af3cfbb195e3a8a
Author: Andy Goldstein <andy.goldstein>
Date:   Thu Dec 5 11:27:59 2013 -0500

    Prune from child gear app-deployments dir
    
    Modify distribute implementation so it keeps the entire app-deployments
    dir (excluding the "current" symlink) in sync between the proxy gear
    and the child gears. Previously, each new deployment would create a new
    deployment dir on the children in app-deployments without pruning older
    deployment dirs, eventually filling up a gear's quota.
    
    Bug 1038634

Comment 3 Gaoyun Pei 2013-12-24 08:46:25 UTC

Verify this bug with rubygem-openshift-origin-node-1.17.5.3-1

1. Create a scaled application.
2. Set scaling settings to 1 gear minimum.
3. Set configuration to retain 1 deployment.
4. Set scaling settings to 2 gears minimum.
5. Make some deployments, check "2nd" gear's "app-deployment" folder.

Only 1 deployment record under the folder.

Comment 5 errata-xmlrpc 2014-01-13 14:55:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0020.html