Hide Forgot
Description of problem: On a scaled application with 3 gears (2 NodeJS, 1 MongoDB), the number of deployments to keep is set to 1. However, 1 of the 2 NodeJS gears is retaining multiple deployments. Eventually this leads the app to reach the quota file limit (40,000), preventing further deployments. So far it is always the same gear retaining deployments, the other is correctly retaining a single one. Version-Release number of selected component (if applicable): rhc 1.16.9 How reproducible: Unsure how easily reproducible this is. Steps below show my theory as to what I believe caused it. Steps to Reproduce: 1. Create a scaled application. 2. Set scaling settings to 1 gear minimum. 3. Set configuration to retain 1 deployment. 4. Set scaling settings to 2 gears minimum. 5. Make some deployments, check "2nd" gear's "app-deployment" folder. Actual results: Contains multiple deployments on the 2nd gear. Expected results: Should only contain 1 deployment on both gears. Additional info: Currently resolving by ssh'ing to the gear at fault, changing to the app deployment folder and running "rm -rf 2*" to remove files.
David, are you saying that app-deployments has more than 1 directory named 2013-…?
That's right. But only in my 2nd gear. (In reply to Andy Goldstein from comment #1) > David, are you saying that app-deployments has more than 1 directory named > 2013-…?
Thanks David, I have been able to reproduce what you're seeing. I'm investigating the root cause now. Thanks for letting us know about the issue.
You're more than welcome and I'm kind of glad it wasn't something I did ;-) Good luck with the bug hunt.
Root cause identified; working on a fix now.
https://github.com/openshift/origin-server/pull/4290
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/754840e2b9dd32a1e30e05392af3cfbb195e3a8a Prune from child gear app-deployments dir Modify distribute implementation so it keeps the entire app-deployments dir (excluding the "current" symlink) in sync between the proxy gear and the child gears. Previously, each new deployment would create a new deployment dir on the children in app-deployments without pruning older deployment dirs, eventually filling up a gear's quota. Bug 1038634
It's fixed, verified on devenv_4102.