Bug 572722

Summary: Rails leaks file descriptors when making database connections
Product: [Fedora] Fedora EPEL Reporter: Todd Zullinger <tmz>
Component: rubygem-activerecordAssignee: Michael Stahnke <mastahnke>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: el5CC: hbrock, mastahnke, urkedal, vanmeeuwen+fedora
Target Milestone: ---Flags: mastahnke: needinfo?
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-06 10:38:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch for leaking fd's with store configs none

Description Todd Zullinger 2010-03-11 21:32:52 UTC 
After updating to puppet-0.25.4 in EPEL, I noticed some of my production systems ran out of open file descriptors after a day or so.  After much head scratching I narrowed this down to something in the rails stack leaving open database connections when storeconfigs was enabled.  The puppetmasterd process leaked one fd every time it compiled a catalog for a node.

I had not noticed this on my test machines because they simply didn't have enough volume to exceed the 1024 open files limit within a week (as logrotate restarts puppetmasterd that often).

I built rails 2.3.5 packages for EL-5 on my test systems and eventually in production as well and that seems to have corrected the problem.  Of course, we can't update rails in EPEL  I tried to poke through the rails.git logs looking for any obvious candidates for cherry-picking to 2.1.1, but I'm not very familiar with the rails code and no pink ponies jumped out at me.

I suspect a lot of folks will run into this and would love to try and get it fixed in EPEL, be it via monkey patching puppet or activerecord.  Any help on that would be greatly appreciated.

Reproducing was simple for me.  I set "storeconfigs = true" in puppet.conf and watched the open file count rise with each puppetd run.  The problem occurs with both the default sqlite and mysql db adapters.  It happened both with webrick and mongrel.
Comment 1 Todd Zullinger 2010-05-19 17:32:36 UTC 
I also filed this upstream with puppet:

   http://projects.puppetlabs.com/issues/3693

Is there any chance we could ship update rails packages in parallel with the 2.1 packages?
Comment 2 Jeroen van Meeuwen 2010-05-20 08:52:56 UTC 
Created attachment 415348 [details]
Patch for leaking fd's with store configs

Would you be able to test the situation with the attached patch? I've tried and end up with the database being disconnected and the connection being cleaned up after the compiler is done storing configurations;

[root@master puppet]# lsof $(for pid in `pgrep puppetmaster`; do echo "-p $pid"; done) | wc -l
166
[root@master puppet]# lsof $(for pid in `pgrep puppetmaster`; do echo "-p $pid"; done) | wc -l
165
Comment 3 Jeroen van Meeuwen 2010-05-20 09:12:21 UTC 
Uch. wrong paste ;-)
Comment 4 Jeroen van Meeuwen 2010-05-20 09:41:12 UTC 
Nevermind the patch, over a longer period of time it results in a MySQL::Error: MYSQL server has gone away
Comment 5 Michael Stahnke 2014-10-23 04:37:57 UTC 
Does this still happen in 2.7 series?

I'm updating the rails stack 2.3.18, so that could help.

Beyond that, there is PuppetDB now as the primary way to do this.
Comment 6 Fedora End Of Life 2017-04-06 10:38:55 UTC 
Fedora EPEL 5 changed to end-of-life (EOL) status on 2017-03-31. Fedora EPEL 5
is no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora
or Fedora EPEL, please feel free to reopen this bug against that version. If
you are unable to reopen this bug, please file a new report against the current
release. If you experience problems, please add a comment to this bug.

Thank you for reporting this bug and we are sorry it could not be fixed.