Bug 572722 - Rails leaks file descriptors when making database connections [NEEDINFO]
Summary: Rails leaks file descriptors when making database connections
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: rubygem-activerecord
Version: el5
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Michael Stahnke
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-03-11 21:32 UTC by Todd Zullinger
Modified: 2017-04-06 10:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-06 10:38:55 UTC
mastahnke: needinfo?


Attachments (Terms of Use)
Patch for leaking fd's with store configs (1.02 KB, patch)
2010-05-20 08:52 UTC, Jeroen van Meeuwen
no flags Details | Diff

Description Todd Zullinger 2010-03-11 21:32:52 UTC
After updating to puppet-0.25.4 in EPEL, I noticed some of my production systems ran out of open file descriptors after a day or so.  After much head scratching I narrowed this down to something in the rails stack leaving open database connections when storeconfigs was enabled.  The puppetmasterd process leaked one fd every time it compiled a catalog for a node.

I had not noticed this on my test machines because they simply didn't have enough volume to exceed the 1024 open files limit within a week (as logrotate restarts puppetmasterd that often).

I built rails 2.3.5 packages for EL-5 on my test systems and eventually in production as well and that seems to have corrected the problem.  Of course, we can't update rails in EPEL  I tried to poke through the rails.git logs looking for any obvious candidates for cherry-picking to 2.1.1, but I'm not very familiar with the rails code and no pink ponies jumped out at me.

I suspect a lot of folks will run into this and would love to try and get it fixed in EPEL, be it via monkey patching puppet or activerecord.  Any help on that would be greatly appreciated.

Reproducing was simple for me.  I set "storeconfigs = true" in puppet.conf and watched the open file count rise with each puppetd run.  The problem occurs with both the default sqlite and mysql db adapters.  It happened both with webrick and mongrel.

Comment 1 Todd Zullinger 2010-05-19 17:32:36 UTC
I also filed this upstream with puppet:

   http://projects.puppetlabs.com/issues/3693

Is there any chance we could ship update rails packages in parallel with the 2.1 packages?

Comment 2 Jeroen van Meeuwen 2010-05-20 08:52:56 UTC
Created attachment 415348 [details]
Patch for leaking fd's with store configs

Would you be able to test the situation with the attached patch? I've tried and end up with the database being disconnected and the connection being cleaned up after the compiler is done storing configurations;

[root@master puppet]# lsof $(for pid in `pgrep puppetmaster`; do echo "-p $pid"; done) | wc -l
166
[root@master puppet]# lsof $(for pid in `pgrep puppetmaster`; do echo "-p $pid"; done) | wc -l
165

Comment 3 Jeroen van Meeuwen 2010-05-20 09:12:21 UTC
Uch. wrong paste ;-)

Comment 4 Jeroen van Meeuwen 2010-05-20 09:41:12 UTC
Nevermind the patch, over a longer period of time it results in a MySQL::Error: MYSQL server has gone away

Comment 5 Michael Stahnke 2014-10-23 04:37:57 UTC
Does this still happen in 2.7 series?

I'm updating the rails stack 2.3.18, so that could help.

Beyond that, there is PuppetDB now as the primary way to do this.

Comment 6 Fedora End Of Life 2017-04-06 10:38:55 UTC
Fedora EPEL 5 changed to end-of-life (EOL) status on 2017-03-31. Fedora EPEL 5
is no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora
or Fedora EPEL, please feel free to reopen this bug against that version. If
you are unable to reopen this bug, please file a new report against the current
release. If you experience problems, please add a comment to this bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.