After updating to puppet-0.25.4 in EPEL, I noticed some of my production systems ran out of open file descriptors after a day or so. After much head scratching I narrowed this down to something in the rails stack leaving open database connections when storeconfigs was enabled. The puppetmasterd process leaked one fd every time it compiled a catalog for a node. I had not noticed this on my test machines because they simply didn't have enough volume to exceed the 1024 open files limit within a week (as logrotate restarts puppetmasterd that often). I built rails 2.3.5 packages for EL-5 on my test systems and eventually in production as well and that seems to have corrected the problem. Of course, we can't update rails in EPEL I tried to poke through the rails.git logs looking for any obvious candidates for cherry-picking to 2.1.1, but I'm not very familiar with the rails code and no pink ponies jumped out at me. I suspect a lot of folks will run into this and would love to try and get it fixed in EPEL, be it via monkey patching puppet or activerecord. Any help on that would be greatly appreciated. Reproducing was simple for me. I set "storeconfigs = true" in puppet.conf and watched the open file count rise with each puppetd run. The problem occurs with both the default sqlite and mysql db adapters. It happened both with webrick and mongrel.
I also filed this upstream with puppet: http://projects.puppetlabs.com/issues/3693 Is there any chance we could ship update rails packages in parallel with the 2.1 packages?
Created attachment 415348 [details] Patch for leaking fd's with store configs Would you be able to test the situation with the attached patch? I've tried and end up with the database being disconnected and the connection being cleaned up after the compiler is done storing configurations; [root@master puppet]# lsof $(for pid in `pgrep puppetmaster`; do echo "-p $pid"; done) | wc -l 166 [root@master puppet]# lsof $(for pid in `pgrep puppetmaster`; do echo "-p $pid"; done) | wc -l 165
Uch. wrong paste ;-)
Nevermind the patch, over a longer period of time it results in a MySQL::Error: MYSQL server has gone away
Does this still happen in 2.7 series? I'm updating the rails stack 2.3.18, so that could help. Beyond that, there is PuppetDB now as the primary way to do this.
Fedora EPEL 5 changed to end-of-life (EOL) status on 2017-03-31. Fedora EPEL 5 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora or Fedora EPEL, please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.