Celerybeat uses pulp.server.async.scheduler which provides correct reconnect support if either Mongo or Qpid go down and then come back later. For normal outage times, minutes or hours, the reconnect support works fine. For outages that last on the order of days the user will eventually receive the following message: pulp.server.async.scheduler:ERROR: [Errno 24] Too many open files One the user sees that message, reconnect support no longer works, and the celerybeat service would need to be restarted. Something about the reconnect support is using a file descriptor with each reconnect attempt. I'm not sure if it is Qpid or Mongo that causes this, so I'm identifying them both as possible causes. To reproduce: 1. Stop all Pulp services 2. Start Mongo 3. Start Qpid 4. Start celerybeat 5. stop Mongo 6. Stop Qpid 7. Observe the reconnects trying over and over 8. Wait a long time (like overnight) 9. Observe the error message above in the logs
The failing component is likely long-term reconnect with qpidd because there is another error with pulp_celerybeat that occurs if mongod is not available [0]. That other error is experienced on a much shorter timeline (minutes) with a different traceback. By deduction I believe this BZ is likely caused by reconnect support with respect to qpidd. [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1129488
Moved to https://pulp.plan.io/issues/470