Created attachment 894068 [details] Traceback Description of problem: About 20-30% of the time, pulp_celerybeat will fail to start. Version-Release number of selected component (if applicable): pulp-server-2.4.0-0.13.beta.el6.noarch on Centos 6.5 (assuming it occurs on rhel 6.5 as well). How reproducible: 20-30% of the time Steps to Reproduce: 1. Restart the pulp_celerybeat service 2. check 'ps aux | grep celerybeat' Actual results: Sometimes it won't be running Expected results: It is always running Additional info: Attaching the traceback i see when it fails to start
Also to note, i NEVER see this error when running the process in the foreground with: sudo -u apache /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.app --workdir=/var/lib/pulp/celery/ -f /var/log/pulp/celerybeat.log -l INFO --pidfile=/var/run/pulp/celerybeat.pid
I believe this bug could be related to #1096539[0], but I am not sure. I think Brian would be more effective at investigating this than I am. For what it's worth, I had a hard time reproducing this problem on Fedora Rawhide, so it could also be a difference in init system or the version of qpidd. [0] https://bugzilla.redhat.com/show_bug.cgi?id=1096539
Randy: I don't think its related as restarting as when restarting just pulp_celerybeat 10 times in a row, it would fail 2-3 times with that traceback. The entire time qpid and mongo were all up. The failures were not consecutive either.
I also don't think it's related. I'd like to retest using this PR: https://github.com/pulp/pulp/pull/967
Merged
The fix for this bug is included in the pulp-2.4.0-0.14.beta builds.
verified [root@mgmt3 ~]# rpm -q pulp-server pulp-server-2.4.0-0.23.beta.el6.noarch [root@mgmt3 ~]# service pulp_celerybeat restart celery init v10.0. Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat Restarting celery periodic task scheduler Stopping pulp_celerybeat... OK Starting pulp_celerybeat... [root@mgmt3 ~]# ps aux | grep celerybeat apache 21626 5.6 0.3 727540 25512 ? Sl 06:27 0:00 /usr/bin/python /usr/bin/celery beat --scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/lib/pulp/celery/ -f /var/log/pulp/celerybeat.log -l INFO --detach --pidfile=/var/run/pulp/celerybeat.pid root 21635 0.0 0.0 103252 816 pts/3 S+ 06:27 0:00 grep celerybeat [root@mgmt3 ~]#
This has been fixed in Pulp 2.4.0-1.