Created attachment 894068 [details]
Description of problem:
About 20-30% of the time, pulp_celerybeat will fail to start.
Version-Release number of selected component (if applicable):
on Centos 6.5 (assuming it occurs on rhel 6.5 as well).
20-30% of the time
Steps to Reproduce:
1. Restart the pulp_celerybeat service
2. check 'ps aux | grep celerybeat'
Sometimes it won't be running
It is always running
Attaching the traceback i see when it fails to start
Also to note, i NEVER see this error when running the process in the foreground with:
sudo -u apache /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.app --workdir=/var/lib/pulp/celery/ -f /var/log/pulp/celerybeat.log -l INFO --pidfile=/var/run/pulp/celerybeat.pid
I believe this bug could be related to #1096539, but I am not sure. I think Brian would be more effective at investigating this than I am.
For what it's worth, I had a hard time reproducing this problem on Fedora Rawhide, so it could also be a difference in init system or the version of qpidd.
Randy: I don't think its related as restarting as when restarting just pulp_celerybeat 10 times in a row, it would fail 2-3 times with that traceback.
The entire time qpid and mongo were all up. The failures were not consecutive either.
I also don't think it's related. I'd like to retest using this PR:
The fix for this bug is included in the pulp-2.4.0-0.14.beta builds.
[root@mgmt3 ~]# rpm -q pulp-server
[root@mgmt3 ~]# service pulp_celerybeat restart
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... OK
[root@mgmt3 ~]# ps aux | grep celerybeat
apache 21626 5.6 0.3 727540 25512 ? Sl 06:27 0:00 /usr/bin/python /usr/bin/celery beat --scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/lib/pulp/celery/ -f /var/log/pulp/celerybeat.log -l INFO --detach --pidfile=/var/run/pulp/celerybeat.pid
root 21635 0.0 0.0 103252 816 pts/3 S+ 06:27 0:00 grep celerybeat
This has been fixed in Pulp 2.4.0-1.