Bug 1096289 - Restarting pulp_celerybeat sometimes fails
Summary: Restarting pulp_celerybeat sometimes fails
Alias: None
Product: Pulp
Classification: Retired
Component: async/tasks   
(Show other bugs)
Version: 2.4.0
Hardware: Unspecified Unspecified
Target Milestone: ---
: 2.4.0
Assignee: Brian Bouterse
QA Contact: Preethi Thomas
Keywords: Triaged
Depends On:
Blocks: 950743
TreeView+ depends on / blocked
Reported: 2014-05-09 15:14 UTC by Justin Sherrill
Modified: 2014-08-20 03:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-08-09 06:55:36 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Traceback (5.39 KB, text/plain)
2014-05-09 15:14 UTC, Justin Sherrill
no flags Details

Description Justin Sherrill 2014-05-09 15:14:58 UTC
Created attachment 894068 [details]

Description of problem:

About 20-30% of the time, pulp_celerybeat will fail to start.

Version-Release number of selected component (if applicable):

on Centos 6.5 (assuming it occurs on rhel 6.5 as well).

How reproducible:
20-30% of the time

Steps to Reproduce:
1. Restart the pulp_celerybeat service
2. check 'ps aux | grep celerybeat'

Actual results:
Sometimes it won't be running

Expected results:
It is always running

Additional info:

Attaching the traceback i see when it fails to start

Comment 1 Justin Sherrill 2014-05-09 15:16:44 UTC
Also to note, i NEVER see this error when running the process in the foreground with:

sudo -u apache /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.app --workdir=/var/lib/pulp/celery/ -f /var/log/pulp/celerybeat.log -l INFO --pidfile=/var/run/pulp/celerybeat.pid

Comment 2 Randy Barlow 2014-05-12 14:43:36 UTC
I believe this bug could be related to #1096539[0], but I am not sure. I think Brian would be more effective at investigating this than I am.

For what it's worth, I had a hard time reproducing this problem on Fedora Rawhide, so it could also be a difference in init system or the version of qpidd.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1096539

Comment 3 Justin Sherrill 2014-05-12 14:54:48 UTC
Randy:  I don't think its related as restarting as when restarting just pulp_celerybeat 10 times in a row, it would fail 2-3 times with that traceback.

The entire time qpid and mongo were all up.  The failures were not consecutive either.

Comment 4 Brian Bouterse 2014-05-15 15:13:44 UTC
I also don't think it's related.  I'd like to retest using this PR:


Comment 5 Brian Bouterse 2014-05-15 15:27:24 UTC

Comment 6 Randy Barlow 2014-05-15 17:00:12 UTC
The fix for this bug is included in the pulp-2.4.0-0.14.beta builds.

Comment 7 Preethi Thomas 2014-07-01 10:31:51 UTC
[root@mgmt3 ~]# rpm -q pulp-server

[root@mgmt3 ~]# service pulp_celerybeat restart
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... OK
Starting pulp_celerybeat...
[root@mgmt3 ~]# ps aux | grep celerybeat
apache   21626  5.6  0.3 727540 25512 ?        Sl   06:27   0:00 /usr/bin/python /usr/bin/celery beat --scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/lib/pulp/celery/ -f /var/log/pulp/celerybeat.log -l INFO --detach --pidfile=/var/run/pulp/celerybeat.pid
root     21635  0.0  0.0 103252   816 pts/3    S+   06:27   0:00 grep celerybeat
[root@mgmt3 ~]#

Comment 8 Randy Barlow 2014-08-09 06:55:36 UTC
This has been fixed in Pulp 2.4.0-1.

Note You need to log in before you can comment on or make changes to this bug.