Bug 1086703

Created attachment 885372 [details]
/var/log/messages

Description of problem:
From time to time pulp-worker service fails and it is impossible to perform any async task.

in /etc/default/pulp_workers:
PULP_CONCURRENCY=4

# systemctl | grep pulp*
pulp_celerybeat.service                          loaded active running   Pulp's Celerybeat
pulp_resource_manager.service                    loaded active running   Pulp Resource Manager
pulp_worker-0.service                            loaded active running   Pulp Worker #0
pulp_worker-1.service                            loaded active running   Pulp Worker #1
pulp_worker-2.service                            loaded active running   Pulp Worker #2
pulp_worker-3.service                            loaded active running   Pulp Worker #3
pulp_workers.service                             loaded active exited    Pulp Celery Workers


# pulp-admin -u admin -p admin rpm repo create --repo-id m1 --feed http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
Successfully created repository [m1]

[root@ip-10-35-136-99 ~]# pulp-admin -u admin -p admin puppet repo sync run  --repo-id m1
+----------------------------------------------------------------------+
                     Synchronizing Repository [m1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


[|]
Waiting to begin...
^C[root@ip-10-35-136-99 ~]systemctl | grep pulp*
pulp_celerybeat.service                          loaded active running   Pulp's Celerybeat
pulp_resource_manager.service                    loaded active running   Pulp Resource Manager
pulp_worker-0.service                            loaded failed failed    Pulp Worker #0
pulp_worker-1.service                            loaded failed failed    Pulp Worker #1
pulp_worker-2.service                            loaded failed failed    Pulp Worker #2
pulp_worker-3.service                            loaded failed failed    Pulp Worker #3
pulp_workers.service                             loaded active exited    Pulp Celery Workers


# systemctl status pulp_worker-0.service
pulp_worker-0.service - Pulp Worker #0
   Loaded: loaded (/run/systemd/system/pulp_worker-0.service; static)
   Active: failed (Result: exit-code) since Fri 2014-04-11 10:42:49 UTC; 2min 0s ago
  Process: 2914 ExecStart=/usr/bin/celery worker -n reserved_resource_worker-0@%%h -A pulp.server.async.app -c 1 --events (code=exited, status=1/FAILURE)
 Main PID: 2914 (code=exited, status=1/FAILURE)

Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: return fmt.format(record)
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: record.message = record.getMessage()
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: msg = msg % self.args
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: TypeError
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: :
Apr 11 10:42:47 ip-10-35-136-99.eu-west-1.compute.internal pulp[2914]: celery.redirected:WARNING: not enough arguments for format string
Apr 11 10:42:49 ip-10-35-136-99.eu-west-1.compute.internal systemd[1]: pulp_worker-0.service: main process exited, code=exited, status=1/FAILURE
Apr 11 10:42:49 ip-10-35-136-99.eu-west-1.compute.internal systemd[1]: Unit pulp_worker-0.service entered failed state.

find attached logs from /valog/messages, pulp and celery

Version-Release number of selected component (if applicable):
pulp-server-2.4.0-0.8.beta.fc20.noarch

How reproducible:


Steps to Reproduce:
1. create repo
2. sync repo
3.

Actual results:
pulp-worker-X fails and it is impossible to perform async tasks.

Expected results:
pulp is expected to perform async calls successfully. 

Additional info: