Bug 1121102

Description of problem:
I observed that when the server is under heavy load the agent replies seems to update the task in the wrong order which leads to the fact that task appear to never finish.

>  Waiting exceeded 20 second(s): {u'exception': None, u'task_type': None, u'_href': u'/pulp/api/v2/tasks/203ece2b-8dd9-4505-bfe3-48940beb430b/', u'task_id': u'203ece2b-8dd9-4505-bfe3-48940beb430b', u'tags': [u'pulp:consumer:ConsumerAuthTest_consumer', u'pulp:repository:zoo', u'pulp:repository_distributor:zoo_distributor', u'pulp:action:agent_unbind'], u'finish_time': u'2014-07-16T11:56:50Z', u'_ns': u'task_status', u'start_time': u'2014-07-16T11:56:50Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': None, u'queue': u'agent', u'state': u'running', u'result': {u'reboot': {u'scheduled': False, u'details': {}}, u'details': {u'general_distributor': [], u'yum_distributor': [], u'rpm': {u'details': [{u'vendor': u'Fedora Project', u'name': u'basesystem', u'epoch': 0, u'version': u'10.0', u'release': u'8.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'perl-Carp', u'epoch': 0, u'version': u'1.26', u'release': u'243.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'perl-Git', u'epoch': 0, u'version': u'1.8.3.1', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'python-oauth2', u'epoch': 0, u'version': u'1.5.211', u'release': u'4.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'python-qpid', u'epoch': 0, u'version': u'0.24', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'tzdata', u'epoch': 0, u'version': u'2013c', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'boost-context', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-iostreams', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-program-options', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-test', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-timer', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'chkconfig', u'epoch': 0, u'version': u'1.3.60', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'gpm-libs', u'epoch': 0, u'version': u'1.20.6', u'release': u'33.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libattr', u'epoch': 0, u'version': u'2.4.46', u'release': u'10.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libdb', u'epoch': 0, u'version': u'5.3.21', u'release': u'11.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libffi', u'epoch': 0, u'version': u'3.0.13', u'release': u'4.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libgpg-error', u'epoch': 0, u'version': u'1.11', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libibverbs', u'epoch': 0, u'version': u'1.1.7', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libstdc++', u'epoch': 0, u'version': u'4.8.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libxml2-python', u'epoch': 0, u'version': u'2.9.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'lua', u'epoch': 0, u'version': u'5.1.4', u'release': u'12.fc19', u'arch': u'x86_64'}, {u'vendor': None, u'name': u'm2crypto', u'epoch': 0, u'version': u'0.21.1.pulp', u'release': u'8.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'perl-PathTools', u'epoch': 0, u'version': u'3.40', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'python-rhsm', u'epoch': 0, u'version': u'1.10.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'python-simplejson', u'epoch': 0, u'version': u'3.2.0', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'qpid-cpp-client', u'epoch': 0, u'version': u'0.24', u'release': u'3.fc19.1', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'qpid-cpp-server', u'epoch': 0, u'version': u'0.24', u'release': u'3.fc19.1', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'readline', u'epoch': 0, u'version': u'6.2', u'release': u'6.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'tcp_wrappers-libs', u'epoch': 0, u'version': u'7.6', u'release': u'73.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'xerces-c', u'epoch': 0, u'version': u'3.1.1', u'release': u'4.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'zlib', u'epoch': 0, u'version': u'1.2.7', u'release': u'10.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'perl', u'epoch': 4, u'version': u'5.16.3', u'release': u'265.fc19', u'arch': u'x86_64'}], u'succeeded': True}}, u'succeeded': True, u'num_changes': 1}, u'error': None, u'_id': {u'$oid': u'53c6688113038b19d15dd0d7'}, u'id': u'53c6688113038b19d15dd0d7'}

Notice the 'result' is set and contains a summary report  as if the task had completed but the state=running.
Seems like that this could be caused by multiple processes in apache consuming replies.

If I set processes=1 in /etc/httpd/conf.d/pulp.conf the problem goes away.


Version-Release number of selected component (if applicable):
2.4.0-0.24.beta

How reproducible:
when the server is under heavy load there is 5% of failure from 300 times run the testcase.

Steps to Reproduce:
1. server should be under heavy load
2. bind\unbind the repo multiple times until you'll hit the situation when the task will never finish.
3.

Actual results:
task never finishes

Expected results:
task finishes

Additional info: