Bug 1132130

+++ This bug was initially created as a clone of Bug #1121102 +++

Description of problem:
I observed that when the server is under heavy load the agent replies seems to update the task in the wrong order which leads to the fact that task appear to never finish.

>  Waiting exceeded 20 second(s): {u'exception': None, u'task_type': None, u'_href': u'/pulp/api/v2/tasks/203ece2b-8dd9-4505-bfe3-48940beb430b/', u'task_id': u'203ece2b-8dd9-4505-bfe3-48940beb430b', u'tags': [u'pulp:consumer:ConsumerAuthTest_consumer', u'pulp:repository:zoo', u'pulp:repository_distributor:zoo_distributor', u'pulp:action:agent_unbind'], u'finish_time': u'2014-07-16T11:56:50Z', u'_ns': u'task_status', u'start_time': u'2014-07-16T11:56:50Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': None, u'queue': u'agent', u'state': u'running', u'result': {u'reboot': {u'scheduled': False, u'details': {}}, u'details': {u'general_distributor': [], u'yum_distributor': [], u'rpm': {u'details': [{u'vendor': u'Fedora Project', u'name': u'basesystem', u'epoch': 0, u'version': u'10.0', u'release': u'8.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'perl-Carp', u'epoch': 0, u'version': u'1.26', u'release': u'243.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'perl-Git', u'epoch': 0, u'version': u'1.8.3.1', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'python-oauth2', u'epoch': 0, u'version': u'1.5.211', u'release': u'4.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'python-qpid', u'epoch': 0, u'version': u'0.24', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'tzdata', u'epoch': 0, u'version': u'2013c', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'boost-context', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-iostreams', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-program-options', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-test', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-timer', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'chkconfig', u'epoch': 0, u'version': u'1.3.60', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'gpm-libs', u'epoch': 0, u'version': u'1.20.6', u'release': u'33.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libattr', u'epoch': 0, u'version': u'2.4.46', u'release': u'10.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libdb', u'epoch': 0, u'version': u'5.3.21', u'release': u'11.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libffi', u'epoch': 0, u'version': u'3.0.13', u'release': u'4.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libgpg-error', u'epoch': 0, u'version': u'1.11', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libibverbs', u'epoch': 0, u'version': u'1.1.7', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libstdc++', u'epoch': 0, u'version': u'4.8.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libxml2-python', u'epoch': 0, u'version': u'2.9.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'lua', u'epoch': 0, u'version': u'5.1.4', u'release': u'12.fc19', u'arch': u'x86_64'}, {u'vendor': None, u'name': u'm2crypto', u'epoch': 0, u'version': u'0.21.1.pulp', u'release': u'8.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'perl-PathTools', u'epoch': 0, u'version': u'3.40', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'python-rhsm', u'epoch': 0, u'version': u'1.10.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'python-simplejson', u'epoch': 0, u'version': u'3.2.0', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'qpid-cpp-client', u'epoch': 0, u'version': u'0.24', u'release': u'3.fc19.1', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'qpid-cpp-server', u'epoch': 0, u'version': u'0.24', u'release': u'3.fc19.1', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'readline', u'epoch': 0, u'version': u'6.2', u'release': u'6.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'tcp_wrappers-libs', u'epoch': 0, u'version': u'7.6', u'release': u'73.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'xerces-c', u'epoch': 0, u'version': u'3.1.1', u'release': u'4.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'zlib', u'epoch': 0, u'version': u'1.2.7', u'release': u'10.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'perl', u'epoch': 4, u'version': u'5.16.3', u'release': u'265.fc19', u'arch': u'x86_64'}], u'succeeded': True}}, u'succeeded': True, u'num_changes': 1}, u'error': None, u'_id': {u'$oid': u'53c6688113038b19d15dd0d7'}, u'id': u'53c6688113038b19d15dd0d7'}

Notice the 'result' is set and contains a summary report  as if the task had completed but the state=running.
Seems like that this could be caused by multiple processes in apache consuming replies.

If I set processes=1 in /etc/httpd/conf.d/pulp.conf the problem goes away.


Version-Release number of selected component (if applicable):
2.4.0-0.24.beta

How reproducible:
when the server is under heavy load there is 5% of failure from 300 times run the testcase.

Steps to Reproduce:
1. server should be under heavy load
2. bind\unbind the repo multiple times until you'll hit the situation when the task will never finish.
3.

Actual results:
task never finishes

Expected results:
task finishes

Additional info:

--- Additional comment from Martin Minar on 2014-07-21 09:43:47 EDT ---

Setting high priority as this bug degrades multiprocessing capability of consumer agent requests.

--- Additional comment from Ina Panova on 2014-08-05 09:18:15 EDT ---

https://github.com/RedHatQE/pulp-automation/commit/9e3d3ec992d3f02f6d6a1dbff897e1104fee7c00

Created temporary workaround to make real and fake consumer work properly.