Bug 1121102 - Agent replies updating the task out of order under heavy load
Summary: Agent replies updating the task out of order under heavy load
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
medium
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Sachin Ghai
URL:
Whiteboard:
Depends On:
Blocks: sat6-pulp-future 1131719 1132130
TreeView+ depends on / blocked
 
Reported: 2014-07-18 11:10 UTC by Ina Panova
Modified: 2018-09-19 15:23 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1132130 (view as bug list)
Environment:
Last Closed: 2015-07-27 14:33:36 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 474 Normal CLOSED - CURRENTRELEASE Agent replies updating the task out of order under heavy load Never

Description Ina Panova 2014-07-18 11:10:14 UTC
Description of problem:
I observed that when the server is under heavy load the agent replies seems to update the task in the wrong order which leads to the fact that task appear to never finish.

>  Waiting exceeded 20 second(s): {u'exception': None, u'task_type': None, u'_href': u'/pulp/api/v2/tasks/203ece2b-8dd9-4505-bfe3-48940beb430b/', u'task_id': u'203ece2b-8dd9-4505-bfe3-48940beb430b', u'tags': [u'pulp:consumer:ConsumerAuthTest_consumer', u'pulp:repository:zoo', u'pulp:repository_distributor:zoo_distributor', u'pulp:action:agent_unbind'], u'finish_time': u'2014-07-16T11:56:50Z', u'_ns': u'task_status', u'start_time': u'2014-07-16T11:56:50Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': None, u'queue': u'agent', u'state': u'running', u'result': {u'reboot': {u'scheduled': False, u'details': {}}, u'details': {u'general_distributor': [], u'yum_distributor': [], u'rpm': {u'details': [{u'vendor': u'Fedora Project', u'name': u'basesystem', u'epoch': 0, u'version': u'10.0', u'release': u'8.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'perl-Carp', u'epoch': 0, u'version': u'1.26', u'release': u'243.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'perl-Git', u'epoch': 0, u'version': u'1.8.3.1', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'python-oauth2', u'epoch': 0, u'version': u'1.5.211', u'release': u'4.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'python-qpid', u'epoch': 0, u'version': u'0.24', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'tzdata', u'epoch': 0, u'version': u'2013c', u'release': u'1.fc19', u'arch': u'noarch'}, {u'vendor': u'Fedora Project', u'name': u'boost-context', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-iostreams', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-program-options', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-test', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'boost-timer', u'epoch': 0, u'version': u'1.53.0', u'release': u'14.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'chkconfig', u'epoch': 0, u'version': u'1.3.60', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'gpm-libs', u'epoch': 0, u'version': u'1.20.6', u'release': u'33.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libattr', u'epoch': 0, u'version': u'2.4.46', u'release': u'10.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libdb', u'epoch': 0, u'version': u'5.3.21', u'release': u'11.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libffi', u'epoch': 0, u'version': u'3.0.13', u'release': u'4.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libgpg-error', u'epoch': 0, u'version': u'1.11', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libibverbs', u'epoch': 0, u'version': u'1.1.7', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libstdc++', u'epoch': 0, u'version': u'4.8.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'libxml2-python', u'epoch': 0, u'version': u'2.9.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'lua', u'epoch': 0, u'version': u'5.1.4', u'release': u'12.fc19', u'arch': u'x86_64'}, {u'vendor': None, u'name': u'm2crypto', u'epoch': 0, u'version': u'0.21.1.pulp', u'release': u'8.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'perl-PathTools', u'epoch': 0, u'version': u'3.40', u'release': u'3.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'python-rhsm', u'epoch': 0, u'version': u'1.10.1', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'python-simplejson', u'epoch': 0, u'version': u'3.2.0', u'release': u'1.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'qpid-cpp-client', u'epoch': 0, u'version': u'0.24', u'release': u'3.fc19.1', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'qpid-cpp-server', u'epoch': 0, u'version': u'0.24', u'release': u'3.fc19.1', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'readline', u'epoch': 0, u'version': u'6.2', u'release': u'6.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'tcp_wrappers-libs', u'epoch': 0, u'version': u'7.6', u'release': u'73.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'xerces-c', u'epoch': 0, u'version': u'3.1.1', u'release': u'4.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'zlib', u'epoch': 0, u'version': u'1.2.7', u'release': u'10.fc19', u'arch': u'x86_64'}, {u'vendor': u'Fedora Project', u'name': u'perl', u'epoch': 4, u'version': u'5.16.3', u'release': u'265.fc19', u'arch': u'x86_64'}], u'succeeded': True}}, u'succeeded': True, u'num_changes': 1}, u'error': None, u'_id': {u'$oid': u'53c6688113038b19d15dd0d7'}, u'id': u'53c6688113038b19d15dd0d7'}

Notice the 'result' is set and contains a summary report  as if the task had completed but the state=running.
Seems like that this could be caused by multiple processes in apache consuming replies.

If I set processes=1 in /etc/httpd/conf.d/pulp.conf the problem goes away.


Version-Release number of selected component (if applicable):
2.4.0-0.24.beta

How reproducible:
when the server is under heavy load there is 5% of failure from 300 times run the testcase.

Steps to Reproduce:
1. server should be under heavy load
2. bind\unbind the repo multiple times until you'll hit the situation when the task will never finish.
3.

Actual results:
task never finishes

Expected results:
task finishes

Additional info:

Comment 3 Jeff Ortel 2014-11-05 21:38:08 UTC
https://github.com/pulp/pulp/pull/1286

Comment 4 Chris Duryee 2014-12-23 20:53:05 UTC
fixed in pulp 2.6.0-0.2.beta

Comment 5 Preethi Thomas 2015-02-04 14:43:37 UTC
please provide some verification steps.

Comment 6 Brian Bouterse 2015-02-28 22:13:28 UTC
Moved to https://pulp.plan.io/issues/474

Comment 7 RHEL Product and Program Management 2015-03-04 11:24:22 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 9 pulp-infra@redhat.com 2015-04-23 16:40:33 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 10 Bryan Kearney 2015-04-27 20:20:42 UTC
The upstream bugs states this was closed in 2.6.0. Movig to ON_QA since Satellite 6.1 is delivering 2.6.0.

Comment 12 Jeff Ortel 2015-06-15 14:39:36 UTC
(In reply to Preethi Thomas from comment #5)
> please provide some verification steps.

This issue can only be reproduced and verified using the automated testing suite.  It does not occur in the real world.  Suggest you discuss with Ina.

Comment 13 Sachin Ghai 2015-07-21 08:08:54 UTC
@Ina: Could you please advise/help to verify this bz ?  thanks

Comment 14 Ina Panova 2015-07-22 12:51:36 UTC
Hi Sachin, it is not possible to do it manually. There is a testcase that would help [0]
[0] https://github.com/RedHatQE/pulp-automation/blob/master/tests/consumer_agent_tests/test_09_consumer_heavy_load_bz1121102.py

Comment 15 Corey Welton 2015-07-27 14:33:36 UTC
Closing this as an Upstream fix.


Note You need to log in before you can comment on or make changes to this bug.