Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1643650 - pulp tasking system in Sat6.4 gets stuck, CV promotion waiting on start the pulp task
Summary: pulp tasking system in Sat6.4 gets stuck, CV promotion waiting on start the p...
Keywords:
Status: CLOSED DUPLICATE of bug 1649938
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installation
Version: 6.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Perry Gagne
URL:
Whiteboard:
Depends On:
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2018-10-26 21:23 UTC by Pavel Moravec
Modified: 2021-08-30 13:01 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-11 16:12:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3687321 0 None None None 2018-11-13 09:12:19 UTC

Description Pavel Moravec 2018-10-26 21:23:13 UTC 
Description of problem:
Under unknown reproducer, pulp tasking system gets stuck in Sat6.4. Symptoms are similar like in https://bugzilla.redhat.com/show_bug.cgi?id=1643382, but:

- Sat 6.4.0 affected
- CV publish&promotion on Satellite gets stuck (that does not differ from pulp tasking system point of view, I think)
- backtraces are quite different


Version-Release number of selected component (if applicable):
Sat 6.4.0


How reproducible:
???

Steps to Reproduce:
??? unknown


Actual results:
- no task is processed
- qpid-stat shows many tasks/messages in resource_manager queue and one task/message in each worker's .dq queue
- celery shows suspiciously small info for many workers:

# celery -A pulp.server.async.app inspect active
-> reserved_resource_worker-2.com: OK
    - empty -
-> reserved_resource_worker-0.com: OK
    - empty -
#

(other workers missing, those two "idle" despite they have message in their queue etc.)


Expected results:
- tasks processed


Additional info:
coredumps (available in case 02239895) show:
- all child worker processes idle, having just one thread waiting on qpidd
- all parent worker processes have threads in very similar state that smells to me like a deadlock/waiting for each other:

Thread 9 (Thread 0x7f3e61f1f740 
#7  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e280127e0, for file /usr/lib64/python2.7/threading.py, line 339, in wait 
#11 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e28003640, for file /usr/lib64/python2.7/threading.py, line 471, in acquire 
#15 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e28004eb0, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 1469, in apply_async 
#20 PyEval_EvalFrameEx (f=f@entry=Frame 0x2633f30, for file /usr/lib/python2.7/site-packages/celery/concurrency/base.py, line 155, in apply_async 
#24 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e1c026990, for file /usr/lib/python2.7/site-packages/celery/worker/request.py, line 531, in execute_using_pool 
#28 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e280082a0, for file /usr/lib/python2.7/site-packages/celery/worker/worker.py, line 226, in _process_task 
#32 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e28001ba0, for file /usr/lib/python2.7/site-packages/celery/worker/strategy.py, line 145, in task_message_handler 
#36 PyEval_EvalFrameEx (f=f@entry=Frame 0x267b5b0, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 568, in on_task_received 
#40 PyEval_EvalFrameEx (f=f@entry=Frame 0x26b4290, for file /usr/lib/python2.7/site-packages/kombu/messaging.py, line 623, in _receive_callback 
#44 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e1c025ba0, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 976, in _callback 
#48 PyEval_EvalFrameEx (f=f@entry=Frame 0x26acfb0, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 1701, in drain_events 
#53 PyEval_EvalFrameEx (f=f@entry=Frame 0x26b6ff0, for file /usr/lib/python2.7/site-packages/kombu/connection.py, line 301, in drain_events 
#57 PyEval_EvalFrameEx (f=f@entry=Frame 0x26b1230, for file /usr/lib/python2.7/site-packages/celery/worker/loops.py, line 118, in synloop 
#62 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e357c7c90, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 594, in start 
#66 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d719050, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 119, in start 
#70 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d714d70, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 318, in start 
#74 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d843208, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 370, in start 
#78 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e58b050, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 119, in start 
#82 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e5a0050, for file /usr/lib/python2.7/site-packages/celery/worker/worker.py, line 203, in start 
#86 PyEval_EvalFrameEx (f=f@entry=Frame 0x1f345f0, for file /usr/lib/python2.7/site-packages/celery/bin/worker.py, line 256, in run 
#91 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e501fca00, for file /usr/lib/python2.7/site-packages/celery/bin/base.py, line 244, in __call__ 
#100 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e50156450, for file /usr/lib/python2.7/site-packages/celery/bin/worker.py, line 221, in run_from_argv 
#104 PyEval_EvalFrameEx (f=f@entry=Frame 0x1f145a0, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 412, in execute 
#108 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e50525a28, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 480, in handle_argv 
#112 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e53c133f0, for file /usr/lib/python2.7/site-packages/celery/bin/base.py, line 281, in execute_from_commandline 
#116 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e5355b988, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 488, in execute_from_commandline 
#120 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e553c4410, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 326, in main 
#124 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e61e25aa0, for file /usr/lib/python2.7/site-packages/celery/__main__.py, line 14, in main 
Thread 8 (Thread 0x7f3e367fc700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e28008ad0, for file /usr/lib/python2.7/site-packages/qpid/compat.py, line 88, in select 
#8  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e1c000b50, for file /usr/lib/python2.7/site-packages/qpid/selector.py, line 152, in run 
#13 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3c6cb938, for file /usr/lib64/python2.7/threading.py, line 765, in run 
#16 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e1c000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#19 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3c6e1750, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 7 (Thread 0x7f3e36ffd700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d719410, for file /usr/lib/python2.7/site-packages/celery/utils/timer2.py, line 84, in run 
#8  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e28000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#11 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d71b210, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 6 (Thread 0x7f3e377fe700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d7a8a28, for file /usr/lib64/python2.7/site-packages/billiard/connection.py, line 976, in _poll 
#8  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d7a8830, for file /usr/lib64/python2.7/site-packages/billiard/connection.py, line 998, in wait 
#12 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d70b960, for file /usr/lib64/python2.7/site-packages/billiard/connection.py, line 465, in _poll 
#16 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d715ad0, for file /usr/lib64/python2.7/site-packages/billiard/connection.py, line 287, in poll 
#20 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d715910, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 1326, in _poll_result 
#24 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e24000b50, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 826, in _process_result 
#26 0x00007f3e61a25c41 in PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d780d38, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 861, in body 
#30 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d714b90, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 478, in run 
#34 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e24000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#37 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d715750, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 5 (Thread 0x7f3e37fff700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d780b60, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 735, in body 
#8  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d7149b0, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 478, in run 
#12 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e2c000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#15 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d715590, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 4 (Thread 0x7f3e3cf06700 
#7  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e2800fd50, for file /usr/lib64/python2.7/threading.py, line 339, in wait 
#11 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d764430, for file /usr/lib64/python2.7/Queue.py, line 168, in get 
#18 0x00007f3e61a25c41 in PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e30000b50, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 561, in body 
#22 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d714410, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 478, in run 
#26 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e30000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#29 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d715210, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 3 (Thread 0x7f3e3d707700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d7859d8, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 538, in body 
#8  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d714230, for file /usr/lib64/python2.7/site-packages/billiard/pool.py, line 478, in run 
#12 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e38000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#15 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e3d715050, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 2 (Thread 0x7f3e4dd0f700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e51f770, for file /usr/lib64/python2.7/site-packages/pymongo/periodic_executor.py, line 103, in _run 
#9  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e51f5a8, for file /usr/lib64/python2.7/threading.py, line 765, in run 
#12 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e40000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#15 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e5203d0, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 
Thread 1 (Thread 0x7f3e4e510700 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e51f3e0, for file /usr/lib64/python2.7/site-packages/pymongo/periodic_executor.py, line 103, in _run 
#9  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e51f218, for file /usr/lib64/python2.7/threading.py, line 765, in run 
#12 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e48000910, for file /usr/lib64/python2.7/threading.py, line 812, in __bootstrap_inner 
#15 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3e4e519e50, for file /usr/lib64/python2.7/threading.py, line 785, in __bootstrap 


or (just Thread9, others are the same):
Thread 9 (Thread 0x7f324884f740 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3222020590, for file /usr/lib/python2.7/site-packages/qpid/compat.py, line 203, in _do_read 
#7  PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400c720, for file /usr/lib/python2.7/site-packages/qpid/compat.py, line 139, in wait 
#11 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400b320, for file /usr/lib/python2.7/site-packages/qpid/concurrency.py, line 96, in wait 
#15 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400fc40, for file /usr/lib/python2.7/site-packages/qpid/concurrency.py, line 57, in wait 
#19 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400c490, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 252, in _wait 
#23 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400a230, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 273, in _ewait 
#27 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400c200, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 637, in _ewait 
#31 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3204010c50, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 928, in _ewait 
#35 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f32209a57f0, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 1015, in sync 
#43 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cc2120, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 1003, in send 
#51 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cc1e10, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 482, in _put 
#56 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f32209a9450, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 1138, in basic_publish 
#60 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cbd850, for file /usr/lib/python2.7/site-packages/kombu/messaging.py, line 203, in _publish 
#64 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cd0a00, for file /usr/lib/python2.7/site-packages/kombu/messaging.py, line 181, in publish 
#69 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cff540, for file /usr/lib/python2.7/site-packages/kombu/pidbox.py, line 265, in _publish_reply 
#73 PyEval_EvalFrameEx (f=f@entry=Frame 0x1583ee0, for file /usr/lib/python2.7/site-packages/kombu/pidbox.py, line 135, in reply 
#77 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3214006be0, for file /usr/lib/python2.7/site-packages/kombu/pidbox.py, line 112, in dispatch 
#82 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3210025c50, for file /usr/lib/python2.7/site-packages/kombu/pidbox.py, line 129, in handle_message 
#86 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3204007930, for file /usr/lib/python2.7/site-packages/celery/worker/pidbox.py, line 42, in on_message 
#90 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f320400d420, for file /usr/lib/python2.7/site-packages/kombu/messaging.py, line 589, in receive 
#94 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f32040061f0, for file /usr/lib/python2.7/site-packages/kombu/messaging.py, line 623, in _receive_callback 
#98 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3204001790, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 976, in _callback 
#102 PyEval_EvalFrameEx (f=f@entry=Frame 0x1d007d0, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 1701, in drain_events 
#107 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f32200d65a8, for file /usr/lib/python2.7/site-packages/kombu/connection.py, line 301, in drain_events 
#111 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cfafd0, for file /usr/lib/python2.7/site-packages/celery/worker/loops.py, line 118, in synloop 
#116 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3222041ad0, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 594, in start 
#120 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3224056230, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 119, in start 
#124 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3224056050, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 318, in start 
#128 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f32240be050, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 370, in start 
#132 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3234ebc230, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 119, in start 
#136 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3234ed0050, for file /usr/lib/python2.7/site-packages/celery/worker/worker.py, line 203, in start 
#140 PyEval_EvalFrameEx (f=f@entry=Frame 0x157dc40, for file /usr/lib/python2.7/site-packages/celery/bin/worker.py, line 256, in run 
#145 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3236b2ca00, for file /usr/lib/python2.7/site-packages/celery/bin/base.py, line 244, in __call__ 
#154 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3236a86450, for file /usr/lib/python2.7/site-packages/celery/bin/worker.py, line 221, in run_from_argv 
#158 PyEval_EvalFrameEx (f=f@entry=Frame 0x14c9770, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 412, in execute 
#162 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3236e55a28, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 480, in handle_argv 
#166 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f323a5433f0, for file /usr/lib/python2.7/site-packages/celery/bin/base.py, line 281, in execute_from_commandline 
#170 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3239e8b988, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 488, in execute_from_commandline 
#174 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f323bcf4410, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 326, in main 
#178 PyEval_EvalFrameEx (f=f@entry=Frame 0x7f3248755aa0, for file /usr/lib/python2.7/site-packages/celery/__main__.py, line 14, in main 

or:
Thread 9 (Thread 0x7ffa43de1740 
#4  PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa000066e0, for file /usr/lib/python2.7/site-packages/qpid/compat.py, line 88, in select 
#8  PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa0000a080, for file /usr/lib/python2.7/site-packages/qpid/compat.py, line 127, in wait 
#12 PyEval_EvalFrameEx (f=f@entry=Frame 0x23e6c30, for file /usr/lib/python2.7/site-packages/qpid/concurrency.py, line 96, in wait 
#16 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa0000a310, for file /usr/lib/python2.7/site-packages/qpid/concurrency.py, line 59, in wait 
#20 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa0c029170, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 252, in _wait 
#24 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa000041f0, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 273, in _ewait 
#28 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa000075f0, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 637, in _ewait 
#32 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1000a510, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 57, in _ecwait 
#36 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1c094050, for file /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py, line 777, in next_receiver 
#44 PyEval_EvalFrameEx (f=f@entry=Frame 0x246af90, for file /usr/lib/python2.7/site-packages/kombu/transport/qpid.py, line 1695, in drain_events 
#49 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1c10d3e0, for file /usr/lib/python2.7/site-packages/kombu/connection.py, line 301, in drain_events 
#53 PyEval_EvalFrameEx (f=f@entry=Frame 0x246a7c0, for file /usr/lib/python2.7/site-packages/celery/worker/loops.py, line 118, in synloop 
#58 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1c104750, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 594, in start 
#62 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1f612230, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 119, in start 
#66 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1f612050, for file /usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py, line 318, in start 
#70 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa1f734e10, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 370, in start 
#74 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa30441230, for file /usr/lib/python2.7/site-packages/celery/bootsteps.py, line 119, in start 
#78 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa30464050, for file /usr/lib/python2.7/site-packages/celery/worker/worker.py, line 203, in start 
#82 PyEval_EvalFrameEx (f=f@entry=Frame 0x1cf6e30, for file /usr/lib/python2.7/site-packages/celery/bin/worker.py, line 256, in run 
#87 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa320bea00, for file /usr/lib/python2.7/site-packages/celery/bin/base.py, line 244, in __call__ 
#96 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa32018450, for file /usr/lib/python2.7/site-packages/celery/bin/worker.py, line 221, in run_from_argv 
#100 PyEval_EvalFrameEx (f=f@entry=Frame 0x15f11c0, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 412, in execute 
#104 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa323e7a28, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 480, in handle_argv 
#108 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa35ad53f0, for file /usr/lib/python2.7/site-packages/celery/bin/base.py, line 281, in execute_from_commandline 
#112 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa3541d988, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 488, in execute_from_commandline 
#116 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa37286410, for file /usr/lib/python2.7/site-packages/celery/bin/celery.py, line 326, in main 
#120 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffa43ce7aa0, for file /usr/lib/python2.7/site-packages/celery/__main__.py, line 14, in main
Comment 2 Pavel Moravec 2018-11-06 08:55:02 UTC 
We hit this once on our internal Sat6.4 (provisioning.usersys.redhat.com) - in a situation when the (powerfull but still) VM got impacted by host system running 100% CPU on all cores - maybe an idea for reproducer..?
Comment 3 Pavel Moravec 2018-11-06 13:09:01 UTC 
(In reply to Pavel Moravec from comment #2)
> We hit this once on our internal Sat6.4 (provisioning.usersys.redhat.com) -
> in a situation when the (powerfull but still) VM got impacted by host system
> running 100% CPU on all cores - maybe an idea for reproducer..?

Very reliable reproducer on my VM (under the same host, details will follow):

# qpid-stat --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 -q | grep resource
  reserved_resource_worker-0.lan.celery.pidbox       Y                 0     2      2       0   1.30k    1.30k        1     2
  reserved_resource_worker-0.lan.dq2            Y                      1    56     55    1.10k  70.9k    69.8k        1     2
  reserved_resource_worker-1.lan.celery.pidbox       Y                 0     2      2       0   1.30k    1.30k        1     2
  reserved_resource_worker-1.lan.dq2            Y                      4    50     46    5.06k  63.3k    58.2k        1     2
  reserved_resource_worker-2.lan.celery.pidbox       Y                 0     2      2       0   1.30k    1.30k        1     2
  reserved_resource_worker-2.lan.dq2            Y                      2    60     58    2.53k  75.9k    73.4k        1     2
  reserved_resource_worker-3.lan.celery.pidbox       Y                 0     2      2       0   1.30k    1.30k        1     2
  reserved_resource_worker-3.lan.dq2            Y                      7    50     43    8.70k  63.3k    54.6k        1     2
  resource_manager                                                     Y                     10   111    101    17.9k   198k     180k        1     2
  resource_manager.lan.dq2                 Y                      0     0      0       0      0        0         0     2
  resource_manager.lan.celery.pidbox                 Y                 0     2      2       0   1.30k    1.30k        1     2
  resource_manager.lan.dq2                      Y                      0     0      0       0      0        0         1     2
# qpid-stat --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 -u | grep resource
  1                    resource_manager                                                     qpid.[::1]:5671-[::1]:44372  celery     19825           Y            CREDIT      57         6
  0                    resource_manager.lan.celery.pidbox            qpid.[::1]:5671-[::1]:44372  celery     19825           Y            CREDIT      2          6
  2                    resource_manager.lan.dq2                      qpid.[::1]:5671-[::1]:44372  celery     19825           Y            CREDIT      0          6
  0                    reserved_resource_worker-3.lan.celery.pidbox  qpid.[::1]:5671-[::1]:44484  celery     20040           Y            CREDIT      2          8
  2                    reserved_resource_worker-3.lan.dq2            qpid.[::1]:5671-[::1]:44484  celery     20040           Y            CREDIT      29         8
  0                    reserved_resource_worker-0.lan.celery.pidbox  qpid.[::1]:5671-[::1]:44454  celery     20031           Y            CREDIT      2          2
  2                    reserved_resource_worker-0.lan.dq2            qpid.[::1]:5671-[::1]:44454  celery     20031           Y            CREDIT      34         2
  0                    reserved_resource_worker-2.lan.celery.pidbox  qpid.[::1]:5671-[::1]:44488  celery     20037           Y            CREDIT      2          2
  2                    reserved_resource_worker-2.lan.dq2            qpid.[::1]:5671-[::1]:44488  celery     20037           Y            CREDIT      29         2
  0                    reserved_resource_worker-1.lan.celery.pidbox  qpid.[::1]:5671-[::1]:44470  celery     20034           Y            CREDIT      2          4
  2                    reserved_resource_worker-1.lan.dq2            qpid.[::1]:5671-[::1]:44470  celery     20034           Y            CREDIT      23         4
# celery -A pulp.server.async.app inspect active --timeout=200
-> reserved_resource_worker-1.lan: OK
    - empty -
-> reserved_resource_worker-0.lan: OK
    - empty -
^C# pulp-admin -u admin -p $pulpAdminPassword tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  publish
Resources:   c65968ce-3b4f-42a5-a208-337a2e3e03ac (repository)
State:       Waiting
Start Time:  Unstarted
Finish Time: Incomplete
Task Id:     6e51fc42-2dff-4a5c-9bce-b86712bcd5e4
Worker Name: reserved_resource_worker-1.lan

Operations:  publish
Resources:   a05dbb22-8756-44f1-9988-de6eec16df30 (repository)
State:       Waiting
Start Time:  Unstarted
Finish Time: Incomplete
Task Id:     7ecb17c1-adc4-40b1-9e8d-d9efd68420dd
Worker Name: reserved_resource_worker-3.lan

..
#


and coredumps match above backtraces
Comment 5 Pavel Moravec 2018-11-06 13:13:35 UTC 
Forgot to mention the reproducer steps:

1) generate some CPU load (here by querying mongo for all collections):

cd /root/pulp_db
for i in $(cat _tables.txt ); do echo $i; mongo pulp_database --eval "load('.mongorc.js'); db.${i}.find().shellPrint()" > /dev/null; done

2) forcefully publish all (exceptsome biggest) repos:

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

for i in 1 2 3 4 5; do
for repo in $(ls /var/lib/pulp/published/yum/master/yum_distributor/ | grep -v -e c1db7f26-b874-4a9b-b0c9-233db901f114 -e d46be6ac-8f46-40da-890e-a983fb7ca6bb); do curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"$repo\",\"override_config\":{\"force_full\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$repo/actions/publish/ & done
sleep 2
done


3) time to time check celery status / qpid queues.
Comment 6 Pavel Moravec 2018-11-06 21:34:11 UTC 
The above reproducer _is_ realiable, reproduced on a dedicated HW:

- Having separate PC and running a VM with Satellite on it
- generating a CPU load on the PC (i.e. run another VM and compile some program in a loop)
- restart pulp services on the Satellite VM
- bulk create more (forcefully re)publish tasks
- check celery status
Comment 13 David Davis 2018-11-09 13:07:24 UTC 
Yes, I am actively working on this. It's my top priority at the moment. I'll try to provide more updates as I have them.
Comment 15 David Davis 2018-11-12 23:09:33 UTC 
An update: it seems that disabling PULP_MAX_TASKS_PER_CHILD fixes the problem (at least for the reproducer we're using to test this bug). It looks like process recycling is the issue. Going to see if it reconnecting to qpid helps at all.
Comment 16 Pavel Moravec 2018-11-13 08:40:03 UTC 
I can confirm that:

- crucial command to trigger the workers stuck is "celery -A pulp.server.async.app inspect active"

- both workarounds prevent the stuck:
  - disabling workers recycling via commenting out "PULP_MAX_TASKS_PER_CHILD=2" in /etc/default/pulp_workers
  - or running workers in "--pool solo" mode
Comment 17 David Davis 2018-11-13 16:35:12 UTC 
I think our only option is to disable PULP_MAX_TASKS_PER_CHILD or at least set it to a higher number. 2 is overly aggressive and prone to causing problems.

I was thinking that we could maybe attempt to reconnect to qpid after forking but it appears that we've been down that road before:

https://github.com/apache/qpid-python/commit/e859964c379a72a3d9fd6502829176ddb4f1b90b#diff-498240a0a2a6c24273a734324dfd386c
Comment 19 Pavel Moravec 2018-12-06 07:06:39 UTC 
Quite reliable reproducer:

have pulp celery worker recycling enabled, and run few times:

celery -A pulp.server.async.app inspect active

That is enough.. :-/
Comment 21 David Davis 2018-12-07 17:14:40 UTC 
Based on comment 17, the solution is to raise or disable PULP_MAX_TASKS_PER_CHILD. I believe this is set in the installer so I'm reassigning.
Comment 22 Pavel Moravec 2018-12-07 20:29:13 UTC 
Technically, the bug is within pulp (or in libraries it uses).

Increasing PULP_MAX_TASKS_PER_CHILD can just alleviate the problem, decrease probability of hitting hit, but it isnt a resolution.

Disabling pulp child recycling prevents this bug, but it is assumed memory accumulation of worker process will happen.

Just 2c from a support guy, I am ok with whatever resolution of the bugzilla, if the fix wont have regressions elsewhere.
Comment 23 David Davis 2018-12-07 20:34:51 UTC 
Pavel,

You're absolutely correct. There is an architectural problem in that celery uses a forking model and qpid doesn't support that. One solution might be to move to rabbitmq but I believe that isn't an option. The other solution would be to move off celery and onto something like rq (which Pulp 3 currently uses). However, I'd worry about introducing regressions. Thus, I think increasing or disabling PULP_MAX_TASKS_PER_CHILD is the safest bet for now. We can reevaluate though if we continue to see more celery problems.
Comment 25 Peter Vreman 2018-12-10 11:42:16 UTC 
Pavel, David,

I understand the issue and that it is hard (timeconsuming and risk) to replace child spawning in the core.
Better spend the time on the future to have pulp3 out quicker.

I will start with the pulp_max_tasks on 10000 and see the frequency. As long as it gets down to once per 60 days than it is acceptable for me.

Peter
Comment 27 Eric Helms 2018-12-11 16:12:21 UTC 

*** This bug has been marked as a duplicate of bug 1649938 ***
Note You need to log in before you can comment on or make changes to this bug.