Bug 692891 - gofer becomes unresponsive w/ Enqueue capacity threshold exceeded error
Summary: gofer becomes unresponsive w/ Enqueue capacity threshold exceeded error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: user-experience
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: Sprint 26
Assignee: Jeff Ortel
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-04-01 14:19 UTC by dgao
Modified: 2013-09-09 16:29 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-02-24 20:14:26 UTC


Attachments (Terms of Use)

Description dgao 2011-04-01 14:19:58 UTC
caused by a CLI pkg install cmd:

agent.log: 
2011-04-01 09:37:37,748 [ERROR][Actions] __call__() @ action.py:117 - Enqueue ca
pacity threshold exceeded on queue "717c5be2-58f8-4afd-9d91-410c61e78445:0.0". (
JournalImpl.cpp:621)(501)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/gofer/agent/action.py", line 115, in __
call__
    self.target()
  File "/usr/lib/gofer/plugins/pulp.py", line 76, in heartbeat
    p.send(topic, ttl=delay, heartbeat=body)
  File "/usr/lib/python2.6/site-packages/gofer/messaging/producer.py", line 56, 
in send
    sender = self.session().sender(address)
  File "<string>", line 6, in sender
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 576,
 in sender
    sender._ewait(lambda: sender.linked)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 783,
 in _ewait
    result = self.session._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 550,
 in _ewait
    result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 194,
 in _ewait
    self.check_error()
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 187,
 in check_error
    raise self.error
ConnectionError: Enqueue capacity threshold exceeded on queue "717c5be2-58f8-4af
d-9d91-410c61e78445:0.0". (JournalImpl.cpp:621)(501)

Comment 1 Jeff Ortel 2011-07-22 21:28:20 UTC
Fixed in gofer 0.43.

For topic subscription, gofer was not explicitly specifying that the queue associated with the 'heartbeat' AMQP exchange was durable:False and link:{x-declare:{auto-delete:True}.  As a result, each time httpd restarted, the queue remained.  Messages published on topic 'heartbeat' would continue to be routed to the old queue(s) for which there are (and never will be) a consumer.  These queues associated w/ dead http processes eventually exceed the queue's capacity.  An exception is only raised when the qpid-cpp-server-store is installed because it is a journalling error.

This is corrected in gofer 0.43.  However, when this condition is raised, we also saw the python-qpid driver go into a tight loop and eat 100% of the cpu.  This problem still exists in the driver.  We'll need to recreate and file JIRA on this separately.

Comment 2 Preethi Thomas 2011-09-01 14:55:18 UTC
moving to verified as I haven't seen this for a while now

[root@preethi ~]# rpm -q gofer
gofer-0.47-1.fc15.noarch

Comment 3 Preethi Thomas 2012-02-24 20:14:26 UTC
Pulp v1.0 is released
Closed Current Release.

Comment 4 Preethi Thomas 2012-02-24 20:19:20 UTC
Pulp v1.0 is released.


Note You need to log in before you can comment on or make changes to this bug.