| Summary: | gofer becomes unresponsive w/ Enqueue capacity threshold exceeded error | ||
|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | dgao |
| Component: | user-experience | Assignee: | Jeff Ortel <jortel> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | unspecified | CC: | skarmark, tsanders |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | Sprint 26 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-02-24 20:14:26 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Fixed in gofer 0.43.
For topic subscription, gofer was not explicitly specifying that the queue associated with the 'heartbeat' AMQP exchange was durable:False and link:{x-declare:{auto-delete:True}. As a result, each time httpd restarted, the queue remained. Messages published on topic 'heartbeat' would continue to be routed to the old queue(s) for which there are (and never will be) a consumer. These queues associated w/ dead http processes eventually exceed the queue's capacity. An exception is only raised when the qpid-cpp-server-store is installed because it is a journalling error.
This is corrected in gofer 0.43. However, when this condition is raised, we also saw the python-qpid driver go into a tight loop and eat 100% of the cpu. This problem still exists in the driver. We'll need to recreate and file JIRA on this separately.
moving to verified as I haven't seen this for a while now [root@preethi ~]# rpm -q gofer gofer-0.47-1.fc15.noarch Pulp v1.0 is released Closed Current Release. Pulp v1.0 is released. |
caused by a CLI pkg install cmd: agent.log: 2011-04-01 09:37:37,748 [ERROR][Actions] __call__() @ action.py:117 - Enqueue ca pacity threshold exceeded on queue "717c5be2-58f8-4afd-9d91-410c61e78445:0.0". ( JournalImpl.cpp:621)(501) Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/gofer/agent/action.py", line 115, in __ call__ self.target() File "/usr/lib/gofer/plugins/pulp.py", line 76, in heartbeat p.send(topic, ttl=delay, heartbeat=body) File "/usr/lib/python2.6/site-packages/gofer/messaging/producer.py", line 56, in send sender = self.session().sender(address) File "<string>", line 6, in sender File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 576, in sender sender._ewait(lambda: sender.linked) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 783, in _ewait result = self.session._ewait(lambda: self.error or predicate(), timeout) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 550, in _ewait result = self.connection._ewait(lambda: self.error or predicate(), timeout) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 194, in _ewait self.check_error() File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 187, in check_error raise self.error ConnectionError: Enqueue capacity threshold exceeded on queue "717c5be2-58f8-4af d-9d91-410c61e78445:0.0". (JournalImpl.cpp:621)(501)