Bug 1178829 - Qpid python client hangs when message with routing key longer than 255 is sent (mutual recursion)
Summary: Qpid python client hangs when message with routing key longer than 255 is sen...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: python-qpid
Version: Development
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: 3.1
: ---
Assignee: Ernie
QA Contact: Petr Matousek
URL:
Whiteboard:
Depends On:
Blocks: 1200971 1201334
TreeView+ depends on / blocked
 
Reported: 2015-01-05 14:21 UTC by Petr Matousek
Modified: 2015-04-14 13:48 UTC (History)
8 users (show)

Fixed In Version: python-qpid-0.30-6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1201334 (view as bug list)
Environment:
Last Closed: 2015-04-14 13:48:44 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Apache JIRA QPID-6445 None None None Never
Red Hat Bugzilla 812376 None None None Never
Red Hat Bugzilla 840479 None None None Never
Red Hat Product Errata RHEA-2015:0805 normal SHIPPED_LIVE Red Hat Enterprise MRG Messaging 3.1 Release 2015-04-14 17:45:54 UTC

Internal Links: 812376 840479

Description Petr Matousek 2015-01-05 14:21:34 UTC
Description of problem:

When producing a message with a routing key longer than 255 characters, the Qpid python client hangs / spins indefinitely. There is seen mutual recursion around check_error/sync/close in the endpoints.py that leads to memory allocation failure.

In the previous python-qpid versions following exception were reported: CodecException: Cannot encode 256 as uint8

Please see Additional information for the stack trace info.

Version-Release number of selected component (if applicable):
python-qpid-0.30-2

How reproducible:
100%

Steps to Reproduce:
1. ./spout "amq.fanout/$(printf 'X%.0s' {1..256})"
2. spout execution never ends

Actual results:
The sender execution never ends when subject longer that 255 characters is used.

Expected results:
Encoding exception reported.


Additional info:

(gdb) py-bt
#1 <built-in method poll of select.poll object at remote 0x7f1d68b52c60>
#3 file '/usr/lib/python2.6/site-packages/qpid/compat.py', in 'select'
#7 file '/usr/lib/python2.6/site-packages/qpid/compat.py', in 'wait'
#11 file '/usr/lib/python2.6/site-packages/qpid/concurrency.py', in 'wait'
#15 file '/usr/lib/python2.6/site-packages/qpid/concurrency.py', in 'wait'
#19 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_wait'
#23 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'detach'
#27 file '<string>', in 'detach'
#31 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'close'
#35 file '<string>', in 'close'
#39 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'check_error'
#42 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_ewait'
#46 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_ewait'
#50 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_ewait'
#54 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'sync'
#58 file '<string>', in 'sync'
#62 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'sync'
#66 file '<string>', in 'sync'
#70 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'close'
#74 file '<string>', in 'close'
#78 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'close'
#82 file '<string>', in 'close'
#86 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'check_error'
#89 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_ewait'
#93 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_ewait'
#97 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in '_ewait'
#101 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'sync'
#105 file '<string>', in 'sync'
#109 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'sync'
#113 file '<string>', in 'sync'
#117 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'close'
#121 file '<string>', in 'close'
#125 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'close'
#129 file '<string>', in 'close'
<snip>
...
</snip>
#2412 file '/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py', in 'sync'
Traceback (most recent call last):
  File "/usr/lib/debug/usr/lib64/libpython2.6.so.1.0.debug-gdb.py", line 1443, in invoke
    frame.print_summary()
  File "/usr/lib/debug/usr/lib64/libpython2.6.so.1.0.debug-gdb.py", line 1295, in print_summary
    py_co.field('co_name')))
RuntimeError: Cannot access memory at address 0x7fdae14e38f0
Error occurred in Python command: Cannot access memory at address 0x7fdae14e38f0

Comment 4 Ernie 2015-01-20 15:44:46 UTC
This appears to be an unintended side-effect of the fix for bug 574571 - 
 Python messaging client unusable after receiving a malformed message

As a result of that change, Encoding errors are now treated as recoverable and the program is allowed to continue to execute.

I recommend that the fix for bug 574571 be revisited. Not all Encoding exceptions should be allowed to continue, only the very specific exceptions address by the bug.

Comment 5 Justin Ross 2015-02-04 22:42:37 UTC
(In reply to Ernie from comment #4)
> This appears to be an unintended side-effect of the fix for bug 574571 - 
>  Python messaging client unusable after receiving a malformed message
> 
> As a result of that change, Encoding errors are now treated as recoverable
> and the program is allowed to continue to execute.
> 
> I recommend that the fix for bug 574571 be revisited. Not all Encoding
> exceptions should be allowed to continue, only the very specific exceptions
> address by the bug.

Ernie, please develop a new fix with more constrained exception handling.

Comment 6 Ernie 2015-02-11 18:47:58 UTC
After further review, this does not appear to have been caused by the fix to bug 574571. The exception in that case is a EncodeError. The exception in this case is a CodecException.

However it may be a side-effect of https://svn.apache.org/r1605855 for https://issues.apache.org/jira/browse/QPID-5852

When I comment out the self.close() in python/qpid/messaging/endpoints.py on line 226, the program ends with with correct exception. 

Pavel, what was the reason for the self.close() there? After running
./spout "amq.fanout/$(printf 'X%.0s' {1..256})"
At that point in endpoints.py, line 226, self._unlinked() returns an empty array.

Comment 7 Pavel Moravec 2015-02-20 09:02:07 UTC
(In reply to Ernie from comment #6)
> After further review, this does not appear to have been caused by the fix to
> bug 574571. The exception in that case is a EncodeError. The exception in
> this case is a CodecException.
> 
> However it may be a side-effect of https://svn.apache.org/r1605855 for
> https://issues.apache.org/jira/browse/QPID-5852
> 
> When I comment out the self.close() in python/qpid/messaging/endpoints.py on
> line 226, the program ends with with correct exception. 
> 
> Pavel, what was the reason for the self.close() there? After running
> ./spout "amq.fanout/$(printf 'X%.0s' {1..256})"
> At that point in endpoints.py, line 226, self._unlinked() returns an empty
> array.

r1605855 was incomplete patch of the JIRA - see https://issues.apache.org/jira/browse/QPID-5852?focusedCommentId=14073594&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14073594.


The self.close() is necessary for setting self._open=False (i.e. to fix the JIRA). Anyway that breaks other stuff..

Thinking more about the JIRA / bz840479, my feeling is the Python client is designed such that after raising ConnectionError, the connection object has deliberately connection.opened()=true and a programmer must close the connection and re-open before reusing it (I would rather see the connection object being closed by the library, but that would require nontrivial change and would be backward incompatible change).

Therefore the bz840479 should be closed as NOTABUG (well, after reverting the patch).

Ernie, does it sound reasonable? Or is bz840479 valid?

Comment 19 Petr Matousek 2015-03-19 12:08:04 UTC
This issue has been fixed. Verified on rhel6.6 (x86_64, i686) and rhel7.1 (x86_64).

Following Encoding exception is reported as expected:
CodecException: Cannot encode 256 as uint8

Packages:
python-qpid-0.30-6

NOTE: due to the fact that fix for bug 840479 was reverted to fix this issue, moving bug 840479 again to ASSIGNED.

-> VERIFIED

Comment 21 errata-xmlrpc 2015-04-14 13:48:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-0805.html


Note You need to log in before you can comment on or make changes to this bug.