Bug 615883 - management clients (qpid-stat, qpid-cluster, qc_list_objests) frequently crash with 'Unknown connection exception'
management clients (qpid-stat, qpid-cluster, qc_list_objests) frequently cras...
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-qmf (Show other bugs)
Development
All Linux
high Severity high
: 1.3
: ---
Assigned To: Ken Giusti
MRG Quality Engineering
:
Depends On: 617357
Blocks: 547295
  Show dependency treegraph
 
Reported: 2010-07-19 04:25 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:12 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-10-20 07:28:44 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
strace'd failing 'qpid-cluster' (246.17 KB, text/x-log)
2010-07-19 07:56 EDT, Frantisek Reznicek
no flags Details

  None (edit)
Description Frantisek Reznicek 2010-07-19 04:25:25 EDT
Description of problem:

There are observed multiple occurrences of qmf client crashes with Unknown connection exception, for instance like this:
[root@mrg-qe-12 fcluster]# qpid-stat -b
[root@mrg-qe-12 fcluster]# qpid-stat -b
Failed: Exception - Unknown connection exception
Traceback (most recent call last):
  File "/usr/bin/qpid-stat", line 504, in ?
    bm.SetBroker(_host)
  File "/usr/bin/qpid-stat", line 152, in SetBroker
    self.broker = self.qmf.addBroker(brokerUrl, _connTimeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 622, in addBroker
    ssl = url.scheme == URL.AMQPS, connTimeout=timeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 1950, in __init__
    raise self.conn_exc
Exception: Unknown connection exception

Another case was triggered by qc_list_objects.py when trying to list all queues and exchanges. Client stored here:
http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/python/

I was able to see the above listed exception more rapidly in clustered qpidd exvironment.


Version-Release number of selected component (if applicable):
python-qpid-0.7.946106-9.el5
qpid-cpp-client*-0.7.946106-7.el5
qpid-cpp-server*-0.7.946106-7.el5
qpid-java-client-0.7.946106-6.el5
qpid-java-common-0.7.946106-6.el5
qpid-tools-0.7.946106-7.el5

but seen also on :
python-qpid-0.7.946106-4.el5
qpid-cpp-client*-0.7.946106-6.el5
qpid-cpp-server*-0.7.946106-6.el5
qpid-dotnet-0.4.738274-2.el5
qpid-java-client-0.7.946106-5.el5
qpid-java-common-0.7.946106-5.el5
qpid-java-example-0.7.946106-5.el5
qpid-tests-0.7.946106-1.el5
qpid-tools-0.7.946106-6.el5
ruby-qpid-0.7.946106-2.el5



How reproducible:
medium

Steps to Reproduce:
1. start standalone / clustered broker (with persistance)
2. run qpid-stat -b / qpid-config / qc_list_objects.py -av in the loop
3. wait for above exception
  
Actual results:
QMF clients crash occasionly with exception.

Expected results:
QMF clients should not crash with exception.

Additional info:

[root@mrg-qe-12 fcluster]# qpid-stat -b ; echo $?
Failed: Exception - Unknown connection exception
Traceback (most recent call last):
  File "/usr/bin/qpid-stat", line 504, in ?
    bm.SetBroker(_host)
  File "/usr/bin/qpid-stat", line 152, in SetBroker
    self.broker = self.qmf.addBroker(brokerUrl, _connTimeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 622, in addBroker
    ssl = url.scheme == URL.AMQPS, connTimeout=timeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 1950, in __init__
    raise self.conn_exc
Exception: Unknown connection exception
1
[root@mrg-qe-12 fcluster]# qpid-cluster ; echo $?
Failed: Exception - Unknown connection exception
1
Comment 1 Frantisek Reznicek 2010-07-19 07:56:04 EDT
Created attachment 432838 [details]
strace'd failing 'qpid-cluster'
Comment 2 Frantisek Reznicek 2010-07-20 03:07:46 EDT
Just another example of the issue:

[root@mrg-qe-10 fcluster]# python ./qc_list.objects.py -av >./qc_list.objects.log
Cannot connect to the broker <default>:<default>
Traceback (most recent call last):
  File "./qc_list.objects.py", line 160, in main
    broker = qmf_session.addBroker(timeout=SESSION_TIMEOUT);
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 622, in addBroker
    ssl = url.scheme == URL.AMQPS, connTimeout=timeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 1950, in __init__
    raise self.conn_exc
Exception: Unknown connection exception


In this case I'm sure the connection to qmf was done via following commands:
qmf_session = Session();
broker = None;
broker = qmf_session.addBroker(timeout=SESSION_TIMEOUT);

i.e. There was not given the target url (of addBroker() call)
Comment 3 Ken Giusti 2010-07-20 13:47:45 EDT
Added a fix to the exception handler so it will correctly identify the type of exception rather than reporting "Unknown error".

Committed revision 965927.

Would like to repo the problem with the above patch - is there a system I can test against (I've been unable to repo on my laptop or my VMs)?
Comment 4 Ken Giusti 2010-07-22 16:16:38 EDT
Moved the bug that caused the reason for the error to be discarded to https://bugzilla.redhat.com/show_bug.cgi?id=617357, and will push the fix for this into mrg_1.3.x.
Comment 5 Ken Giusti 2010-07-23 16:18:24 EDT
There's a new python-qmf package that will correctly display the reason for the connection failure:  

Package: python-qmf-0.7.946106-7.el5
Tag: dist-5E-mrg-devel

Please re-test with the above package installed, and update this BZ with the exception messages that are displayed.

Thanks,
Comment 6 Frantisek Reznicek 2010-07-27 06:11:41 EDT
The issue is resolved, python-qmf now reports reason insted of 'Unknown connection exception' verified on RHEL 4.8 / 5.5 i386 / x86_64 on packages:
python-qmf-0.7.946106-7.el5
python-qpid-0.7.946106-9.el5
qmf-0.7.946106-10.el5
qmf-devel-0.7.946106-10.el5
qpid-cpp-client-*0.7.946106-10.el5
qpid-cpp-mrg-debuginfo-0.7.946106-10.el5
qpid-cpp-server-*0.7.946106-10.el5
qpid-java-client-0.7.946106-6.el5
qpid-java-common-0.7.946106-6.el5
qpid-tools-0.7.946106-7.el5
rh-qpid-cpp-tests-0.7.946106-10.el5
ruby-qmf-0.7.946106-10.el5

-> VERIFIED

Note You need to log in before you can comment on or make changes to this bug.