Bug 615883

Summary: management clients (qpid-stat, qpid-cluster, qc_list_objests) frequently crash with 'Unknown connection exception'
Product: Red Hat Enterprise MRG Reporter: Frantisek Reznicek <freznice>
Component: qpid-qmfAssignee: Ken Giusti <kgiusti>
Status: CLOSED ERRATA QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: DevelopmentCC: esammons, kgiusti
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-20 11:28:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 617357    
Bug Blocks: 547295    
Attachments:
Description Flags
strace'd failing 'qpid-cluster' none

Description Frantisek Reznicek 2010-07-19 08:25:25 UTC
Description of problem:

There are observed multiple occurrences of qmf client crashes with Unknown connection exception, for instance like this:
[root@mrg-qe-12 fcluster]# qpid-stat -b
[root@mrg-qe-12 fcluster]# qpid-stat -b
Failed: Exception - Unknown connection exception
Traceback (most recent call last):
  File "/usr/bin/qpid-stat", line 504, in ?
    bm.SetBroker(_host)
  File "/usr/bin/qpid-stat", line 152, in SetBroker
    self.broker = self.qmf.addBroker(brokerUrl, _connTimeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 622, in addBroker
    ssl = url.scheme == URL.AMQPS, connTimeout=timeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 1950, in __init__
    raise self.conn_exc
Exception: Unknown connection exception

Another case was triggered by qc_list_objects.py when trying to list all queues and exchanges. Client stored here:
http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/python/

I was able to see the above listed exception more rapidly in clustered qpidd exvironment.


Version-Release number of selected component (if applicable):
python-qpid-0.7.946106-9.el5
qpid-cpp-client*-0.7.946106-7.el5
qpid-cpp-server*-0.7.946106-7.el5
qpid-java-client-0.7.946106-6.el5
qpid-java-common-0.7.946106-6.el5
qpid-tools-0.7.946106-7.el5

but seen also on :
python-qpid-0.7.946106-4.el5
qpid-cpp-client*-0.7.946106-6.el5
qpid-cpp-server*-0.7.946106-6.el5
qpid-dotnet-0.4.738274-2.el5
qpid-java-client-0.7.946106-5.el5
qpid-java-common-0.7.946106-5.el5
qpid-java-example-0.7.946106-5.el5
qpid-tests-0.7.946106-1.el5
qpid-tools-0.7.946106-6.el5
ruby-qpid-0.7.946106-2.el5



How reproducible:
medium

Steps to Reproduce:
1. start standalone / clustered broker (with persistance)
2. run qpid-stat -b / qpid-config / qc_list_objects.py -av in the loop
3. wait for above exception
  
Actual results:
QMF clients crash occasionly with exception.

Expected results:
QMF clients should not crash with exception.

Additional info:

[root@mrg-qe-12 fcluster]# qpid-stat -b ; echo $?
Failed: Exception - Unknown connection exception
Traceback (most recent call last):
  File "/usr/bin/qpid-stat", line 504, in ?
    bm.SetBroker(_host)
  File "/usr/bin/qpid-stat", line 152, in SetBroker
    self.broker = self.qmf.addBroker(brokerUrl, _connTimeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 622, in addBroker
    ssl = url.scheme == URL.AMQPS, connTimeout=timeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 1950, in __init__
    raise self.conn_exc
Exception: Unknown connection exception
1
[root@mrg-qe-12 fcluster]# qpid-cluster ; echo $?
Failed: Exception - Unknown connection exception
1

Comment 1 Frantisek Reznicek 2010-07-19 11:56:04 UTC
Created attachment 432838 [details]
strace'd failing 'qpid-cluster'

Comment 2 Frantisek Reznicek 2010-07-20 07:07:46 UTC
Just another example of the issue:

[root@mrg-qe-10 fcluster]# python ./qc_list.objects.py -av >./qc_list.objects.log
Cannot connect to the broker <default>:<default>
Traceback (most recent call last):
  File "./qc_list.objects.py", line 160, in main
    broker = qmf_session.addBroker(timeout=SESSION_TIMEOUT);
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 622, in addBroker
    ssl = url.scheme == URL.AMQPS, connTimeout=timeout)
  File "/usr/lib/python2.4/site-packages/qmf/console.py", line 1950, in __init__
    raise self.conn_exc
Exception: Unknown connection exception


In this case I'm sure the connection to qmf was done via following commands:
qmf_session = Session();
broker = None;
broker = qmf_session.addBroker(timeout=SESSION_TIMEOUT);

i.e. There was not given the target url (of addBroker() call)

Comment 3 Ken Giusti 2010-07-20 17:47:45 UTC
Added a fix to the exception handler so it will correctly identify the type of exception rather than reporting "Unknown error".

Committed revision 965927.

Would like to repo the problem with the above patch - is there a system I can test against (I've been unable to repo on my laptop or my VMs)?

Comment 4 Ken Giusti 2010-07-22 20:16:38 UTC
Moved the bug that caused the reason for the error to be discarded to https://bugzilla.redhat.com/show_bug.cgi?id=617357, and will push the fix for this into mrg_1.3.x.

Comment 5 Ken Giusti 2010-07-23 20:18:24 UTC
There's a new python-qmf package that will correctly display the reason for the connection failure:  

Package: python-qmf-0.7.946106-7.el5
Tag: dist-5E-mrg-devel

Please re-test with the above package installed, and update this BZ with the exception messages that are displayed.

Thanks,

Comment 6 Frantisek Reznicek 2010-07-27 10:11:41 UTC
The issue is resolved, python-qmf now reports reason insted of 'Unknown connection exception' verified on RHEL 4.8 / 5.5 i386 / x86_64 on packages:
python-qmf-0.7.946106-7.el5
python-qpid-0.7.946106-9.el5
qmf-0.7.946106-10.el5
qmf-devel-0.7.946106-10.el5
qpid-cpp-client-*0.7.946106-10.el5
qpid-cpp-mrg-debuginfo-0.7.946106-10.el5
qpid-cpp-server-*0.7.946106-10.el5
qpid-java-client-0.7.946106-6.el5
qpid-java-common-0.7.946106-6.el5
qpid-tools-0.7.946106-7.el5
rh-qpid-cpp-tests-0.7.946106-10.el5
ruby-qmf-0.7.946106-10.el5

-> VERIFIED