Description of problem: During the qmf client testing against deployed MRG/M + MRG/G. Below pasted client (qmf_list_objects.py) shows that some runs end with unhandled exception: Exception in thread Thread-4 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap File "/usr/lib/python2.4/threading.py", line 422, in run File "/usr/lib/python2.4/site-packages/qpid/queue.py", line 84, in run File "/usr/lib/python2.4/site-packages/qmf/console.py", line 2253, in _v2Cb File "/usr/lib/python2.4/site-packages/qpid/generator.py", line 25, in <lambda> File "/usr/lib/python2.4/site-packages/qpid/connection.py", line 219, in invoke File "/usr/lib/python2.4/site-packages/qpid/connection.py", line 192, in write_op File "/usr/lib/python2.4/threading.py", line 112, in release exceptions.TypeError: 'NoneType' object is not callable Unhandled exception in thread started by Error in sys.excepthook: Original exception was: The test was run on RHEL 5.5 i386. Version-Release number of selected component (if applicable): condor-7.4.3-0.13.el5 condor-debuginfo-7.4.3-0.13.el5 condor-kbdd-7.4.3-0.13.el5 condor-qmf-7.4.3-0.13.el5 condor-test-7.4.3-0.13.el5 condor-vm-gahp-7.4.3-0.13.el5 python-qmf-0.7.934605-1.el5 python-qpid-0.7.938298-1.el5 qmf-0.7.939184-1.el5 qmf-devel-0.7.939184-1.el5 qpid-cpp-client-0.7.939184-1.el5 qpid-cpp-client-devel-0.7.939184-1.el5 qpid-cpp-server-0.7.939184-1.el5 qpid-cpp-server-store-0.7.939184-1.el5 qpid-tests-0.7.934605-1.el5 qpid-tools-0.7.934605-2.el5 ruby-qmf-0.7.939184-1.el5 How reproducible: ~ 10% (once per 10 runs) Steps to Reproduce: 1. install packages, start condor and qpidd services + configure the condor service to use QMF plugins 2. python ./qmf_list_objects.py in the loop 3. Actual results: QMF client shows unhandled threading exception. Expected results: QMF client should not show unhandled threading exception. Testing client qmf_list_objects.py: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/usr/bin/env python from qmf.console import Session from sys import exit gridobjs = [ 'master', 'grid', 'collector', 'negotiator', 'slot', 'scheduler', 'jobserver', 'submitter', 'submission' ] session = Session(); try: session.addBroker() except: print 'Unable to connect to broker' exit(1) for objs in gridobjs: objects = session.getObjects(_class=objs, _package='com.redhat.grid') for obj in objects: print '\n' print objs," = ",obj.Name print "*****Properties*****" for prop in obj.getProperties(): print " ",prop[0],"=",prop[1] print "*****Statistics*****" for stat in obj.getStatistics(): print " ",stat[0],"=",stat[1] print "*****Methods********" for meth in obj.getMethods(): print " ",meth Additional info: master = mrg-qe-05.lab.eng.brq.redhat.com *****Properties***** CondorPlatform = $CondorPlatform: I386-LINUX_RHEL5 $ CondorVersion = $CondorVersion: 7.4.3 May 5 2010 BuildID: RH-7.4.3-0.13.el5 PRE-RELEASE $ DaemonStartTime = 1273650856000000000 Pool = mrg-qe-05.lab.eng.brq.redhat.com System = mrg-qe-05.lab.eng.brq.redhat.com Name = mrg-qe-05.lab.eng.brq.redhat.com Machine = mrg-qe-05.lab.eng.brq.redhat.com MyAddress = <10.34.33.58:52470> RealUid = 0 *****Statistics***** MonitorSelfAge = 0 MonitorSelfCPUUsage = 0.0363834649324 MonitorSelfImageSize = 48692.0 MonitorSelfRegisteredSocketCount = 2 MonitorSelfResidentSetSize = 7088 MonitorSelfTime = 0 *****Methods******** Start(Subsystem) Stop(Subsystem) collector = Personal Condor *****Properties***** Pool = mrg-qe-05.lab.eng.brq.redhat.com System = mrg-qe-05.lab.eng.brq.redhat.com CondorPlatform = $CondorPlatform: I386-LINUX_RHEL5 $ CondorVersion = $CondorVersion: 7.4.3 May 5 2010 BuildID: RH-7.4.3-0.13.el5 PRE-RELEASE $ Name = Personal Condor MyAddress = <10.34.33.58:9618> *****Statistics***** RunningJobs = 0 IdleJobs = 0 HostsTotal = 8 HostsClaimed = 0 HostsUnclaimed = 8 HostsOwner = 0 *****Methods******** negotiator = mrg-qe-05.lab.eng.brq.redhat.com *****Properties***** Pool = mrg-qe-05.lab.eng.brq.redhat.com System = mrg-qe-05.lab.eng.brq.redhat.com Name = mrg-qe-05.lab.eng.brq.redhat.com Machine = mrg-qe-05.lab.eng.brq.redhat.com MyAddress = <10.34.33.58:57555> CondorPlatform = $CondorPlatform: I386-LINUX_RHEL5 $ CondorVersion = $CondorVersion: 7.4.3 May 5 2010 BuildID: RH-7.4.3-0.13.el5 PRE-RELEASE $ DaemonStartTime = 1273650859000000000 *****Statistics***** MonitorSelfAge = 0 MonitorSelfCPUUsage = 0.0 MonitorSelfImageSize = 0.0 MonitorSelfRegisteredSocketCount = 0 MonitorSelfResidentSetSize = 0 MonitorSelfTime = 0 *****Methods******** GetLimits() SetLimit(Name, Max) GetStats(Name) SetPriority(Name, Priority) SetPriorityFactor(Name, PriorityFactor) SetUsage(Name, Usage) GetRawConfig(Name) SetRawConfig(Name, Value) Reconfig() scheduler = mrg-qe-05.lab.eng.brq.redhat.com *****Properties***** CondorPlatform = $CondorPlatform: I386-LINUX_RHEL5 $ CondorVersion = $CondorVersion: 7.4.3 May 5 2010 BuildID: RH-7.4.3-0.13.el5 PRE-RELEASE $ DaemonStartTime = 1273650859000000000 Pool = mrg-qe-05.lab.eng.brq.redhat.com System = mrg-qe-05.lab.eng.brq.redhat.com JobQueueBirthdate = 1272620499000000000 MaxJobsRunning = 8311 Machine = mrg-qe-05.lab.eng.brq.redhat.com MyAddress = <10.34.33.58:42983> Name = mrg-qe-05.lab.eng.brq.redhat.com *****Statistics***** MonitorSelfAge = 0 MonitorSelfCPUUsage = 0.0 MonitorSelfImageSize = 50284.0 MonitorSelfRegisteredSocketCount = 4 MonitorSelfResidentSetSize = 8316 MonitorSelfTime = 1273653259000000000 NumUsers = 0 TotalHeldJobs = 0 TotalIdleJobs = 0 TotalJobAds = 0 TotalRemovedJobs = 0 TotalRunningJobs = 0 *****Methods******** Submit(Ad) GetAd(Id) SetAttribute(Id, Name, Value) Hold(Id, Reason) Release(Id, Reason) Remove(Id, Reason) Fetch(Id, File, Start, End) GetStates(Submission, State) GetJobs(Submission) echo(sequence, body) Exception in thread Thread-4 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap File "/usr/lib/python2.4/threading.py", line 422, in run File "/usr/lib/python2.4/site-packages/qpid/queue.py", line 84, in run File "/usr/lib/python2.4/site-packages/qmf/console.py", line 2253, in _v2Cb File "/usr/lib/python2.4/site-packages/qpid/generator.py", line 25, in <lambda> File "/usr/lib/python2.4/site-packages/qpid/connection.py", line 219, in invoke File "/usr/lib/python2.4/site-packages/qpid/connection.py", line 192, in write_op File "/usr/lib/python2.4/threading.py", line 112, in release exceptions.TypeError: 'NoneType' object is not callable Unhandled exception in thread started by Error in sys.excepthook: Original exception was: [root@mrg-qe-05 freznice]# cat /var/lib/condor/condor_config.local CONDOR_DEVELOPERS = NONE CONDOR_DEVELOPERS_COLLECTOR = NONE CONDOR_HOST = $(FULL_HOSTNAME) COLLECTOR_NAME = Personal Condor START = TRUE SUSPEND = FALSE PREEMPT = FALSE KILL = FALSE DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD, STARTD NEGOTIATOR_INTERVAL = 20 #SCHEDD_CLUSTER_INITIAL_VALUE = 1000000 SCHEDD_CLUSTER_MAXIMUM_VALUE = 10 #UPDATE_INTERVAL=5 SCHEDD_CLUSTER_MAXIMUM_VALUE = 2 SCHEDD_CLUSTER_MAXIMUM_VALUE = 10 SCHEDD.PLUGINS = $(LIB)/plugins/MgmtScheddPlugin-plugin.so COLLECTOR.PLUGINS = $(LIB)/plugins/MgmtCollectorPlugin-plugin.so NEGOTIATOR.PLUGINS = $(LIB)/plugins/MgmtNegotiatorPlugin-plugin.so MASTER.PLUGINS = $(LIB)/plugins/MgmtMasterPlugin-plugin.so QMF_BROKER_HOST = $(FULL_HOSTNAME) QMF_BROKER_PORT = 5672
The qmf_list_objects.py script needs to call "delBroker()" to delete the broker returned by "addBroker()" before exiting. Example: bkr = s.addBroker() ..... s.delBroker(bkr) There is a feature request to add a "close()" interface to the session that will clean up all brokers automatically, but until then you'll need to call "delBroker()" by hand. https://issues.apache.org/jira/browse/QPID-2730