Description of problem: During the qmf scale testing - see https://bugzilla.redhat.com/show_bug.cgi?id=587410 - at approximately 2000 condor_configd consoles we start seeing the following failure: Traceback (most recent call last): File "/root/kgiusti/configuration-tools/condor_configd", line 370, in ? sys.exit(main()) File "/root/kgiusti/configuration-tools/condor_configd", line 357, in main service.check_config_ver() File "/root/kgiusti/configuration-tools/condor_configd", line 241, in check_config_ver self.node_obj.checkin() File "/usr/lib/python2.4/site-packages/qmf/console.py", line 305, in <lambda> return lambda *args, **kwargs : self._invoke(name, args, kwargs) File "/usr/lib/python2.4/site-packages/qmf/console.py", line 428, in _invoke seq = self._sendMethodRequest(name, args, kwargs, sync, timeout) File "/usr/lib/python2.4/site-packages/qmf/console.py", line 409, in _sendMethodRequest self._broker._send(smsg, exchange) File "/usr/lib/python2.4/site-packages/qmf/console.py", line 2163, in _send self.amqpSession.message_transfer(destination=dest, message=msg) File "/usr/lib/python2.4/site-packages/qpid/generator.py", line 25, in <lambda> method = lambda self, *args, **kwargs: self.invoke(op, args, kwargs) File "/usr/lib/python2.4/site-packages/qpid/session.py", line 138, in invoke return self.do_invoke(op, args, kwargs) File "/usr/lib/python2.4/site-packages/qpid/session.py", line 183, in do_invoke self.sync(self.timeout) File "/usr/lib/python2.4/site-packages/qpid/session.py", line 101, in sync raise Timeout() qpid.exceptions.Timeout This occurs during the simultaneous startup of many condor_configd processes, all communicating with one wallaby-agent via one qpidd broker. This causes the broker to run at 100% cpu while the condor_configd processes initialize and download the configuration from the wallaby-agent. The value for this timeout is 10 seconds. This is the default assigned by the Session class __init__ method. As a test, we bumped the default timeout from 10 to 120 seconds. This eliminated the timeout failure when we run the scale tests. We need to tune the default for this timeout such that the scale targets for BZ587410 can me met. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Have submitted what I believe will fix this issue upstream: http://svn.apache.org/viewvc?view=revision&revision=945716 leaving bug open until I can run the scale tests against this fix.
Ran the condor_configd scale tests (configuration-tools 2.7-0.4) against python-qmf 0.7.946106-3 - no timeout failures encountered. Fix verified.