Bug 593062

Description of problem:

During the qmf scale testing - see 
https://bugzilla.redhat.com/show_bug.cgi?id=587410 - at approximately 2000 condor_configd consoles we start seeing the following failure:


 Traceback (most recent call last):
   File "/root/kgiusti/configuration-tools/condor_configd", line 370, in ?
     sys.exit(main())
   File "/root/kgiusti/configuration-tools/condor_configd", line 357, in main
     service.check_config_ver()
   File "/root/kgiusti/configuration-tools/condor_configd", line 241, in check_config_ver
     self.node_obj.checkin()
   File "/usr/lib/python2.4/site-packages/qmf/console.py", line 305, in <lambda>
     return lambda *args, **kwargs : self._invoke(name, args, kwargs)
   File "/usr/lib/python2.4/site-packages/qmf/console.py", line 428, in _invoke
     seq = self._sendMethodRequest(name, args, kwargs, sync, timeout)
   File "/usr/lib/python2.4/site-packages/qmf/console.py", line 409, in _sendMethodRequest
     self._broker._send(smsg, exchange)
   File "/usr/lib/python2.4/site-packages/qmf/console.py", line 2163, in _send
     self.amqpSession.message_transfer(destination=dest, message=msg)
   File "/usr/lib/python2.4/site-packages/qpid/generator.py", line 25, in <lambda>
     method = lambda self, *args, **kwargs: self.invoke(op, args, kwargs)
   File "/usr/lib/python2.4/site-packages/qpid/session.py", line 138, in invoke
     return self.do_invoke(op, args, kwargs)
   File "/usr/lib/python2.4/site-packages/qpid/session.py", line 183, in do_invoke
     self.sync(self.timeout)
   File "/usr/lib/python2.4/site-packages/qpid/session.py", line 101, in sync
     raise Timeout()
 qpid.exceptions.Timeout


This occurs during the simultaneous startup of many condor_configd processes, all communicating with one wallaby-agent via one qpidd broker.   This causes the broker to run at 100% cpu while the condor_configd processes initialize and download the configuration from the wallaby-agent.

The value for this timeout is 10 seconds.  This is the default assigned by the Session class __init__ method.

As a test, we bumped the default timeout from 10 to 120 seconds.  This eliminated the timeout failure when we run the scale tests.

We need to tune the default for this timeout such that the scale targets for BZ587410 can me met.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info: