Description of problem:
Calling receiver.fetch(timeout=10) in a loop, when network drops packages for a while, this causes uncaught exception KeyError.

High prio/sev due to https://bugzilla.redhat.com/show_bug.cgi?id=1169416#c5.

Upstream qpid fails with infinite loop (see QPID-6297 for details).


Version-Release number of selected component (if applicable):
python-qpid-0.22-19.el6.noarch
python-qpid-0.30-3.el6.noarch


How reproducible:
100%


Steps to Reproduce:
0. decrease TCP keepalives for faster reproducer:
echo 2 > /proc/sys/net/ipv4/tcp_retries2

1. Run below script against qpidd:
#!/usr/bin/env python
from qpid.messaging import *
import datetime

conn = Connection("localhost:5672", reconnect=1)
timeout=10

try:
  conn.open()
  sess = conn.session()

  recv = sess.receiver("testQueue;{create:always}")
  
  while (1):
    print "%s: before fetch, timeout=%s" %(datetime.datetime.now(), timeout)
    msg = Message()
    try:
      msg = recv.fetch(timeout=timeout)
    except ReceiverError, e:
      print e
    print "%s: after fetch, msg=%s" %(datetime.datetime.now(), msg)

  sess.close()

except ReceiverError, e:
  print e
except KeyboardInterrupt:
  pass

conn.close()

2. Simulate network outage:
iptables -A OUTPUT -p tcp --dport 5672 -j REJECT; date

3. Once python script writes "No handlers could be found for logger "qpid.messaging"", flush iptables (iptables -F)

4. Wait 10 seconds


Actual results:
python-qpid-0.22* raises exception:
File "/root/Python_MRG/test_gofer_like.py", line 18, in <module> msg = recv.fetch(timeout=timeout)
File "<string>", line 6, in fetch
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1041, in fetch self._ecwait(lambda: not self.draining)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait result = self._ewait(lambda: self.closed or predicate(), timeout)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 993, in _ewait result = self.session._ewait(lambda: self.error or predicate(), timeout)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 580, in _ewait result = self.connection._ewait(lambda: self.error or predicate(), timeout)
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 219, in _ewait self.check_error()
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 212, in check_error raise e InternalError: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 663, in write op.dispatch(self)
File "/usr/lib/python2.7/site-packages/qpid/ops.py", line 84, in dispatch getattr(target, handler)(self, *args)
File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 888, in do_session_detached sst = self._sessions.pop(dtc.channel) KeyError: 0

python-qpid-0.30* raises exception:
  File "Python_MRG/test_gofer_like.py", line 18, in <module>
    msg = recv.fetch(timeout=timeout)
  File "<string>", line 6, in fetch
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1066, in fetch
    self._ecwait(lambda: not self.draining)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
    result = self._ewait(lambda: self.closed or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1018, in _ewait
    result = self.session._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 595, in _ewait
    result = self.connection._ewait(lambda: self.error or predicate(), timeout)
..
RuntimeError: maximum recursion depth exceeded


Expected results:
No exception raised


Additional info: