Bug 1295957

Description of problem:
Having qdrouterd shut down for some time (such that goferd retries several times to reconnect) and starting it again, goferd connects to it properly. BUT there is a chance it _will_ start consume 100% CPU after a while (within 1 minute since successful reconnect).

The bug is supposed to be in proton reactor - see below for analysis.


Version-Release number of selected component (if applicable):
qpid-proton-c-0.9-11.el7.x86_64
python-gofer-2.6.8-1.el7sat.noarch
python-gofer-proton-2.6.8-1.el7sat.noarch
qpid-proton-debuginfo-0.9-7.el7.x86_64
gofer-2.6.8-1.el7sat.noarch
python-qpid-proton-0.9-11.el7.x86_64


How reproducible:
??? (100% within several minutes when one knows the tricks)


Steps to Reproduce:
1. Follow reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=1272596 but stop the qdrouterd bouncing after some time - or increase the delay when qdrouterd is running substantially
2. Monitor CPU usage of goferd - notice that the high CPU usage (once hit) is put to normal values after qdrouterd restart (i.e. qdrouterd restart is a workaround) - thats why stop the qdrouterd bouncing


Actual results:
- goferd taking 100% CPU
- it has 1 established TCP connection to qdrouterd (i.e. not the original BZ where no connection was opened at the end)


Expected results:
- no high CPU usage


Additional info:
The only busy thread is Thread 21 with usual backtrace:

/usr/lib64/python2.7/site-packages/proton/__init__.py, line 3632, in wrap
/usr/lib64/python2.7/site-packages/proton/__init__.py, line 3736, in dispatch
/usr/lib64/python2.7/site-packages/proton/reactor.py, line 142, in process
/usr/lib64/python2.7/site-packages/proton/utils.py, line 252, in wait
/usr/lib64/python2.7/site-packages/proton/utils.py, line 144, in receive
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/consumer.py, line 106, in get
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py, line 44, in _fn
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py, line 614, in get
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py, line 39, in _fn
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py, line 648, in next
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py, line 39, in _fn
/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py, line 88, in _read
/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py, line 58, in run
/usr/lib/python2.7/site-packages/gofer/common.py, line 267, in _fn
/usr/lib64/python2.7/threading.py, line 811, in __bootstrap_inner
/usr/lib64/python2.7/threading.py, line 784, in __bootstrap

Adding a logging event to /usr/lib64/python2.7/site-packages/proton/reactor.py, line 142, it is already inside the busy loop

Adding a logging event to /usr/lib64/python2.7/site-packages/proton/utils.py, line 252, just one such event is logged every 10 seconds (hint, hint, why 10 seconds? be aware I have in /usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py "MAX_DELAY = 1" so not _this_ parameter).

Therefore, the busy loop is:
/usr/lib64/python2.7/site-packages/proton/utils.py:

                deadline = time.time() + timeout
                while not condition() and not self.disconnected:
                    self.container.process()
                    if deadline < time.time():
                        txt = "Connection %s timed out" % self.url
                        if msg: txt += ": " + msg
                        raise Timeout(txt)


I can try preparing proton reactor standalone reproducer, if/once confirmed this is reactor bug.