Bug 1001772

Fixed upstream: https://svn.apache.org/r1518233 

(applying after https://bugzilla.redhat.com/show_bug.cgi?id=981636 and https://bugzilla.redhat.com/show_bug.cgi?id=967734 will make merge easier)

Backported: http://git.app.eng.bos.redhat.com/?p=rh-qpid.git;a=shortlog;h=refs/heads/0.22-mrg-BZ1001772

Already connected consumer is not able to reconnect after broker restart.

1. service qpidd stop
2. $cppapi/drain -f --connection-options "{protocol:'amqp1.0', reconnect: True}" amq.direct
3. service qpidd start
4. $cppapi/spout amq.direct
5. message received

This is OK so far, the client was started when the broker was down and was able to connect after the broker was started

6. service qpidd restart
7. $cppapi/spout amq.direct
8. message NOT received, the consumer has not reconnect

Note: Producer (ie. spout) do not suffer from that (may reconnect multiple times)

Client's seems to be stuck in fetch method call:

Thread 2 (Thread 0xb77beb70 (LWP 23957)):
#0  0x008af416 in __kernel_vsyscall ()
#1  0x0043d5e6 in epoll_wait () from /lib/libc.so.6
#2  0x04fd50cc in qpid::sys::Poller::wait (this=0x856a580, timeout=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:566
#3  0x04fd58a3 in qpid::sys::Poller::run (this=0x856a580) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:518
#4  0x04fc9981 in qpid::sys::(anonymous namespace)::runRunnable (p=0x856a580) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Thread.cpp:35
#5  0x004f9b39 in start_thread () from /lib/libpthread.so.0
#6  0x0043cd6e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb77c09d0 (LWP 23956)):
#0  0x008af416 in __kernel_vsyscall ()
#1  0x004fd794 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0044c9f4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x009685d1 in wait (this=0x856dc58, until=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Condition.h:69
#4  wait (this=0x856dc58, until=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Monitor.h:45
#5  qpid::messaging::amqp::ConnectionContext::waitUntil (this=0x856dc58, until=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/messaging/amqp/ConnectionContext.cpp:435
#6  0x0096887c in qpid::messaging::amqp::ConnectionContext::waitUntil (this=0x856dc58, ssn=..., lnk=..., until=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/messaging/amqp/ConnectionContext.cpp:460
#7  0x00968b0b in qpid::messaging::amqp::ConnectionContext::get (this=0x856dc58, ssn=..., lnk=..., message=..., timeout=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/messaging/amqp/ConnectionContext.cpp:216
#8  0x0096a7ad in qpid::messaging::amqp::ConnectionContext::fetch (this=0x856dc58, ssn=..., lnk=..., message=..., timeout=...)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/messaging/amqp/ConnectionContext.cpp:151
#9  0x00972195 in qpid::messaging::amqp::ReceiverHandle::fetch (this=0x856f7d0, message=..., timeout=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/messaging/amqp/ReceiverHandle.cpp:55
#10 0x009bd3ae in qpid::messaging::Receiver::fetch (this=0xbfc83490, message=..., timeout=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/messaging/Receiver.cpp:47
#11 0x0804d94d in main ()

The issue here is that the receiver in question has no capacity set and the 1.0 path doesn't allocate credit in this case after failover (qpid-receive has non-zero capacity by default which is why it works and drain doesn't). I've committed a fix upstream: https://svn.apache.org/r1546415

QE note: issue from comment 7 has been fixed

Reconnect capability available and working for amqp1.0 c++ client.  Verified on rhel6.5 (x86_64, i386).

packages under test:
qpid-cpp-*-0.22-30.el6

-> VERIFIED