Description of problem:
qpid broker sometimes forgets to send connection.heartbeat. It forgets so on quite random basis, the more connections it handles the (much) bigger probability of doing so is. I.e. for 100 connections with heartbeat=5, there are few such "misses" in 30 minutes, for 500 conns with hb=5, usually all connections are affected within 30minutes.

Reproducer is so so trivial: just have sufficient connections with heartbeats set.

I noticed this is a regression due to bz1093996 / QPID-5758 / svn commit 1594220. I.e. regression between MRG-M 3.0 (not present) and MRG-M 3.1 (present). I confirmed the regression by running today's upstream qpid (bug present) and qpid with commit 1594220 rolled back (no bug).


Version-Release number of selected component (if applicable):
qpid-cpp-server-0.30-2


How reproducible:
100% within 30 minutes


Steps to Reproduce:
1. qpidd --queue-purge-interval=20 --auth=no --max-connections=10000 &
2. run below written script ./reproduce_ttl_timeout.sh that creates 500 queues on 500 connections with heartbeat=5. The script checks every 10 seconds if number of client connections has not decreased.


Actual results:
script terminates soon with:

error: found just 225 connections instead of 500, in iteration 2


Expected results:
script to terminate after 30 minutes with "no error"


Additional info:
script:

noDelQueues=500

maxIter=180
mySleep=10

echo "creating connections.."
for i in $(seq 1 $noDelQueues); do qpid-receive --connection-options "{'heartbeat':5}" -a "autoDelQueueNoBound_${i}; {create:always, node:{ x-declare:{auto-delete:True, arguments:{'qpid.auto_delete_timeout':1}}}}" -f --print-content=no > /dev/null 2>&1 & sleep 0.1; done

iter=0
while true; do
  iter=$(($((iter))+1))
  conns=$(pgrep qpid-receive | wc -w)
  echo "$(date): iteration:$iter connections:$conns"
  if [ $conns -lt $noDelQueues ]; then
    echo "error: found just $conns connections instead of $noDelQueues, in iteration $iter"
    break
  fi
  if [ $iter -eq $maxIter ]; then
    echo "no error"
    break
  fi
  sleep $mySleep
done