Bug 1624896

Description of problem:
A race condition leads to qpidd segfault when:

- ACLs are set (to whatever content, maybe just the object exists and no ACL file present),
- broker is shutting down, and
- broker is just deleting auto-delete queue with deletion timeout

The backtrace is:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f6a1a84763b in qpid::broker::Queue::tryAutoDelete (this=0x1e205a0, expectedVersion=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1361
#2  0x00007f6a1a3532db in qpid::sys::Timer::fire (this=<value optimized out>, t=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/Timer.cpp:222
#3  0x00007f6a1a354905 in qpid::sys::Timer::run (this=0x186dd80) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/Timer.cpp:156
#4  0x00007f6a1a2e355a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/posix/Thread.cpp:35
#5  0x00007f6a197dcaa1 in start_thread (arg=0x7f6a1783a700) at pthread_create.c:301
#6  0x00007f6a1952993d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)


Version-Release number of selected component (if applicable):
qpid-cpp-server 1.36.0-20


How reproducible:
100% within 30 minutes (race condition)


Steps to Reproduce:
1. Run below script:

echo "acl allow all all" > /tmp/qpidd.acl
killall qpidd
sleep 1

while true; do
	qpidd --log-to-stdout=no --log-to-stderr=no --log-to-file=/tmp/qpidd.log --auth=no --acl-file=/tmp/qpidd.acl --log-enable=debug+ --log-hires-timestamp=yes & pid=$!
	sleep 2
	echo "$(date): creating some content.."
	for i in $(seq 1 100); do qpid-send -a "someQueue_${i}; {create:always}" -m 100 & done
	while [ $(pgrep qpid-send | wc -l) -gt 0 ]; do sleep 1; done
	echo "$(date): spawning queues"
	for j in $(seq 1 3); do
		for i in $(seq 1 30); do
			qName="autoDelQueue_${j}_${i}"
			qpid-receive -a "${qName}; {create:always, node:{ x-declare:{auto-delete:True, arguments:{'qpid.auto_delete_timeout':1}}}}" --print-content=no --timeout=1 &
		done &
	done &
	echo "$(date): queues spawned"
	sleep 2.5
	echo "$(date): broker shutting down"
	kill $pid
	sleep 3
done

2. Let the script running several iterations and monitor in another terminal:

tail -f /tmp/qpidd.log | grep -e "Auto-delete queue deleted" -e "shutting down" -e "shut-down"

3. there should be some "Auto-delete queue deleted:" logs just before "shutting down" log - if not, modify the "sleep 2.5" accordingly.

4. Let it running till segfaults during shutdown occur.


Actual results:
4. some segfaults during shutdown occur (once per a minute on my system)


Expected results:
4. no segfault


Additional info:
The test:

            if (broker->getAcl())

is insufficient for the case when broker->acl class has been just destroyed, such that calling:

                broker->getAcl()->recordDestroyQueue(name);

fails.