Bug 1590760

forgot to add the delete_queue source code (simply C++ client deleting the queue via QMF):

# cat delete_queue.cpp
#include <cstdlib>
#include <iostream>

#include <sstream>

#include <qpid/messaging/Address.h>
#include <qpid/messaging/Connection.h>
#include <qpid/messaging/Message.h>
#include <qpid/messaging/Sender.h>
#include <qpid/messaging/Receiver.h>
#include <qpid/messaging/Session.h>


using namespace qpid::messaging;
using namespace qpid::types;

using std::stringstream;
using std::string;

int main(int argc, char** argv) {
    const char* queue_name = argc>1 ? argv[1] : "queue_name";
    const char* url = argc>2 ? argv[2] : "amqp:tcp:127.0.0.1:5672";

    Connection connection(url/*, connectionOptions*/);
    try {
        connection.open();
        Session session = connection.createSession();
        Sender sender = session.createSender("qmf.default.direct/broker");
        Address responseQueue("#reply-queue; {create:always, node:{x-declare:{auto-delete:true}}}");
        Receiver receiver = session.createReceiver(responseQueue);

        Message message;
        Variant::Map content;
	Variant::Map OID;
	Variant::Map arguments;
	OID["_object_name"] = "org.apache.qpid.broker:broker:amqp-broker";
	arguments["type"] = "queue";
	arguments["name"] = queue_name;
	
        content["_object_id"] = OID;
        content["_method_name"] = "delete";
        content["_arguments"] = arguments;
	
        encode(content, message);
	message.setReplyTo(responseQueue);
	message.setProperty("x-amqp-0-10.app-id", "qmf2");
	message.setProperty("qmf.opcode", "_method_request");

        sender.send(message, true);
	
	Message response;
	if (receiver.fetch(response,qpid::messaging::Duration(30000)) == true)
	{
		qpid::types::Variant::Map recv_props = response.getProperties();
		if (recv_props["x-amqp-0-10.app-id"] == "qmf2")
			if (recv_props["qmf.opcode"] == "_method_response")
				std::cout << "Response: OK" << std::endl;
			else if (recv_props["qmf.opcode"] == "_exception")
				std::cerr << "Error: " << response.getContent() << std::endl;
			else
				std::cerr << "Invalid response received!" << std::endl;
		else
			std::cerr << "Invalid response not of qmf2 type received!" << std::endl;
	}
	else
		std::cout << "Timeout: No response received within 30 seconds!" << std::endl;

        connection.close();
        return 0;
    } catch(const std::exception& error) {
        std::cout << error.what() << std::endl;
        connection.close();
    }
    return 1;
}

.. and confirmed it is reproducible in upstream

There is a time gap between de-registering a queue from the QueueRegistry (with its QueueRegistry lock) and marking the queue as deleted (later with the Queue::messageLock via Queue::destroyed()).  

During this gap, QueueRegistry methods (QueueRegistry::destroyIfUntouched, QueueRegistry::destroy) consider the queue deleted, Queue methods assume it is still alive.

The gap is usually small and matters not.  But it can be arbitrarily large depending on circumstances and the test case shows that.  On beefier hardware, increasing the number of queues in the script (queues=125) increases thread competition so that the gap may be large enough to see the failure sooner.

The existing use of multiple locks is obviously important for performance when destruction is expensive and requires disengaging multiple observers, rationalizing exchanges, emitting management events, and unwinding durable storage.  The commit log shows a history of MP issues around deletion including deadlocks, so a conservative fix is probably in order.  See also QPID-7306.

Query whether a separate Queue::deletionPending lock/mutex would provide better MP logic with minimal performance impact (i.e. only causing contention on corner cases) when processing code affected by Queue::deleted.

This issue was tested with RHEL 6 and 7, with following packages:

qpid-cpp-server-1.36.0-21

Fix works as expected.
->VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0390