Hide Forgot
Created attachment 829390 [details] logs obtained by user Description of problem: qpidd is being used as part of a cluster, used with openstack, deployed with an haproxy. One of the hosts running a qpidd cluster member is rebooted - qpidd fails to restart. Version-Release number of selected component (if applicable): [root@Controller01 .qpidd]# rpm -qa | grep qpid qpid-qmf-0.18-18.el6_4.x86_64 qpid-cpp-client-0.18-17.el6_4.x86_64 qpid-cpp-server-0.18-17.el6_4.x86_64 python-qpid-qmf-0.18-18.el6_4.x86_64 qpid-cpp-server-cluster-0.18-17.el6_4.x86_64 python-qpid-0.18-5.el6_4.noarch qpid-tools-0.18-8.el6.noarch How reproducible: Occasional. Steps to Reproduce: 1. 2. 3. Actual results: qpidd fails to start. Expected results: qpidd should start and join the cluster. Additional info: See attached cluster logs - cluster is issuing a critical log message regarding an exclusive queue. Openstack uses these exclusive queues as temporary reply queues. [root@Controller01 .qpidd]# cat /etc/qpidd.conf # GENERATED BY PUPPET # # Configuration file for qpidd. Entries are of the form: # name=value # # (Note: no spaces on either side of '='). Using default settings: # "qpidd --help" or "man qpidd" for more details. cluster-name="openstack" #cluster-size=3 port=5672 max-connections=500 worker-threads=17 connection-backlog=10 auth=no realm=QPID log-enable=debug+:cluster log-to-file=/tmp/qpidd.log
I have 3 brokers, whenever I kill one of them, rebooting the server, It fails to start: v 28 12:52:27 Controller03 qpidd[13540]: 2013-11-28 12:52:27 [System] error Connection 192.168.30.202:5672-192.168.30.200:60498 No protocol received closing Nov 28 12:52:27 Controller03 qpidd[13540]: 2013-11-28 12:52:27 [System] error Connection 192.168.30.202:5672-192.168.30.200:60500 No protocol received closing Nov 28 12:52:33 Controller03 qpidd[13540]: 2013-11-28 12:52:33 [System] error Connection 192.168.30.202:5672-192.168.30.200:60616 No protocol received closing Nov 28 12:52:33 Controller03 qpidd[13540]: 2013-11-28 12:52:33 [System] error Connection 192.168.30.202:5672-192.168.30.200:60617 No protocol received closing
Another note: to start the node that's not working, I have to stop the remaining nodes and start them again.
MEssage after starting the node I rebooted: qpidd[3334]: 2013-11-28 17:32:00 [Protocol] error Execution exception: resource-locked: Cannot subscribe to exclusive queue reply_d0f7bd5f86c8451ab219e23d242cb9f2 (qpid/broker/SessionAdapter.cpp:414) Nov 28 17:32:00 Controller03 qpidd[3334]: 2013-11-28 17:32:00 [System] error not-found: Unknown destination 0 session=anonymous.45b850d8-2042-4d5d-b155-804f9a038ce4:0 (qpid/broker/SemanticState.cpp:580) Nov 28 17:32:00 Controller03 qpidd[3334]: 2013-11-28 17:32:00 [HA] critical cluster(192.168.30.202:3334 UPDATEE) catch-up connection closed prematurely 192.168.30.200:5672-192.168.30.200:41163(192.168.30.202:3334-30 local,catchup) Nov 28 17:32:00 Controller03 qpidd[3334]: 2013-11-28 17:32:00 [HA] notice cluster(192.168.30.202:3334 LEFT) leaving cluster openstack