Bug 486991 - clustered qpidd+store: recovery from cluster and journal: Exchange already exists: amq.direct (MessageStoreImpl.cpp:488)
Summary: clustered qpidd+store: recovery from cluster and journal: Exchange already ex...
Keywords:
Status: CLOSED DUPLICATE of bug 483807
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 1.1
Hardware: All
OS: Linux
high
high
Target Milestone: 1.3
: ---
Assignee: Kim van der Riet
QA Contact: Jan Sarenik
URL:
Whiteboard:
: 485254 491022 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-02-23 16:09 UTC by Frantisek Reznicek
Modified: 2015-11-16 00:06 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-05-08 13:52:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Frantisek Reznicek 2009-02-23 16:09:27 UTC
Description of problem:
There is qpidd start-up issue in multi-host persistent qpidd cluster.
After cluster re-start (all qpidd instances died) the cluster node started first starts up correctly, but the second one (and the others) didn't start up.

Although the second qpidd transcript claims the replication will be done from cluster, not from journal it exits with message:
...
2009-feb-23 10:31:51 notice Recovering from cluster, no recovery from local journal
2009-feb-23 10:31:51 debug Exception constructed: Exchange already exists: amq.direct (MessageStoreImpl.cpp:488)
2009-feb-23 10:31:51 debug Journal "TplStore": Destroyed
2009-feb-23 10:31:51 critical Broker start-up failed: Exchange already exists: amq.direct (MessageStoreImpl.cpp:488)


Version-Release number of selected component (if applicable):
qpidd-0.4.744917-1.el5, rhm-0.4.3116-3.el5, openais-0.80.5-2.el5


How reproducible:
100%

Steps to Reproduce:
1. let's have 2 hosts, both prepared for clustering (openais-0.80.5-2.el5 configured and running)
2. start 1st qpidd on host A
   qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd.log 2>&1
3. start 1st qpidd on host B
   qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd.log 2>&1
4. kill both brokers (for instance kill -2 <pids>)
5. start-up 2nd qpidd on host A from the previous msgstore dir(should be ok)
   qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd2.log 2>&1
6. start-up 2nd qpidd on host A from the previous msgstore dir(should exit quickly)
   qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd2.log 2>&1
7. check the qpidd2.log on host B

  
Actual results:
qpidd doesn't start-up well at point 6.

Expected results:
qpidd should start-up without any issue.

Additional info: (transcript from both hosts, A~root@nec-em13, B~root@nec-em12)

================host A================
[root@nec-em13 ~]# mkdir _oat
[root@nec-em13 ~]# cd _oat/
[root@nec-em13 _oat]# qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd.log 2>&1

[1]+  Stopped                 qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster > qpidd.log 2>&1
[root@nec-em13 _oat]# bg
[1]+ qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster > qpidd.log 2>&1 &
[root@nec-em13 _oat]# jobs
[1]+  Running                 qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster > qpidd.log 2>&1 &
[root@nec-em13 _oat]# kill -2 %1
[root@nec-em13 _oat]#
[1]+  Done                    qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster > qpidd.log 2>&1
[root@nec-em13 _oat]# qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd2.log 2>&1
[root@nec-em13 _oat]# cat qpidd.log
2009-feb-23 10:30:19 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2009-feb-23 10:30:19 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so
2009-feb-23 10:30:19 info Loaded Module: /usr/lib/qpid/daemon/rdma.so
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/daemon/acl.so
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/client/rdmaconnector.so
2009-feb-23 10:30:20 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/client/sslconnector.so
2009-feb-23 10:30:20 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2009-feb-23 10:30:20 info Management enabled
2009-feb-23 10:30:20 info ManagementBroker generated broker ID: 5069f942-2919-45e4-b249-bdc56c5f72e7
2009-feb-23 10:30:20 debug ManagementBroker boot sequence: 1
2009-feb-23 10:30:20 debug ManagementBroker added package org.apache.qpid.broker
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:system
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:broker
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:agent
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:vhost
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:queue
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:exchange
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:binding
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:connection
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:link
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:bridge
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:session
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:clientConnect
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:clientConnectFail
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:clientDisconnect
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkUp
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkDown
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:queueDeclare
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:queueDelete
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:exchangeDeclare
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:exchangeDelete
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:bind
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:unbind
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:subscribe
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.broker:unsubscribe
2009-feb-23 10:30:20 notice Journal "TplStore": Created
2009-feb-23 10:30:20 debug Journal "TplStore": Journal directory = "data/rhm/tpl/"; Base file name = "tpl"
2009-feb-23 10:30:20 notice Store module initialized; dir=data
2009-feb-23 10:30:20 info > Default files per journal: 8
2009-feb-23 10:30:20 info > Default journal file size: 24 (wpgs)
2009-feb-23 10:30:20 info > Default write cache page size: 32 (Kib)
2009-feb-23 10:30:20 info > Default number of write cache pages: 32
2009-feb-23 10:30:20 info > TPL files per journal: 8
2009-feb-23 10:30:20 info > TPL journal file size: 24 (wpgs)
2009-feb-23 10:30:20 info > TPL write cache page size: 4 (Kib)
2009-feb-23 10:30:20 info > TPL number of write cache pages: 64
2009-feb-23 10:30:20 debug Initialized CPG handle 0x12bdfb00000000
2009-feb-23 10:30:20 debug ManagementBroker added package org.apache.qpid.cluster
2009-feb-23 10:30:20 debug ManagementBroker added class org.apache.qpid.cluster:cluster
2009-feb-23 10:30:20 debug 10.16.64.155:28313(INIT) config change: 10.16.64.155:28313
2009-feb-23 10:30:20 notice SASL disabled: No Authentication Performed
2009-feb-23 10:30:20 notice Listening on TCP port 5672
2009-feb-23 10:30:20 info Registered xml exchange
libibverbs: Fatal: couldn't read uverbs ABI version.
2009-feb-23 10:30:20 info Rdma: Disabled: no rdma devices found
2009-feb-23 10:30:20 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it.
2009-feb-23 10:30:20 info Registered replication exchange
2009-feb-23 10:30:20 info Policy file not specified. ACL Disabled, no ACL checking being done!
2009-feb-23 10:30:20 debug ManagementBroker added package com.redhat.rhm.store
2009-feb-23 10:30:20 debug ManagementBroker added class com.redhat.rhm.store:store
2009-feb-23 10:30:20 debug ManagementBroker added class com.redhat.rhm.store:journal
2009-feb-23 10:30:20 notice 10.16.64.155:28313(INIT) joining cluster mycluster with url=amqp:tcp:10.16.64.155:5672
2009-feb-23 10:30:20 notice Broker running
2009-feb-23 10:30:20 debug 10.16.64.155:28313(INIT) cluster-id = 97f613e7-e3ef-434b-af4f-099f2dc91165
2009-feb-23 10:30:20 info 10.16.64.155:28313(READY) member update: 10.16.64.155:28313(member)
2009-feb-23 10:30:20 notice 10.16.64.155:28313(READY) first in cluster
2009-feb-23 10:30:43 debug 10.16.64.155:28313(READY) config change: 10.16.64.155:28313 10.16.64.142:15244
2009-feb-23 10:30:43 info 10.16.64.155:28313(OFFER) send update-offer to 10.16.64.142:15244
2009-feb-23 10:30:43 info 10.16.64.155:28313(UPDATER) stall for update to 10.16.64.142:15244 at amqp:tcp:10.16.64.142:5672
2009-feb-23 10:30:43 debug ConnectionImpl created for \x80-\x8A
2009-feb-23 10:30:43 info Connecting to tcp:10.16.64.142:5672
2009-feb-23 10:30:43 debug TCPConnector created for \x80-\x8A
2009-feb-23 10:30:43 debug RECV [60257 10.16.64.142:5672] INIT(0-10)
2009-feb-23 10:30:43 debug CyrusSasl::start(ANONYMOUS)
2009-feb-23 10:30:43 debug min_ssf: 0, max_ssf: 256
2009-feb-23 10:30:43 debug getUserFromSettings(): guest
2009-feb-23 10:30:43 debug CyrusSasl::start(ANONYMOUS): selected ANONYMOUS response: 'guest.bos.redhat.com'
2009-feb-23 10:30:43 debug Known-brokers for connection:
2009-feb-23 10:30:43 debug No security layer in place
2009-feb-23 10:30:43 debug No failover listener for catch-up connection.
2009-feb-23 10:30:43 debug SessionState::SessionState @: 0xb2000724
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating state to 10.16.64.142:15244 at amqp:tcp:10.16.64.142:5672
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange amq.direct
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange amq.failover
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange amq.fanout
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange amq.match
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange amq.topic
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updating exchange qpid.management
2009-feb-23 10:30:43 debug Exception constructed: Closed by client
2009-feb-23 10:30:43 debug 10.16.64.155:28313 updated state to 10.16.64.142:15244 at amqp:tcp:10.16.64.142:5672
2009-feb-23 10:30:43 info 10.16.64.155:28313(UPDATER) sent update
2009-feb-23 10:30:43 info 10.16.64.155:28313(READY) member update: 10.16.64.142:15244(member) 10.16.64.155:28313(member)
2009-feb-23 10:31:08 notice 10.16.64.155:28313(READY) shutting down
2009-feb-23 10:31:08 debug Shutting down CPG
2009-feb-23 10:31:08 notice Shut down
2009-feb-23 10:31:08 debug Journal "TplStore": Destroyed
[root@nec-em13 _oat]# cat qpidd2.log
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/rdma.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/acl.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/client/rdmaconnector.so
2009-feb-23 10:31:30 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/client/sslconnector.so
2009-feb-23 10:31:30 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2009-feb-23 10:31:30 info Management enabled
2009-feb-23 10:31:30 debug ManagementBroker restored broker ID: 5069f942-2919-45e4-b249-bdc56c5f72e7
2009-feb-23 10:31:30 debug ManagementBroker boot sequence: 2
2009-feb-23 10:31:30 debug ManagementBroker added package org.apache.qpid.broker
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:system
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:broker
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:agent
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:vhost
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:queue
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:exchange
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:binding
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:connection
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:link
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:bridge
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:session
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:clientConnect
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:clientConnectFail
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:clientDisconnect
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkUp
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkDown
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:queueDeclare
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:queueDelete
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:exchangeDeclare
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:exchangeDelete
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:bind
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:unbind
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:subscribe
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.broker:unsubscribe
2009-feb-23 10:31:30 notice Journal "TplStore": Created
2009-feb-23 10:31:30 debug Journal "TplStore": Journal directory = "data/rhm/tpl/"; Base file name = "tpl"
2009-feb-23 10:31:30 notice Store module initialized; dir=data
2009-feb-23 10:31:30 info > Default files per journal: 8
2009-feb-23 10:31:30 info > Default journal file size: 24 (wpgs)
2009-feb-23 10:31:30 info > Default write cache page size: 32 (Kib)
2009-feb-23 10:31:30 info > Default number of write cache pages: 32
2009-feb-23 10:31:30 info > TPL files per journal: 8
2009-feb-23 10:31:30 info > TPL journal file size: 24 (wpgs)
2009-feb-23 10:31:30 info > TPL write cache page size: 4 (Kib)
2009-feb-23 10:31:30 info > TPL number of write cache pages: 64
2009-feb-23 10:31:30 debug Initialized CPG handle 0x15dac09600000000
2009-feb-23 10:31:30 debug ManagementBroker added package org.apache.qpid.cluster
2009-feb-23 10:31:30 debug ManagementBroker added class org.apache.qpid.cluster:cluster
2009-feb-23 10:31:30 debug 10.16.64.155:28327(INIT) config change: 10.16.64.155:28327
2009-feb-23 10:31:30 notice SASL disabled: No Authentication Performed
2009-feb-23 10:31:30 notice Listening on TCP port 5672
2009-feb-23 10:31:30 info Registered xml exchange
libibverbs: Fatal: couldn't read uverbs ABI version.
2009-feb-23 10:31:30 info Rdma: Disabled: no rdma devices found
2009-feb-23 10:31:30 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it.
2009-feb-23 10:31:30 info Registered replication exchange
2009-feb-23 10:31:30 info Policy file not specified. ACL Disabled, no ACL checking being done!
2009-feb-23 10:31:30 debug ManagementBroker added package com.redhat.rhm.store
2009-feb-23 10:31:30 debug ManagementBroker added class com.redhat.rhm.store:store
2009-feb-23 10:31:30 debug ManagementBroker added class com.redhat.rhm.store:journal
2009-feb-23 10:31:30 notice 10.16.64.155:28327(INIT) joining cluster mycluster with url=amqp:tcp:10.16.64.155:5672
2009-feb-23 10:31:30 notice Broker running
2009-feb-23 10:31:30 debug 10.16.64.155:28327(INIT) cluster-id = 0d81149d-0e4a-439c-a223-ca9b3c4d5dd0
2009-feb-23 10:31:30 info 10.16.64.155:28327(READY) member update: 10.16.64.155:28327(member)
2009-feb-23 10:31:30 notice 10.16.64.155:28327(READY) first in cluster
2009-feb-23 10:31:51 debug 10.16.64.155:28327(READY) config change: 10.16.64.155:28327 10.16.64.142:15257
2009-feb-23 10:31:51 debug 10.16.64.155:28327(READY) config change: 10.16.64.155:28327 ( 10.16.64.142:15257 (process-down) )
2009-feb-23 10:31:56 notice 10.16.64.155:28327(READY) shutting down
2009-feb-23 10:31:56 debug Shutting down CPG
2009-feb-23 10:31:56 notice Shut down
2009-feb-23 10:31:56 debug Journal "TplStore": Destroyed

================host B================
[root@nec-em12 ~]# mkdir _oat
[root@nec-em12 ~]# cd _oat/
[root@nec-em12 _oat]# qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd.log 2>&1 &
[1] 15244
[root@nec-em12 _oat]#
[root@nec-em12 _oat]# jobs
[1]+  Running                 qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster > qpidd.log 2>&1 &
[root@nec-em12 _oat]# kill -2 %1
[root@nec-em12 _oat]#
[1]+  Done                    qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster > qpidd.log 2>&1
[root@nec-em12 _oat]# qpidd --auth no --data-dir data --log-enable debug+ --cluster-name mycluster >qpidd2.log 2>&1
[root@nec-em12 _oat]# vi qpidd2.log
[root@nec-em12 _oat]#

[root@nec-em12 _oat]# cat qpidd.log
2009-feb-23 10:30:42 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2009-feb-23 10:30:42 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2009-feb-23 10:30:43 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/client/rdmaconnector.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2009-feb-23 10:30:43 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2009-feb-23 10:30:43 info Management enabled
2009-feb-23 10:30:43 info ManagementBroker generated broker ID: 864fbba4-f9a7-4163-8009-f6d689e08be1
2009-feb-23 10:30:43 debug ManagementBroker boot sequence: 1
2009-feb-23 10:30:43 debug ManagementBroker added package org.apache.qpid.broker
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:system
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:broker
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:agent
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:vhost
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:queue
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:exchange
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:binding
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:connection
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:link
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:bridge
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:session
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:clientConnect
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:clientConnectFail
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:clientDisconnect
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkUp
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkDown
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:queueDeclare
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:queueDelete
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:exchangeDeclare
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:exchangeDelete
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:bind
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:unbind
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:subscribe
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.broker:unsubscribe
2009-feb-23 10:30:43 debug Initialized CPG handle 0x1ccf7c2300000000
2009-feb-23 10:30:43 debug ManagementBroker added package org.apache.qpid.cluster
2009-feb-23 10:30:43 debug ManagementBroker added class org.apache.qpid.cluster:cluster
2009-feb-23 10:30:43 debug 10.16.64.142:15244(INIT) config change: 10.16.64.142:15244 10.16.64.155:28313
2009-feb-23 10:30:43 notice Journal "TplStore": Created
2009-feb-23 10:30:43 debug Journal "TplStore": Journal directory = "data/rhm/tpl/"; Base file name = "tpl"
2009-feb-23 10:30:43 notice Store module initialized; dir=data
2009-feb-23 10:30:43 info > Default files per journal: 8
2009-feb-23 10:30:43 info > Default journal file size: 24 (wpgs)
2009-feb-23 10:30:43 info > Default write cache page size: 32 (Kib)
2009-feb-23 10:30:43 info > Default number of write cache pages: 32
2009-feb-23 10:30:43 info > TPL files per journal: 8
2009-feb-23 10:30:43 info > TPL journal file size: 24 (wpgs)
2009-feb-23 10:30:43 info > TPL write cache page size: 4 (Kib)
2009-feb-23 10:30:43 info > TPL number of write cache pages: 64
2009-feb-23 10:30:43 notice Recovering from cluster, no recovery from local journal
2009-feb-23 10:30:43 notice SASL disabled: No Authentication Performed
2009-feb-23 10:30:43 notice Listening on TCP port 5672
2009-feb-23 10:30:43 info Registered xml exchange
2009-feb-23 10:30:43 notice 10.16.64.142:15244(INIT) joining cluster mycluster with url=amqp:tcp:10.16.64.142:5672
2009-feb-23 10:30:43 info Policy file not specified. ACL Disabled, no ACL checking being done!
2009-feb-23 10:30:43 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it.
libibverbs: Fatal: couldn't read uverbs ABI version.
2009-feb-23 10:30:43 info Rdma: Disabled: no rdma devices found
2009-feb-23 10:30:43 debug ManagementBroker added package com.redhat.rhm.store
2009-feb-23 10:30:43 debug ManagementBroker added class com.redhat.rhm.store:store
2009-feb-23 10:30:43 debug ManagementBroker added class com.redhat.rhm.store:journal
2009-feb-23 10:30:43 info Registered replication exchange
2009-feb-23 10:30:43 notice Broker running
2009-feb-23 10:30:43 info 10.16.64.142:15244(JOINER) joining cluster: 10.16.64.142:15244(unknown) 10.16.64.155:28313(unknown)
2009-feb-23 10:30:43 debug 10.16.64.142:15244(JOINER) cluster-id = 97f613e7-e3ef-434b-af4f-099f2dc91165
2009-feb-23 10:30:43 info 10.16.64.142:15244(UPDATEE) receiving update from 10.16.64.155:28313
2009-feb-23 10:30:43 debug RECV [10.16.64.155:60257] INIT(128-138)
2009-feb-23 10:30:43 debug SASL: No Authentication Performed
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) new connection: 10.16.64.142:15244-0x179883a0(local,catchup)
2009-feb-23 10:30:43 debug SessionState::SessionState update_shared@anonymous: 0x17989000
2009-feb-23 10:30:43 debug update_shared@anonymous: attached on broker.
2009-feb-23 10:30:43 debug Attached channel 1 to update_shared@anonymous
2009-feb-23 10:30:43 debug update_shared@anonymous: ready to send, activating output.
2009-feb-23 10:30:43 debug update_shared@anonymous: receiver command-point set to: (0+0)
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange amq.direct
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange amq.failover
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange amq.fanout
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange amq.match
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange amq.topic
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) decoded exchange qpid.management
2009-feb-23 10:30:43 debug update_shared@anonymous: recv cmd 0: {ExchangeDeclareBody: exchange=qpid.qpid-update; type=fanout; alternate-exchange=; auto-delete=1; arguments={}; }
2009-feb-23 10:30:43 debug update_shared@anonymous: receiver marked completed: 0 incomplete: { } unknown-completed: { [0,0] }
2009-feb-23 10:30:43 debug update_shared@anonymous: recv cmd 1: {QueueDeclareBody: queue=qpid.qpid-update; alternate-exchange=; auto-delete=1; arguments={}; }
2009-feb-23 10:30:43 debug Configured queue with no-local=0
2009-feb-23 10:30:43 debug Configured queue qpid.qpid-update with qpid.trace.id='' and qpid.trace.exclude='' i.e. 0 elements
2009-feb-23 10:30:43 debug update_shared@anonymous: receiver marked completed: 1 incomplete: { } unknown-completed: { [0,1] }
2009-feb-23 10:30:43 debug update_shared@anonymous: recv cmd 2: {ExecutionSyncBody: }
2009-feb-23 10:30:43 debug update_shared@anonymous: receiver marked completed: 2 incomplete: { } unknown-completed: { [0,2] }
2009-feb-23 10:30:43 debug update_shared@anonymous: detached on broker.
2009-feb-23 10:30:43 debug 10.16.64.142:15244(UPDATEE) incoming update complete on connection 10.16.64.142:15244-0x179883a0(local,catchup)
2009-feb-23 10:30:43 info 10.16.64.142:15244(CATCHUP) received update, starting catch-up
2009-feb-23 10:30:43 info 10.16.64.142:15244(CATCHUP) member update: 10.16.64.142:15244(member) 10.16.64.155:28313(member)
2009-feb-23 10:30:43 notice 10.16.64.142:15244(READY) caught up, active cluster member
2009-feb-23 10:30:43 debug DISCONNECTED [10.16.64.155:60257]
2009-feb-23 10:30:43 debug 10.16.64.142:15244(READY) closed update connection 10.16.64.142:15244-0(updated)
2009-feb-23 10:30:43 debug 10.16.64.142:15244(READY) deleted connection: 10.16.64.142:15244-0(updated)
2009-feb-23 10:30:45 info Passivating links
2009-feb-23 10:31:08 debug 10.16.64.142:15244(READY) config change: 10.16.64.142:15244 (joined) ( 10.16.64.155:28313 (process-down) )
2009-feb-23 10:31:08 info 10.16.64.142:15244(READY) member update: 10.16.64.142:15244(member)
2009-feb-23 10:31:08 info 10.16.64.142:15244(READY) last broker standing, update queue policies
2009-feb-23 10:31:09 info Activating links
2009-feb-23 10:31:16 notice 10.16.64.142:15244(READY) shutting down
2009-feb-23 10:31:16 debug Shutting down CPG
2009-feb-23 10:31:16 notice Shut down
2009-feb-23 10:31:16 debug Journal "TplStore": Destroyed


[root@nec-em12 _oat]# cat qpidd2.log
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2009-feb-23 10:31:51 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/client/rdmaconnector.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2009-feb-23 10:31:51 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2009-feb-23 10:31:51 info Management enabled
2009-feb-23 10:31:51 debug ManagementBroker restored broker ID: 864fbba4-f9a7-4163-8009-f6d689e08be1
2009-feb-23 10:31:51 debug ManagementBroker boot sequence: 2
2009-feb-23 10:31:51 debug ManagementBroker added package org.apache.qpid.broker
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:system
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:broker
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:agent
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:vhost
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:queue
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:exchange
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:binding
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:connection
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:link
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:bridge
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:session
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:clientConnect
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:clientConnectFail
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:clientDisconnect
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkUp
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkDown
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:queueDeclare
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:queueDelete
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:exchangeDeclare
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:exchangeDelete
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:bind
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:unbind
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:subscribe
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.broker:unsubscribe
2009-feb-23 10:31:51 debug Initialized CPG handle 0x6b8b456700000000
2009-feb-23 10:31:51 debug ManagementBroker added package org.apache.qpid.cluster
2009-feb-23 10:31:51 debug ManagementBroker added class org.apache.qpid.cluster:cluster
2009-feb-23 10:31:51 debug 10.16.64.142:15257(INIT) config change: 10.16.64.142:15257 10.16.64.155:28327
2009-feb-23 10:31:51 notice Journal "TplStore": Created
2009-feb-23 10:31:51 debug Journal "TplStore": Journal directory = "data/rhm/tpl/"; Base file name = "tpl"
2009-feb-23 10:31:51 notice Store module initialized; dir=data
2009-feb-23 10:31:51 info > Default files per journal: 8
2009-feb-23 10:31:51 info > Default journal file size: 24 (wpgs)
2009-feb-23 10:31:51 info > Default write cache page size: 32 (Kib)
2009-feb-23 10:31:51 info > Default number of write cache pages: 32
2009-feb-23 10:31:51 info > TPL files per journal: 8
2009-feb-23 10:31:51 info > TPL journal file size: 24 (wpgs)
2009-feb-23 10:31:51 info > TPL write cache page size: 4 (Kib)
2009-feb-23 10:31:51 info > TPL number of write cache pages: 64
2009-feb-23 10:31:51 notice Recovering from cluster, no recovery from local journal
2009-feb-23 10:31:51 debug Exception constructed: Exchange already exists: amq.direct (MessageStoreImpl.cpp:488)
2009-feb-23 10:31:51 debug Journal "TplStore": Destroyed
2009-feb-23 10:31:51 critical Broker start-up failed: Exchange already exists: amq.direct (MessageStoreImpl.cpp:488)

Comment 1 Gordon Sim 2009-02-23 16:19:33 UTC
Believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=483807, but as it has a nice concrete set of steps for observing the issue, this is probably the one to keep open?

Comment 2 Gordon Sim 2009-03-02 11:30:37 UTC
*** Bug 485254 has been marked as a duplicate of this bug. ***

Comment 3 Gordon Sim 2009-03-06 15:08:31 UTC
Workaround here is to use a fresh (empty) store when adding nodes to an existing cluster.

Comment 4 Gordon Sim 2009-03-19 09:06:04 UTC
*** Bug 491022 has been marked as a duplicate of this bug. ***

Comment 5 Charlie Wyse 2009-04-01 17:53:49 UTC
We have been able to resolve the issue by deleting /var/lib/rhm before starting qpidd.  Is there any negative affects this might have on qpidd that anyone could point out for me?

Comment 7 Kim van der Riet 2009-05-08 13:52:33 UTC

*** This bug has been marked as a duplicate of bug 483807 ***


Note You need to log in before you can comment on or make changes to this bug.