Bug 479031

Summary: Cluster member can't be added while management session open
Product: Red Hat Enterprise MRG Reporter: Ted Ross <tross>
Component: qpid-cppAssignee: Alan Conway <aconway>
Status: CLOSED ERRATA QA Contact: Frantisek Reznicek <freznice>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.1CC: aconway, dominic.forrest, esammons, gsim
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Having an open management session no longer prevents a broker from being added to a cluster.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-14 15:58:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The reproducer for bz479031 none

Description Ted Ross 2009-01-06 16:34:20 UTC 
This occurs with the following RPMs:

qpidd-0.4.728142-1.fc10.i386
python-qpid-0.4.728142-2.fc10.noarch


I've set up a two-broker cluster running on the same system.  The brokers are running on ports 5672 and 10000.

Here's the sequence:

1) Start broker on 5672
2) Start broker on 10000
3) Run qpid-tool against the broker on 5672 (I used the real IP address, not localhost)
4) Stop the broker on 10000
5) Restart the broker on 10000  - It fails (catch-up connection closed prematurely)
6) Exit qpid-tool
7) Restart the broker on 10000  - Success
Comment 1 Ted Ross 2009-01-06 16:36:18 UTC 
Full trace from the failed restart:

$ /usr/sbin/qpidd --cluster-name TED -t --auth no -p 10000 --data-dir .qpidd10000
2009-jan-06 11:35:06 info Loaded Module: /usr/lib/qpid/daemon/acl.so
2009-jan-06 11:35:06 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2009-jan-06 11:35:06 warning SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-jan-06 11:35:06 info Loaded Module: /usr/lib/qpid/client/sslconnector.so
2009-jan-06 11:35:06 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2009-jan-06 11:35:06 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
2009-jan-06 11:35:06 info Management enabled
2009-jan-06 11:35:06 debug ManagementBroker restored broker ID: a139cb60-9716-437b-8060-5b619e835cf9
2009-jan-06 11:35:06 debug ManagementBroker boot sequence: 45
2009-jan-06 11:35:06 debug ManagementBroker added package org.apache.qpid.broker
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:system
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:broker
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:agent
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:vhost
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:queue
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:exchange
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:binding
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:connection
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:link
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:bridge
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:session
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:clientConnect
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:clientConnectFail
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:clientDisconnect
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkUp
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:brokerLinkDown
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:queueDeclare
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:queueDelete
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:exchangeDeclare
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:exchangeDelete
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:bind
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:unbind
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:subscribe
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.broker:unsubscribe
2009-jan-06 11:35:06 notice Journal "TplStore": Created
2009-jan-06 11:35:06 debug Journal "TplStore": Journal directory = ".qpidd10000/rhm/tpl/"; Base file name = "tpl"
2009-jan-06 11:35:06 notice Store module initialized; dir=.qpidd10000
2009-jan-06 11:35:06 info > Default files per journal: 8
2009-jan-06 11:35:06 info > Default journal file size: 24 (wpgs)
2009-jan-06 11:35:06 info > Default write cache page size: 32 (Kib)
2009-jan-06 11:35:06 info > Default number of write cache pages: 32
2009-jan-06 11:35:06 info > TPL files per journal: 8
2009-jan-06 11:35:06 info > TPL journal file size: 24 (wpgs)
2009-jan-06 11:35:06 info > TPL write cache page size: 4 (Kib)
2009-jan-06 11:35:06 info > TPL number of write cache pages: 64
2009-jan-06 11:35:06 warning Deleting stale binding
2009-jan-06 11:35:06 info Listening on TCP port 10000
2009-jan-06 11:35:06 info Policy file not specified. ACL Disabled, no ACL checking being done!
2009-jan-06 11:35:06 warning SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it.
2009-jan-06 11:35:06 debug Initialized CPG handle 0x491ecf6c00000000
2009-jan-06 11:35:06 debug ManagementBroker added package org.apache.qpid.cluster
2009-jan-06 11:35:06 debug ManagementBroker added class org.apache.qpid.cluster:cluster
2009-jan-06 11:35:06 notice 10.16.18.254:17495(INIT) joining cluster TED with url=amqp:tcp:10.16.18.254:10000,tcp:10.16.15.108:10000,tcp:192.168.122.1:10000
2009-jan-06 11:35:06 debug ManagementBroker added package com.redhat.rhm.store
2009-jan-06 11:35:06 debug ManagementBroker added class com.redhat.rhm.store:store
2009-jan-06 11:35:06 debug ManagementBroker added class com.redhat.rhm.store:journal
2009-jan-06 11:35:06 debug 10.16.18.254:17495(INIT) config change: 10.16.18.254:17495 10.16.18.254:16272 (joined) 
2009-jan-06 11:35:06 trace 10.16.18.254:17495(INIT) PUSH: [event 10.16.18.254:17495-0 control 36 bytes]
2009-jan-06 11:35:06 trace 10.16.18.254:17495(INIT) DLVR: [event 10.16.18.254:17495-0 control 36 bytes] Frame[BEbe; channel=0; {ClusterConfigChangeBody: current=\xFE\x12\x10
\x00\x00DW\xFE\x12\x10
\x00\x00?\x90; }]
2009-jan-06 11:35:06 info 10.16.18.254:17495(NEWBIE) joining cluster: 10.16.18.254:16272(unknown) 10.16.18.254:17495(unknown) 
2009-jan-06 11:35:06 trace  MCAST [event 10.16.18.254:17495-0 control 94 bytes]
2009-jan-06 11:35:06 debug RECV [10.16.18.254:59986] INIT(128-138)
2009-jan-06 11:35:06 warning SASL: No Authentication Performed
2009-jan-06 11:35:06 debug 10.16.18.254:17495(NEWBIE) new connection: 10.16.18.254:17495-0x8558ed8(local,catchup)
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT 10.16.18.254:59986 INIT(0-10)
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=0; {ConnectionStartBody: server-properties={qpid.federation_tag:V2:36:str16(a139cb60-9716-437b-8060-5b619e835cf9)}; mechanisms=str16{V2:9:str16(ANONYMOUS)}; locales=str16{V2:5:str16(en_US)}; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495(NEWBIE) PUSH: [event 10.16.18.254:17495-0 control 94 bytes]
2009-jan-06 11:35:06 trace 10.16.18.254:17495(NEWBIE) PUSH: [event 10.16.18.254:16272-0 control 42 bytes]
2009-jan-06 11:35:06 trace 10.16.18.254:17495(NEWBIE) DLVR: [event 10.16.18.254:17495-0 control 94 bytes] Frame[BEbe; channel=0; {ClusterDumpRequestBody: url=amqp:tcp:10.16.18.254:10000,tcp:10.16.15.108:10000,tcp:192.168.122.1:10000; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495(NEWBIE) DLVR: [event 10.16.18.254:16272-0 control 42 bytes] Frame[BEbe; channel=0; {ClusterDumpOfferBody: dumpee=18307713070350222423; cluster-id=842c0431-a926-4b88-87b0-6ca83776fb33; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(NEWBIE) cluster-id = 842c0431-a926-4b88-87b0-6ca83776fb33
2009-jan-06 11:35:06 info 10.16.18.254:17495(DUMPEE) receiving dump from 10.16.18.254:16272
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 52 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=0; {ConnectionStartOkBody: client-properties={}; mechanism=PLAIN; response=xxxxxx; locale=en_US; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=0; {ConnectionTuneBody: channel-max=32767; max-frame-size=65535; heartbeat-min=0; heartbeat-max=0; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 49 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=0; {ConnectionTuneOkBody: channel-max=32767; max-frame-size=65535; heartbeat=0; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=0; {ConnectionOpenBody: virtual-host=/; capabilities=void{}; insist=1; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=0; {ConnectionOpenOkBody: known-hosts=str16{}; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 29 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {SessionAttachBody: name=dump_shared; }]
2009-jan-06 11:35:06 debug SessionState::SessionState dump_shared@guest@QPID: 0x859a058
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: attached on broker.
2009-jan-06 11:35:06 debug Attached channel 1 to dump_shared@guest@QPID
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: ready to send, activating output.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=1; {SessionAttachedBody: name=dump_shared; }]
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=1; {SessionCommandPointBody: command-id=0; command-offset=0; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 20 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {SessionRequestTimeoutBody: timeout=0; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=1; {SessionTimeoutBody: timeout=0; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 116 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {SessionCommandPointBody: command-id=0; command-offset=0; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver command-point set to: (0+0)
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=\x00\x00\x06direct\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange 
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=
amq.direct\x01\x06direct\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange amq.direct
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 1551 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=
                                             amq.failover\x00
                                                             amq.failover\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange amq.failover
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=
amq.fanout\x01\x06fanout\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange amq.fanout
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=	amq.match\x01\x07headers\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange amq.match
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=	amq.topic\x01\x05topic\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange amq.topic
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionExchangeBody: encoded=\x0Fqpid.management\x00
management\x00\x00\x00\x04\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded exchange qpid.management
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeDeclareBody: exchange=\x00qpid-dump\x00; type=fanout; alternate-exchange=; auto-delete=1; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 0: {ExchangeDeclareBody: exchange=\x00qpid-dump\x00; type=fanout; alternate-exchange=; auto-delete=1; arguments={}; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 0 incomplete: { } unknown-completed: { [0,0] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channemessage_queue\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug Configured queue with no-local=0
2009-jan-06 11:35:06 debug Configured queue message_queue with qpid.trace.id='' and qpid.trace.exclude='' i.e. 0 elements
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded queue message_queue
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=message_queue; exchange=\x00qpid-dump\x00; binding-key=; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 1: {ExchangeBindBody: queue=message_queue; exchange=\x00qpid-dump\x00; binding-key=; arguments={}; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 1 incomplete: { } unknown-completed: { [0,1] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=message_queue; exchange=; binding-key=message_queue; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 2: {ExchangeBindBody: queue=message_queue; exchange=; binding-key=message_queue; arguments={}; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 2 incomplete: { } unknown-completed: { [0,2] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=message_queue; exchange=amq.direct; binding-key=routing_key; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 3: {ExchangeBindBody: queue=message_queue; exchange=amq.direct; binding-key=routing_key; arguments={}; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 3 incomplete: { } unknown-completed: { [0,3] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeUnbindBody: queue=message_queue; exchange=\x00qpid-dump\x00; binding-key=; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 4: {ExchangeUnbindBody: queue=message_queue; exchange=\x00qpid-dump\x00; binding-key=; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 4 incomplete: { } unknown-completed: { [0,4] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionQueueBody: encoded= mgmt-localhost.localdomain.17482\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug Configured queue with no-local=0
2009-jan-06 11:35:06 debug Configured queue mgmt-localhost.localdomain.17482 with qpid.trace.id='' and qpid.trace.exclude='' i.e. 0 elements
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded queue mgmt-localhost.localdomain.17482
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 5: {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; arguments={}; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 5 incomplete: { } unknown-completed: { [0,5] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=; binding-key=mgmt-localhost.localdomain.17482; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 6: {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=; binding-key=mgmt-localhost.localdomain.17482; arguments={}; }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 6 incomplete: { } unknown-completed: { [0,6] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=qpid.management; binding-key=console.#; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 7: {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=qpid.management; binding-key=console.#; arguments={}; }
2009-jan-06 11:35:06 debug Message 0x859bc28 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 7 incomplete: { } unknown-completed: { [0,7] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=qpid.management; binding-key=schema.#; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 8: {ExchangeBindBody: queue=mgmt-localhost.localdomain.17482; exchange=qpid.management; binding-key=schema.#; arguments={}; }
2009-jan-06 11:35:06 debug Message 0x859c2e0 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 8 incomplete: { } unknown-completed: { [0,8] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeUnbindBody: queue=mgmt-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 9: {ExchangeUnbindBody: queue=mgmt-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; }
2009-jan-06 11:35:06 debug Message 0x859c700 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 9 incomplete: { } unknown-completed: { [0,9] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ClusterConnectionQueueBody: encoded= repl-localhost.localdomain.17482\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00; }]
2009-jan-06 11:35:06 debug Configured queue with no-local=0
2009-jan-06 11:35:06 debug Configured queue repl-localhost.localdomain.17482 with qpid.trace.id='' and qpid.trace.exclude='' i.e. 0 elements
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) decoded queue repl-localhost.localdomain.17482
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=repl-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 10: {ExchangeBindBody: queue=repl-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; arguments={}; }
2009-jan-06 11:35:06 debug Message 0x859d0f0 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 10 incomplete: { } unknown-completed: { [0,10] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=repl-localhost.localdomain.17482; exchange=; binding-key=repl-localhost.localdomain.17482; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 11: {ExchangeBindBody: queue=repl-localhost.localdomain.17482; exchange=; binding-key=repl-localhost.localdomain.17482; arguments={}; }
2009-jan-06 11:35:06 debug Message 0x859d700 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 11 incomplete: { } unknown-completed: { [0,11] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeBindBody: queue=repl-localhost.localdomain.17482; exchange=amq.direct; binding-key=repl-localhost.localdomain.17482; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 12: {ExchangeBindBody: queue=repl-localhost.localdomain.17482; exchange=amq.direct; binding-key=repl-localhost.localdomain.17482; arguments={}; }
2009-jan-06 11:35:06 debug Message 0x859dde0 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 12 incomplete: { } unknown-completed: { [0,12] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExchangeUnbindBody: queue=repl-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 13: {ExchangeUnbindBody: queue=repl-localhost.localdomain.17482; exchange=\x00qpid-dump\x00; binding-key=; }
2009-jan-06 11:35:06 debug Message 0x859e1d8 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 13 incomplete: { } unknown-completed: { [0,13] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {QueueDeclareBody: queue=\x00qpid-dump\x00; alternate-exchange=; auto-delete=1; arguments={}; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 14: {QueueDeclareBody: queue=\x00qpid-dump\x00; alternate-exchange=; auto-delete=1; arguments={}; }
2009-jan-06 11:35:06 debug Configured queue with no-local=0
2009-jan-06 11:35:06 debug Configured queue \x00qpid-dump\x00 with qpid.trace.id='' and qpid.trace.exclude='' i.e. 0 elements
2009-jan-06 11:35:06 debug Message 0x859eb98 enqueued on mgmt-localhost.localdomain.17482[0x859b190]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 14 incomplete: { } unknown-completed: { [0,14] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {ExecutionSyncBody: }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: recv cmd 15: {ExecutionSyncBody: }
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: receiver marked completed: 15 incomplete: { } unknown-completed: { [0,15] }
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=1; {SessionCompletedBody: commands={ [0,15] }; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace RECVB [10.16.18.254:59986]: 29 bytes
2009-jan-06 11:35:06 trace 10.16.18.254:17495(DUMPEE) RECV 10.16.18.254:17495-0x8558ed8(local,catchup): Frame[BEbe; channel=1; {SessionDetachBody: name=dump_shared; }]
2009-jan-06 11:35:06 debug dump_shared@guest@QPID: detached on broker.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59986]: Frame[BEbe; channel=1; {SessionDetachedBody: name=dump_shared; code=0; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x8558ed8(local,catchup) write idle.
2009-jan-06 11:35:06 debug RECV [10.16.18.254:59987] INIT(128-138)
2009-jan-06 11:35:06 warning SASL: No Authentication Performed
2009-jan-06 11:35:06 debug 10.16.18.254:17495(DUMPEE) new connection: 10.16.18.254:17495-0x85df790(local,catchup)
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x85df790(local,catchup) write idle.
2009-jan-06 11:35:06 trace SENT 10.16.18.254:59987 INIT(0-10)
2009-jan-06 11:35:06 trace SENT [10.16.18.254:59987]: Frame[BEbe; channel=0; {ConnectionStartBody: server-properties={qpid.federation_tag:V2:36:str16(a139cb60-9716-437b-8060-5b619e835cf9)}; mechanisms=str16{V2:9:str16(ANONYMOUS)}; locales=str16{V2:5:str16(en_US)}; }]
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x85df790(local,catchup) write idle.
2009-jan-06 11:35:06 trace 10.16.18.254:17495-0x85df790(local,catchup) write idle.
2009-jan-06 11:35:06 debug DISCONNECTED [10.16.18.254:59987]
2009-jan-06 11:35:06 critical 10.16.18.254:17495(DUMPEE) catch-up connection closed prematurely 10.16.18.254:17495-0x85df790(local,catchup)
2009-jan-06 11:35:06 notice 10.16.18.254:17495(LEFT) leaving cluster TED
2009-jan-06 11:35:06 notice 10.16.18.254:17495(LEFT) shutting down 
2009-jan-06 11:35:06 debug Shutting down CPG
2009-jan-06 11:35:06 notice Shut down
2009-jan-06 11:35:06 debug Journal "TplStore": Destroyed
Comment 2 Alan Conway 2009-02-17 14:18:45 UTC 
Trace from the other broker during the failed restart would be useful, may give some clue as to why it is closing the catch-up connection.
Comment 4 Alan Conway 2010-02-01 16:00:43 UTC 
The problem longer appears in r904656, fixed by some previous change.
Comment 5 Frantisek Reznicek 2010-05-13 13:58:51 UTC 
Created attachment 413767 [details]
The reproducer for bz479031

./run.sh 5 to execute
Comment 6 Frantisek Reznicek 2010-05-13 14:00:40 UTC 
Above attached reproducer proves that issue has been fixed, tested on RHEL 5.5 i386 / x86_64 on packages:
[root@mrg-qe-01 tests]# rpm -qa | grep qpid | sort
python-qpid-0.7.934605-1.el5
qpid-cpp-client-0.7.935473-1.el5
qpid-cpp-client-devel-0.7.935473-1.el5
qpid-cpp-client-devel-docs-0.7.935473-1.el5
qpid-cpp-client-ssl-0.7.935473-1.el5
qpid-cpp-mrg-debuginfo-0.7.935473-1.el5
qpid-cpp-server-0.7.935473-1.el5
qpid-cpp-server-cluster-0.7.935473-1.el5
qpid-cpp-server-devel-0.7.935473-1.el5
qpid-cpp-server-ssl-0.7.935473-1.el5
qpid-cpp-server-store-0.7.935473-1.el5
qpid-cpp-server-xml-0.7.935473-1.el5
qpid-java-client-0.7.934605-1.el5
qpid-java-common-0.7.934605-1.el5
qpid-tests-0.7.930108-1.el5
qpid-tools-0.7.934605-2.el5
ruby-qpid-0.7.935473-1.el5


-> VERIFIED
Comment 7 Jaromir Hradilek 2010-10-07 13:56:40 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Open management session no longer prevents a broker from being added to a cluster.
Comment 8 Douglas Silas 2010-10-11 12:02:18 UTC 
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Open management session no longer prevents a broker from being added to a cluster.+Having an open management session no longer prevents a broker from being added to a cluster.
Comment 10 errata-xmlrpc 2010-10-14 15:58:36 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html