Bug 502495

Summary: qpid-stat does not provice precise information about running brokers if more than one cluster is set up
Product: Red Hat Enterprise MRG Reporter: Frantisek Reznicek <freznice>
Component: python-qpidAssignee: Ted Ross <tross>
Status: CLOSED NOTABUG QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.1.1CC: esammons, tross
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-08-25 21:19:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Two clusters set-up reproducer none

Description Frantisek Reznicek 2009-05-25 14:07:37 UTC 
Created attachment 345317 [details]
Two clusters set-up reproducer

Description of problem:

qpid-stat does not provice precise information about running brokers if more than one cluster is set up.
In this particular situation qpid-stat shows only nodes which belongs to first set-up qpidd cluster.
See details in the additional info section below...(seen on RHEL 5.3 x86_64)

Version-Release number of selected component (if applicable):
openais-0.80.3-22.el5_3.6
openais-debuginfo-0.80.3-22.el5_3.6
openais-devel-0.80.3-22.el5_3.6
python-qpid-0.5.752581-1.el5
qpidc-0.5.752581-5.el5
qpidc-debuginfo-0.5.752581-5.el5
qpidc-devel-0.5.752581-5.el5
qpidc-rdma-0.5.752581-5.el5
qpidc-ssl-0.5.752581-5.el5
qpidd-0.5.752581-5.el5
qpidd-acl-0.5.752581-5.el5
qpidd-cluster-0.5.752581-5.el5
qpidd-devel-0.5.752581-5.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-5.el5
qpidd-ssl-0.5.752581-5.el5
qpidd-xml-0.5.752581-5.el5
qpid-java-client-0.5.751061-2.el5
qpid-java-common-0.5.751061-2.el5


How reproducible:
100%

Steps to Reproduce:
1. run attached script ./run.sh and watch until you see this:
  [root@dhcp-lab-200 bz502193]# ./run.sh
  Stopping OpenAIS daemon (aisexec):                         [  OK  ]
  Starting OpenAIS daemon (aisexec):                         [  OK  ]
  Client[s] compile
  .ready
  starting brokers in the cluster:...done
  tcp        0      0 0.0.0.0:5672   0.0.0.0:*        LISTEN      28658/qpidd
  tcp        0      0 0.0.0.0:10001  0.0.0.0:*        LISTEN      28676/qpidd
  tcp        0      0 0.0.0.0:10002  0.0.0.0:*        LISTEN      28694/qpidd
  broker[s] running
  starting brokers in the cluster:..done
  tcp        0      0 0.0.0.0:5672   0.0.0.0:*        LISTEN      28658/qpidd
  tcp        0      0 0.0.0.0:10001  0.0.0.0:*        LISTEN      28676/qpidd
  tcp        0      0 0.0.0.0:10002  0.0.0.0:*        LISTEN      28694/qpidd
  tcp        0      0 0.0.0.0:10003  0.0.0.0:*        LISTEN      28728/qpidd
  tcp        0      0 0.0.0.0:10004  0.0.0.0:*        LISTEN      28744/qpidd
  broker[s] running

2. qpid-stat -b does not correspond following lists in point 3.
3. netstat -nlp | grep qpidd ; ps auxw | grep qpidd | grep -v grep
  
Actual results:
Only first cluster members are visible vis 'qpid-stat -b'

Expected results:
All cluster nodes of all clusters should be visible

Additional info (see transcript):

[root@dhcp-lab-200 ~]# qpid-stat -b
Brokers
  broker              cluster                                                              uptime  conn  sess  exch  queue
  ==========================================================================================================================
  10.34.33.200:5672   dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary(ACTIVE)  50s        1     1     8     3
  10.34.33.200:10001  dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary(ACTIVE)  50s        1     1     8     5
  10.34.33.200:10002  dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary(ACTIVE)  50s        1     1     8     7
[root@dhcp-lab-200 ~]# netstat -nlp | grep qpidd
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      28658/qpidd
tcp        0      0 0.0.0.0:10001               0.0.0.0:*                   LISTEN      28676/qpidd
tcp        0      0 0.0.0.0:10002               0.0.0.0:*                   LISTEN      28694/qpidd
tcp        0      0 0.0.0.0:10003               0.0.0.0:*                   LISTEN      28728/qpidd
tcp        0      0 0.0.0.0:10004               0.0.0.0:*                   LISTEN      28744/qpidd
[root@dhcp-lab-200 ~]# gps qpidd
root     28658  0.0  0.3 344776 13664 pts/6    Sl+  16:03   0:00 qpidd -p 5672 --auth no --log-enable debug+ --cluster-name dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary --replication-queue REPLICATION_QUEUE --create-replication-queue --data-dir data_0
root     28676  0.0  0.3 342920 13520 pts/6    Sl+  16:03   0:00 qpidd -p 10001 --auth no --log-enable debug+ --cluster-name dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary --replication-queue REPLICATION_QUEUE --create-replication-queue --data-dir data_1
root     28694  0.0  0.3 303620 12684 pts/6    Sl+  16:03   0:00 qpidd -p 10002 --auth no --log-enable debug+ --cluster-name dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary --replication-queue REPLICATION_QUEUE --create-replication-queue --data-dir data_2
root     28728  0.0  0.3 344772 12584 pts/6    Sl+  16:03   0:00 qpidd -p 10003 --auth no --log-enable debug+ --cluster-name dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_backup --data-dir data_3
root     28744  0.0  0.3 303616 11908 pts/6    Sl+  16:03   0:00 qpidd -p 10004 --auth no --log-enable debug+ --cluster-name dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_backup --data-dir data_4
root     28886  0.0  0.0  61124   752 pts/4    S+   16:04   0:00 grep qpidd
[root@dhcp-lab-200 ~]# qpid-stat -b
Brokers
  broker              cluster                                                              uptime  conn  sess  exch  queue
  ==========================================================================================================================
  10.34.33.200:5672   dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary(ACTIVE)  1m 30s     1     1     8     3
  10.34.33.200:10001  dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary(ACTIVE)  1m 30s     1     1     8     5
  10.34.33.200:10002  dhcp-lab-200.englab.brq.redhat.com_cluster_bz502193_primary(ACTIVE)  1m 20s     1     1     8     7
[root@dhcp-lab-200 ~]# qpid-config -a localhost:10003 queues
Queue Name                                        Attributes
======================================================================
reply-dhcp-lab-200.englab.brq.redhat.com.28909.1  auto-del excl
topic-dhcp-lab-200.englab.brq.redhat.com.28909.1  auto-del excl
[root@dhcp-lab-200 ~]# qpid-config -a localhost:10004 queues
Queue Name                                        Attributes
======================================================================
reply-dhcp-lab-200.englab.brq.redhat.com.28918.1  auto-del excl
topic-dhcp-lab-200.englab.brq.redhat.com.28918.1  auto-del excl
Comment 1 Ted Ross 2009-08-25 21:19:38 UTC 
qpid-stat makes a connection to a broker (localhost:5672 by default) and displays statistics from that broker.  If the broker is a node in a cluster, qpid-stat contacts the other nodes in the broker so it can provide stats across the entire cluster.  It will never show information for multiple clusters.

I was unable to get the reproducer to run, so I've not actually seen your results.

Since a broker instance (defined by an address and a port) can be in at most one cluster, qpid-stat will provide information from at most one cluster.