From the log messages posted above it appears the broker shut down due to problems communicating with CPG, not from management traffic. Was there anything in the openais or corosync logs?

There have been several bugs recently fixed for issues with management and clustering. They all involve a broker exiting with a critical message containing "invalid argument". Was this seen in any of the broker logs? All the fixes will be in the coming MRG 2.0 release.

From Steve Dake:

[Does corosync work on VMs?]

Yes, but we have some speculation that there may be problems with single
cpu VMs and deadlock (deadlock triggered by running corosync in realtime
priority.  Another possibility could be an OOM situation.

Ask them to rerun tests with cpu=2 for the vms.

Ask them to rerun tests with current cpu settings but with -p command
line option to corosync:

corosync: 
usage:
        -f     : Start application in foreground.
        -p     : Do not set process priority.
        -v     : Display version and SVN revision of Corosync and exit.

Alan, 
   We have version openais-0.80.6-28.el5 for openais.  I will look into how the VM's where created and get back to you.  As it stands now even when I put the mgmt-pub-interval=15 in the configuration the brokers send management traffic every second.  

I have the logs etc but they are really big. I will try to post them in a minute.

Are there any active clients connected to the broker? Are you running sesame or some other management tool that might be making requests every second? In your test scenario, what clients (including management tools or agents) are running?

Can you give some step-by-step details of how I can reproduce this: the VM setup, the qpidd config, openais config, any running services (e.g. sesame) that interact with qpidd and command lines for any clients you run during the test.

Alan,
Q: Are there any active clients connected to the broker?
    
Answer: We initially run qpid-tool and qpid-cluster to make sure the cluster is up and running. After that we leave the brokers alone.  

Q: Are you running sesame or some other management tool that might be making requests every second?

Answer: We are not running any management tools.

Q: In your test scenario, what clients (including management tools or agents) are running?

Answer: No clients are connected to these brokers.

The qpidd.conf file looks like this:

# "qpidd --help" or "man qpidd" for more details.
cluster-mechanism=ANONYMOUS
port=5681
auth=no
cluster-name="poccluster"
cluster-size=2

log-to-file=/tmp/qpid-clustering/bora/qpidd.log
data-dir=/tmp/qpid-clustering/bora/

trace=yes
debug=yes
log-enable=info+
log-enable=debug+:cluster
log-enable=trace+:management
mgmt-pub-interval=15


The openais.conf:

# Please read the openais.conf.5 manual page

totem {
	version: 2
	secauth: off
	threads: 0
	interface {
		ringnumber: 0
		bindnetaddr: 192.168.53.0
		mcastaddr: 226.94.1.1
		mcastport: 5405
	}
}

# This directive controls how OpenAIS logs it's messages. All variables here
# are optional.
logging {
        # These three options control where log messages are sent. Logs can be
        # sent to two or all three. The three options are: 'to_file',
        # 'to_syslog' and 'to_stderr'. All three can either be 'yes' or 'no'.
        # When set to 'yes', logs are sent to the relative destination. The
        # default is to write to the syslog and to stderr.
        #
        # In version 0.70, there were the following options:
        # logoutput: syslog
        # logoutput: stderr
        # logoutput: file
        # NOTE: These are deprecated and should not be used.

        # Default is 'no'. If set, 'logfile' must be set below.
        to_file: yes

        # Default is 'yes'.
        to_syslog: yes

        # Default is 'yes'.
        to_stderr: yes

        # When 'logoutput: file' is set, this is required. It is the full
        # path and file name to write the logs to.
        logfile: /var/log/bova-ais.log

        # Setting this to 'on', the default, generates a lot of debug messages
        # in the log. It is generally not advised unless you are tracing a
        # specific bug.
        debug: on

        # Setting this to 'on', the default, will add a time stamp to log
        # messages.
        timestamp: on

        # Setting this to 'on' will replace the logger name in the log entries
        # with the file and line generating the log entry. The default is
        # 'off'.
        fileline: off

        # When writing to syslog, this sets the syslog facility to use. Valid
        # options are:
        # daemon, local0, local1, local2, local3, local4, local5, local6 and
        # local7
        # The default is 'daemon'.
        syslog_facility: daemon

        # This is an optional directive that controls detailed logging
        # features. Generally, this is only needed by developers.
        #logger {
                # This specifies the identity logging being specified.
                # MADI: What?
                #ident: ?

                # This enables or disables debug log messages for the component
                # identified above. The default is 'off'.
                #debug: off

                # This specifies which tags should be logged for this
                # component. This is only valid when debug is enabled above.
                # Multiple tags are specified with a pipe (|) as the logical
                # OR seperator. The default is 'none'.
                #tags: enter|return|trace1|trace2
        #}
}


amf {
	mode: disabled
}


I will update the BZ with the VM setup.

Lester

The failure in their logs was due to a network partition or "split brain". Am working with the customer to set up a qpid cluster using CMAN to manage a quorum to avoid this condition.