Description of problem:

Lets have following broker clustering configuration:
- two broker nodes ran on one machine (ports: 5672 and 5673) with the same max-session-rate=N parameter to limit the message throughput
- run qpid-perftest on the first one

The client (qpid-perftest) execution finishes ok.

The broker on port 5673 (which replicates from the head one on port 5672) produces a lots of warnings like:
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation

This behavior might be caused by missing cluster node data replication flow control as discussed in bug 623660. 

The amount of warning message[s] should be reduced the bug 629756 way.

This is observed on RHEL5.6 i386 / x86_64.


Version-Release number of selected component (if applicable):
[root@mrg-qe-09 _s]# rpm -qa |grep qpid | sort
python-qpid-0.7.946106-15.el5
qpid-cpp-client-0.7.946106-28.el5
qpid-cpp-client-devel-0.7.946106-28.el5
qpid-cpp-client-devel-docs-0.7.946106-28.el5
qpid-cpp-client-rdma-0.7.946106-28.el5
qpid-cpp-client-ssl-0.7.946106-28.el5
qpid-cpp-mrg-debuginfo-0.7.946106-28.el5
qpid-cpp-server-0.7.946106-28.el5
qpid-cpp-server-cluster-0.7.946106-28.el5
qpid-cpp-server-devel-0.7.946106-28.el5
qpid-cpp-server-rdma-0.7.946106-28.el5
qpid-cpp-server-ssl-0.7.946106-28.el5
qpid-cpp-server-store-0.7.946106-28.el5
qpid-cpp-server-xml-0.7.946106-28.el5
qpid-dotnet-0.4.738274-2.el5
qpid-java-client-0.7.946106-15.el5
qpid-java-common-0.7.946106-15.el5
qpid-java-example-0.7.946106-15.el5
qpid-tools-0.7.946106-12.el5
rh-qpid-cpp-tests-0.7.946106-28.el5

How reproducible:
100%

Steps to Reproduce:
See Additional info section for details.
  
Actual results:
The tons of throttling warnings in the broker log.

Expected results:
Reduced number of throttling warnings in the broker log.

Additional info (transcript):

[root@mrg-qe-09 cluster_test_bz674338]# service openais restart
Stopping OpenAIS daemon (aisexec): aisexec
Starting OpenAIS daemon (aisexec):  aisexec
[root@mrg-qe-09 cluster_test_bz674338]# qpidd -d --log-to-file=/tmp/qpidd.5672.log --cluster-name=XXX --log-enable=info+ --data-dir=/tmp/dd5672 --max-session-rate=5000
2011-02-08 10:58:15 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2011-02-08 10:58:15 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/watchdog.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2011-02-08 10:58:16 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
[root@mrg-qe-09 cluster_test_bz674338]# qpidd -d --log-to-file=/tmp/qpidd.5673.log --port 5673 --cluster-name=XXX --log-enable=info+ --data-dir=/tmp/dd5673 --max-session-rate=5000
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/watchdog.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2011-02-08 10:58:31 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
[root@mrg-qe-09 cluster_test_bz674338]# qpid-cluster
  Cluster Name: XXX
Cluster Status: ACTIVE
  Cluster Size: 2
       Members: ID=10.34.45.9:25539 URL=amqp:tcp:10.34.33.62:5672,tcp:10.34.44.9:5672,tcp:10.34.45.9:5672
              : ID=10.34.45.9:25552 URL=amqp:tcp:10.34.33.62:5673,tcp:10.34.44.9:5673,tcp:10.34.45.9:5673
[root@mrg-qe-09 cluster_test_bz674338]# qpid-perftest -s --count 150000
4237.01 4232.08 8464.37 8.26599
[root@mrg-qe-09 cluster_test_bz674338]# wc -l /tmp/qpidd.5672.log
65 /tmp/qpidd.5672.log
[root@mrg-qe-09 cluster_test_bz674338]# wc -l /tmp/qpidd.5673.log
47268 /tmp/qpidd.5673.log
[root@mrg-qe-09 cluster_test_bz674338]# grep -c "producer throttling violation" /tmp/qpidd.5672.log
0
[root@mrg-qe-09 cluster_test_bz674338]# grep -c "producer throttling violation" /tmp/qpidd.5673.log
47210
[root@mrg-qe-09 cluster_test_bz674338]# grep "producer throttling violation" /tmp/qpidd.5673.log | tail
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation
2011-02-08 10:59:21 warning anonymous.52b2fbb9-8b6a-4cf3-afc3-2c6f56f3125d: producer throttling violation