Description of problem: The new feature was introduced in qpid 0.14 - "Support for timestamping messages on arrival at the broker." A broker can be configured to add a timestamp to the message it receives from a client. The problem is that when a broker which is part of a cluster, receives a message from another broker, and this message has a timestamp in it already, this timestamp gets rewritten. This leads to an inconsistency in cluster. For example, I have a cluster consisting of just two brokers. They are both running: [root@lzhaldyb-rhel62i ~]# qpid-cluster Cluster Name: lzhaldyb-cluster Cluster Status: ACTIVE Cluster Size: 2 Members: ID=192.168.62.1:1984 URL=amqp:tcp:10.34.33.227:5672,tcp:192.168.62.1:5672 : ID=192.168.62.2:3691 URL=amqp:tcp:10.34.33.226:5672,tcp:192.168.62.2:5672 Step 1: send two messages to one of the brokers. [root@lzhaldyb-rhel62i ~]# ./spout -c 1 -i message_$(date +%s) test_cluster_timestamps && sleep 8 && ./spout -c 1 -i message_$(date +%s) test_cluster_timestamps Both brokers have both messages with identical timestamps after this step: [root@lzhaldyb-rhel62i ~]# ./drain -c 0 'test_cluster_timestamps; {mode: browse}' Message(properties={spout-id:message_1330528317:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528317}, content='') Message(properties={spout-id:message_1330528325:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528325}, content='') [root@lzhaldyb-rhel62x ~]# ./drain -c 0 'test_cluster_timestamps; {mode: browse}' Message(properties={spout-id:message_1330528317:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528317}, content='') Message(properties={spout-id:message_1330528325:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528325}, content='') Step 2: turn off one of the brokers and send two more messages to the active one. [root@lzhaldyb-rhel62x ~]# service qpidd stop Stopping Qpid AMQP daemon: [ OK ] [root@lzhaldyb-rhel62i ~]# qpid-cluster Cluster Name: lzhaldyb-cluster Cluster Status: ACTIVE Cluster Size: 1 Members: ID=192.168.62.1:2296 URL=amqp:tcp:10.34.33.227:5672,tcp:192.168.62.1:5672 [root@lzhaldyb-rhel62i ~]# ./drain -c 0 'test_cluster_timestamps; {mode: browse}' Message(properties={spout-id:message_1330528317:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528317}, content='') Message(properties={spout-id:message_1330528325:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528325}, content='') Message(properties={spout-id:message_1330528499:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528499}, content='') Message(properties={spout-id:message_1330528507:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528507}, content='') Step 3: turn the second broker back on. On this step, the first broker sends the messages to the second one, which sets the timestamps to the current value: [root@lzhaldyb-rhel62x ~]# service qpidd start Starting Qpid AMQP daemon: [ OK ] [root@lzhaldyb-rhel62x ~]# ./drain -c 0 'test_cluster_timestamps; {mode: browse}' Message(properties={spout-id:message_1330528317:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330532153}, content='') Message(properties={spout-id:message_1330528325:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330532153}, content='') Message(properties={spout-id:message_1330528499:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330532153}, content='') Message(properties={spout-id:message_1330528507:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330532153}, content='') whereas the timestamps on the first broker remain the same: [root@lzhaldyb-rhel62i ~]# ./drain -c 0 'test_cluster_timestamps; {mode: browse}' Message(properties={spout-id:message_1330528317:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528317}, content='') Message(properties={spout-id:message_1330528325:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528325}, content='') Message(properties={spout-id:message_1330528499:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528499}, content='') Message(properties={spout-id:message_1330528507:0, x-amqp-0-10.routing-key:test_cluster_timestamps, x-amqp-0-10.timestamp:1330528507}, content='') As a result we have a cluster with two brokers, each of which has the exact same messages, but timestamps of these messages differ among brokers. Version-Release number of selected component (if applicable): qpid-cpp-client-0.14-7.el6.i686 qpid-cpp-client-devel-0.14-7.el6.i686 qpid-cpp-client-ssl-0.14-7.el6.i686 qpid-cpp-server-0.14-7.el6.i686 qpid-cpp-server-cluster-0.14-7.el6.i686 qpid-cpp-server-devel-0.14-7.el6.i686 qpid-cpp-server-ssl-0.14-7.el6.i686 qpid-cpp-server-store-0.14-7.el6.i686 qpid-cpp-server-xml-0.14-7.el6.i686 How reproducible: always Steps to Reproduce: 1. Configure cluster, turn one of the clustered brokers off. 2. Send a few messages to the currently active broker. 3. Turn the broker from Step 1 back on. 4. Compare timestamps of the messages on the newly started broker and the one they were initially sent to by the client. Actual results: The timestamps of the same message differ on different brokers. Expected results: Timestamps are consistent throughout the broker. Additional info: