Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 667970

Summary: RFE: provide incoming and outgoing message count statistics for connection
Product: Red Hat Enterprise MRG Reporter: Gordon Sim <gsim>
Component: qpid-cppAssignee: Gordon Sim <gsim>
Status: CLOSED ERRATA QA Contact: Frantisek Reznicek <freznice>
Severity: medium Docs Contact:
Priority: low    
Version: 1.3CC: esammons, freznice, iboverma, jneedle, mhusnain, tross
Target Milestone: 2.0Keywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qpid-cpp-mrg-0.9.1073306-1 Doc Type: Enhancement
Doc Text:
Cause: There was no statistic defined for a connection tracking the number of messages sent in either direction (only frames and bytes) Consequence: Could not accurately determine how many messages had been sent over a connection. Change: Added such statistics Result: Can now monitor the message flow over connections. Release Note Entry: A new feature is added to track statistics about the number of messages transferred over the connection instead of tracking only the numbers of frames and bytes transferred across the connection.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2011-06-23 15:49:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The issue unit test reproducer
none
Suggested fixes to attached test none

Description Gordon Sim 2011-01-07 14:18:19 UTC 
Description of problem:

The connection object visible via management (e.g. with qpid-tool) has statistics for frames and bytes sent over the connection in each direction (note that the stats of frames/bytes *to* the client are currently not correctly populated and are always at zero). It would be beneficial to also record the number of messages, as that is often more meaningful.

Steps to test:

1. start broker
2. start receiver (e.g. qpid-receive --address amq.fanout)
3. start sender (e.g. qpid-send --address amq.fanout --content-stdin)
4. send some messages from sender such that receiver gets them (e.g. type in some text on std in)
5. using qpid-tool verify the message counts
Comment 1 Gordon Sim 2011-01-12 12:55:47 UTC 
Fixed upstream: http://svn.apache.org/viewcvs?view=rev&rev=1057578
Comment 3 Gordon Sim 2011-03-07 12:38:27 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
There was no statistic defined for a connection tracking the number of messages sent in either direction (only frames and bytes)
Consequence:
Could not accurately determine how many messages had been sent over a connection.
Change:
Added such statistics
Result:
Can now monitor the message flow over connections.
Comment 4 Frantisek Reznicek 2011-03-14 16:02:21 UTC 
Although the created unit test qpid_tests.broker_0_10.management.ManagementTest.test_connection_stats is passing I found a problem during development of custom unit test.


The problem is following.

When a client is transmitting the messages to the broker's queue, stats *FromClient are correct. The reception part does not work as expected.

Let's focus on reception only and especially on msgsToClient data.

Before the client creates first subscription to the queue, msgsToClient's data are correctly 0, after first subscription the data jump to value which is stored in msgsFromClient.
Similar behavior defect is seen in framesToClient and bytesToClient stats.


-> ASSIGNED
Comment 5 Frantisek Reznicek 2011-03-14 16:09:09 UTC 
Created attachment 484235 [details]
The issue unit test reproducer

The attached issue is triggered by python unit test reproducer.

Run following way:
NR_OF_LOOPS=5 PYTHONPATH=${PYTHONPATH}:. qpid-python-test -m custom_py_units  custom_py_units.connection.LLAPITests.test_connection_stats


Following transcript show the behavior:

[root@mrg-qe-07 qpid_ptest_unit_tests]# service qpidd restart
Stopping Qpid AMQP daemon:                                 [  OK  ]
Starting Qpid AMQP daemon:                                 [  OK  ]
2011-03-14 11:54:18 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2011-03-14 11:54:18 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2011-03-14 11:54:18 info Loaded Module: /usr/lib/qpid/daemon/acl.so
2011-03-14 11:54:18 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
2011-03-14 11:54:18 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so
2011-03-14 11:54:18 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so
[root@mrg-qe-07 qpid_ptest_unit_tests]# NR_OF_LOOPS=5 PYTHONPATH=${PYTHONPATH}:. qpid-python-test -m custom_py_units  custom_py_units.connection.LLAPITests.test_connection_stats
custom_py_units.connection.LLAPITests.test_connection_stats .............................................................................. start
  setup: in_loops:5, msg_cnt:42, BATCH_CNT:5, msg_batches:[11, 21, 32, 42]
  rx(0):133/48|4968/1303|42/0
  0 1 2 3 4 5 6 7 8 9 10 rx(11):140/181|5175/6221|42/42
custom_py_units.connection.LLAPITests.test_connection_stats .............................................................................. fail
Error during test:
  Traceback (most recent call last):
    File "/usr/bin/qpid-python-test", line 311, in run
      phase()
    File "/root/MRG/Messaging/qpid_ptest_unit_tests/custom_py_units/connection.py", line 181, in test_connection_stats
      self.assertEqual(loc_conns_qmf[indx].msgsToClient, msg_processed);
    File "/usr/lib/python2.3/unittest.py", line 302, in failUnlessEqual
      raise self.failureException, \
  AssertionError: 42L != 11
Totals: 1 tests, 0 passed, 0 skipped, 0 ignored, 1 failed

Comment out the connection.py:181 to get view how reception in batches influence msgsToClient...

[root@mrg-qe-07 qpid_ptest_unit_tests]# NR_OF_LOOPS=5 PYTHONPATH=${PYTHONPATH}:. qpid-python-test -m custom_py_units  custom_py_units.connection.LLAPITests.test_connection_stats
custom_py_units.connection.LLAPITests.test_connection_stats .............................................................................. start
  setup: in_loops:5, msg_cnt:43, BATCH_CNT:5, msg_batches:[11, 22, 32, 43]
  rx(0):136/49|5075/1329|43/0
  0 1 2 3 4 5 6 7 8 9 10 rx(11):143/185|5282/6360|43/43
  11 12 13 14 15 16 17 18 19 20 21 rx(22):143/185|5282/6360|43/43
  22 23 24 25 26 27 28 29 30 31 rx(32):143/185|5282/6360|43/43
  32 33 34 35 36 37 38 39 40 41 42 rx(43):143/185|5282/6360|43/43
custom_py_units.connection.LLAPITests.test_connection_stats .............................................................................. fail

The behavior is observed the same on RHEL4/6 (5 not tested yet)
Comment 6 Frantisek Reznicek 2011-03-14 16:13:34 UTC 
(In reply to comment #5)

> [root@mrg-qe-07 qpid_ptest_unit_tests]# NR_OF_LOOPS=5
> PYTHONPATH=${PYTHONPATH}:. qpid-python-test -m custom_py_units 
> custom_py_units.connection.LLAPITests.test_connection_stats
> custom_py_units.connection.LLAPITests.test_connection_stats
> ..............................................................................
> start
>   setup: in_loops:5, msg_cnt:43, BATCH_CNT:5, msg_batches:[11, 22, 32, 43]
>   rx(0):136/49|5075/1329|43/0
>   0 1 2 3 4 5 6 7 8 9 10 rx(11):143/185|5282/6360|43/43
>   11 12 13 14 15 16 17 18 19 20 21 rx(22):143/185|5282/6360|43/43
>   22 23 24 25 26 27 28 29 30 31 rx(32):143/185|5282/6360|43/43
>   32 33 34 35 36 37 38 39 40 41 42 rx(43):143/185|5282/6360|43/43

Expected is to see 
   rx(0):136/49|5075/1329|43/0
   0 1 2 3 4 5 6 7 8 9 10 rx(11): ... |43/11
   11 12 13 14 15 16 17 18 19 20 21 rx(22): ... |43/22
   22 23 24 25 26 27 28 29 30 31 rx(32): ... |43/32
   32 33 34 35 36 37 38 39 40 41 42 rx(43): ... |43/43
Comment 7 Gordon Sim 2011-03-14 17:55:08 UTC 
Created attachment 484266 [details]
Suggested fixes to attached test

The test issues infinite message credit which allows the broker to send all the messages as soon as the subscription is issued, independently of the batch in which the test then processes the message.

This patch shows the changes I made to use credit to control the flow of messages in line with the desired batching (as far as I understood it). There was also one typo where 'I' was used in place of indx, and some confusion around stop()/start() on local queue v. equivalent protocol commands.

In short I believe the test is in error and the feature works as expected in this case.
Comment 8 Gordon Sim 2011-03-14 17:59:08 UTC 
Setting back to MODIFIED.
Comment 9 Frantisek Reznicek 2011-03-15 10:13:32 UTC 
The credit effect on delivery understood.

Test fixed and feature retested using following units:
qpid_tests.broker_0_10.management.ManagementTest.test_connection_stats
custom_py_units.connection.LLAPITests.test_connection_stats

on RHEL 4.9 / 5.6 / 6.1 on packages:
python-qmf-debuginfo-0.9.1079953-3.el5.x86_64
python-qpid-0.9.1079482-1.el5.noarch
python-qpid-qmf-0.9.1079953-3.el5.x86_64
qpid-cpp-client-0.9.1079953-1.el5.x86_64
qpid-cpp-client-devel-0.9.1079953-1.el5.x86_64
qpid-cpp-client-devel-docs-0.9.1079953-1.el5.x86_64
qpid-cpp-client-rdma-0.9.1079953-1.el5.x86_64
qpid-cpp-client-ssl-0.9.1079953-1.el5.x86_64
qpid-cpp-mrg-debuginfo-0.9.1079953-1.el5.x86_64
qpid-cpp-server-0.9.1079953-1.el5.x86_64
qpid-cpp-server-cluster-0.9.1079953-1.el5.x86_64
qpid-cpp-server-devel-0.9.1079953-1.el5.x86_64
qpid-cpp-server-rdma-0.9.1079953-1.el5.x86_64
qpid-cpp-server-ssl-0.9.1079953-1.el5.x86_64
qpid-cpp-server-store-0.9.1079953-1.el5.x86_64
qpid-cpp-server-xml-0.9.1079953-1.el5.x86_64
qpid-dotnet-0.4.738274-2.el5.x86_64
qpid-java-client-0.9.1080013-2.el5.noarch
qpid-java-common-0.9.1080013-2.el5.noarch
qpid-java-example-0.9.1080013-2.el5.noarch
qpid-qmf-0.9.1079953-3.el5.x86_64
qpid-qmf-devel-0.9.1079953-3.el5.x86_64
qpid-tests-0.9.1073306-1.el5.noarch
qpid-tools-0.9.1078967-1.el5.noarch
rh-qpid-cpp-tests-0.9.1079953-1.el5.x86_64
ruby-qpid-qmf-0.9.1079953-3.el5.x86_64
sesame-0.9.4443-3.el5.x86_64
sesame-debuginfo-0.9.4443-3.el5.x86_64

The feature works as expected.

-> VERIFIED
Comment 10 Misha H. Ali 2011-05-31 07:02:18 UTC 
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,8 +1,8 @@
-Cause:
+Cause: There was no statistic defined for a connection tracking the number of messages sent in either direction (only frames and bytes)
-There was no statistic defined for a connection tracking the number of messages sent in either direction (only frames and bytes)
+Consequence: Could not accurately determine how many messages had been sent over a connection.
-Consequence:
+Change: Added such statistics
-Could not accurately determine how many messages had been sent over a connection.
+Result: Can now monitor the message flow over connections.
-Change:
+
-Added such statistics
+Release Note Entry:
-Result:
+
-Can now monitor the message flow over connections.+A new feature is added to track statistics about the number of messages transferred over the connection instead of tracking only the numbers of frames and bytes transferred across the connection.
Comment 11 Misha H. Ali 2011-06-06 03:16:23 UTC 
Technical note can be viewed in the release notes for 2.0 at the documentation stage here:

http://documentation-stage.bne.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/2.0/html-single/MRG_Release_Notes/index.html#tabl-MRG_Release_Notes-RHM_Update_Notes-RHM_Update_Notes
Comment 12 errata-xmlrpc 2011-06-23 15:49:53 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2011-0890.html