Bug 1014064 - Threshold alert "Threshold crossed decreasing" is raised only if upper threshold was reached before and vice versa
Threshold alert "Threshold crossed decreasing" is raised only if upper thresh...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: Messaging_Installation_and_Configuration_Guide (Show other bugs)
Development
Unspecified Unspecified
unspecified Severity medium
: 3.0
: ---
Assigned To: Jared MORGAN
Petra Svobodová
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-01 06:55 EDT by Petra Svobodová
Modified: 2015-08-09 21:23 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-01-22 10:28:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Petra Svobodová 2013-10-01 06:55:14 EDT
Description of problem:
Threshold alert "Threshold crossed decreasing" is raised only if upper threshold was reached; if the upper threshold is not defined, the alert is not raised too.

For example:
./qc2_spout --content "akdjnrnjko" --count 5 "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"
./qc2_drain "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"

does not raise "threshold crossed decreasing" alert, but sequence:
./qc2_spout --content "akdjnrnjko" --count 10 "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"
./qc2_drain "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"

raises the "threshold crossed decreasing" alert.

However the "threshold crossed increasing" alert is raised only if lower threshold was raised before the event:
./qc2_spout --content "akdjnrnjko" --count 10 "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"

run after beaker restart raises the "threshold crossed upward alert".

But if I receive some messages (the lower threshold must not be reached) and then will reach the upper threshold again, the "threshold crossed upward" alert is not raised, see below:

./qc2_drain --count 4 "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"

./qc2_spout --count 10 --content "JIKOHUJ" "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}


Version-Release number of selected component (if applicable):
[root@dhcp-lab-252 clients]# rpm -qa | grep qpid
qpid-cpp-debuginfo-0.22-16.el6.x86_64
qpid-cpp-client-0.22-18.el6.x86_64
python-qpid-qmf-0.22-14.el6.x86_64
qpid-java-common-0.23-1.el6.noarch
qpid-cpp-server-ssl-0.22-18.el6.x86_64
qpid-snmpd-1.0.0-12.el6.x86_64
qpid-cpp-server-ha-0.22-18.el6.x86_64
qpid-jca-0.22-1.el6.noarch
qpid-proton-c-0.5-6.el6.x86_64
qpid-cpp-server-0.22-18.el6.x86_64
python-qpid-0.22-5.el6.noarch
qpid-cpp-client-rdma-0.22-18.el6.x86_64
qpid-cpp-client-ssl-0.22-18.el6.x86_64
qpid-java-client-0.23-1.el6.noarch
qpid-java-example-0.23-1.el6.noarch
qpid-cpp-server-devel-0.22-18.el6.x86_64
qpid-tools-0.22-5.el6.noarch
qpid-cpp-server-store-0.22-18.el6.x86_64
qpid-cpp-server-xml-0.22-18.el6.x86_64
perl-qpid-0.22-5.el6.x86_64
qpid-cpp-client-devel-docs-0.22-18.el6.noarch
qpid-jca-xarecovery-0.22-1.el6.noarch
qpid-qmf-0.22-14.el6.x86_64
qpid-cpp-client-devel-0.22-18.el6.x86_64
ruby-qpid-qmf-0.22-14.el6.x86_64
qpid-cpp-server-rdma-0.22-18.el6.x86_64
rh-qpid-cpp-tests-0.22-18.el6.x86_64
qpid-proton-c-devel-0.5-6.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Start  receiver client ./qc2_drain --forever "qmf.default.topic/agent.ind.event.org_apache_qpid_broker.queueThresholdCrossedDownward.#"
2. In another terminal send messages: 
./qc2_spout --content "akdjnrnjko" --count 5 "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"
3. Receive the messages:
./qc2_drain "threshold_queue; {create:always, node:{x-declare:{arguments:{'qpid.alert_size_down':11,'qpid.alert_size_up':100}}}}"


Actual results:
No alert (received as message in the "qmf.default.topic/agent.ind.event.org_apache_qpid_broker.queueThresholdCrossedDownward.#" address in the first terminal) is raised.


Expected results:
Message alert is caught in the address "qmf.default.topic/agent.ind.event.org_apache_qpid_broker.queueThresholdCrossedDownward.#".

Additional info:
Comment 1 Ted Ross 2013-10-01 08:12:39 EDT
I claim that the reproducer is producing correct results.

As I understand it, you are creating a queue with an upward threshold of 100 bytes and a downward threshold of 11 bytes.  You then produce 50 bytes, then drain all 50 bytes.

Since the queue size never crossed the upward threshold, it is correct that the downward threshold alert was _not_ generated.

This feature implements hysteresis in threshold alert generation.  It prevents floods of threshold alerts being sent.  Imagine if the queue was flowing steadily with 20 byte messages being produced and consumed continuously.  You certainly would not want a crossing-downward alert sent for every message.
Comment 2 Petra Svobodová 2013-10-02 02:25:42 EDT
This behavior seems to be correct, but it should be more detailed in the documentation. I will change this bug to documentation one.
Comment 3 Petra Svobodová 2013-10-02 03:05:13 EDT
Please add to documentation that: 

In above light current documentation in MICG is not complete:

Threshold crossed increasing
The increasing event is raised when the queue depth goes from (upper-threshold - 1) to upper-threshold.

Threshold crossed decreasing
    The decreasing event is raised when the queue depth goes from (lower-threshold + 1) to lower-threshold. 

Should be:
Threshold crossed increasing
    The increasing event is raised when the queue depth goes from (upper-threshold - 1) to upper-threshold and queue depth crosses lower threshold before.

Threshold crossed decreasing
    The decreasing event is raised when the queue depth goes from (lower-threshold + 1) to lower-threshold and queue depth crosses upper threshold before.

Note:
Both threshold should be set to receive alert events correctly.
Comment 4 Joshua Wulf 2013-10-02 22:49:29 EDT
http://deathstar1.usersys.redhat.com:3000/builds/18173-Messaging_Installation_and_Configuration_Guide/#Queue_Threshold_Alerts_Edge-triggered


New doc text:

Note: The upper and lower thresholds should have a gap between them to limit event rates, and both must be set to generate event notifications.

Events

There are two different events:

Threshold crossed increasing

The increasing event is raised when the queue depth goes from (upper-threshold - 1) to upper-threshold. This event is not retriggered by fluctuation around the upper-threshold, until a decreasing event is triggered. This prevents multiple continuous triggering of the increasing event.

Threshold crossed decreasing

The decreasing event is raised when the queue depth goes from (lower-threshold + 1) to lower-threshold. After a decreasing event, a further rise to the upper-threshold will trigger the increasing event.
Comment 5 Petra Svobodová 2013-10-10 09:24:05 EDT
The new text is good, but could you possibly fill in a note that the decreasing event is raised only if the increasing one was raised before it (similarly like the increasing event)?

current version:
The decreasing event is raised when the queue depth goes from (lower-threshold + 1) to lower-threshold. After a decreasing event, a further rise to the upper-threshold will trigger the increasing event.

Should be:
The decreasing event is raised when the queue depth goes from (lower-threshold + 1) to lower-threshold. The decreasing event is triggered only if the increasing event was raised before it. After a decreasing event, a further rise to the upper-threshold will trigger the increasing event.
Comment 6 Joshua Wulf 2013-10-10 22:34:21 EDT
I've introduced the idea of the "increasing event flag", which makes it easier to describe the behaviour. It's probably internally implemented using something like this, but as a conceptual entity it makes it clearer:

http://deathstar1.usersys.redhat.com:3000/builds/18173-Messaging_Installation_and_Configuration_Guide/#Queue_Threshold_Alerts_Edge-triggered

Threshold crossed increasing

The increasing event is raised when the queue depth goes from (upper-threshold - 1) to upper-threshold and the increasing event flag is not already set. When the an increasing event occurs the increasing event flag is set. The increasing event flag must be cleared (by a decreasing event) before further increasing events will be raised. This prevents multiple retriggering of this event by fluctuation of queue depth around the upper-threshold.

Threshold crossed decreasing

The decreasing event is raised when the increasing event flag is set and the queue depth goes from (lower-threshold + 1) to lower-threshold. The decreasing event clears the increasing event flag, allowing further increasing events to be triggered and preventing multiple retriggering of this event by fluctuation of queue depth around the lower-threshold.
Comment 9 Petra Svobodová 2013-10-25 02:33:30 EDT
Documentation of queue threshold alerts is current and reflects the current state.

--> VERIFIED

Note You need to log in before you can comment on or make changes to this bug.