Bug 1507942

Summary: Weird line in Memory Utilization chart when stressed
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Filip Balák <fbalak>
Component: web-admin-tendrl-monitoring-integrationAssignee: Ankush Behl <anbehl>
Status: CLOSED ERRATA QA Contact: Martin Kudlej <mkudlej>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.3CC: bmekala, julim, nthomas, ppenicka, rghatvis, sankarshan
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tendrl-monitoring-integration-1.5.4-3.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-18 04:39:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Memory Utilization chart none

Description Filip Balák 2017-10-31 14:12:39 UTC
Created attachment 1345940 [details]
Memory Utilization chart

Description of problem:
When I run stress to increase memory consumption to be more than 80% then there appears strange orange line in Memory Utilization chart in Hosts dashboard. This line does not have any label and orientation of its filling is up (other lines have some filling from 0% to the value that represents them). The orange line doesn't reflect cached memory. For that is used another line that is visualized correctly.

Version-Release number of selected component (if applicable):
tendrl-api-httpd-1.5.3-2.el7rhgs.noarch
tendrl-grafana-selinux-1.5.3-2.el7rhgs.noarch
tendrl-selinux-1.5.3-2.el7rhgs.noarch
tendrl-node-agent-1.5.3-3.el7rhgs.noarch
tendrl-ui-1.5.3-2.el7rhgs.noarch
tendrl-grafana-plugins-1.5.3-2.el7rhgs.noarch
tendrl-notifier-1.5.3-1.el7rhgs.noarch
tendrl-ansible-1.5.3-2.el7rhgs.noarch
tendrl-commons-1.5.3-1.el7rhgs.noarch
tendrl-api-1.5.3-2.el7rhgs.noarch
tendrl-monitoring-integration-1.5.3-2.el7rhgs.noarch
glusterfs-3.8.4-50.el7rhgs.x86_64

How reproducible:
80%

Steps to Reproduce:
1. Import cluster with volume.
2. Stress the memory consumption above 80%. E.g.:
   stress --vm-bytes $(awk '/MemFree/{printf "%d\n", $2 * 2;}' < /proc/meminfo)k --vm-keep -m 1
3. Check Memory Utilization chart in Hosts dashboard.

Actual results:
There appears orange line with no label.

Expected results:
There should be no line that is not described in chart legend.

Additional info:

Comment 1 Ankush Behl 2017-11-02 10:14:28 UTC
@fbalak The orange color line is a threshold line, which is set to 80%. If a threshold is increased by 80% the warning will be raised which then will be shown in tendrl-ui.

Comment 2 Filip Balák 2017-11-02 10:45:13 UTC
Ok, the line seemed to me little odd because there was no tooltip and the colour is almost the same as colour for cached memory but this is probably not a bug. It would be good to change the colour to red and decrease its opacity if possible. Maybe it would be also good to add a tooltip for this line that says something like "Warning - Almost out of memory".

Comment 3 Martin Kudlej 2017-11-02 12:02:38 UTC
I think this is bug because it should be clear what is it.

Comment 4 Ankush Behl 2017-11-04 20:37:13 UTC
@julim can you provide a proper tooltip message for the warning threshold bar.

Comment 5 Ankush Behl 2017-11-04 21:31:15 UTC
@fbalak I tried using the red color bar with low opacity but it still feels the same because this time the "used" memory bar in memory panel seems to be conflicting(both are getting red). So I think we can add a tooltip to the panel which can clearly state that there is a warning threshold bar set at 80%.

Comment 6 Petr Penicka 2017-11-08 14:13:22 UTC
Triage Nov 8: pending Ju's input.

Comment 7 Ju Lim 2017-11-08 19:13:19 UTC
@anbehl

How about the following tooltip: "Memory utilization has exceeded the 80% warning threshold"?

Comment 8 Ankush Behl 2017-11-08 19:29:46 UTC
@julim Sorry I forgot to mention the context for tool-tip. So we need tool-tip that specifies user that we have a threshold bar set at 80% so don't get confused.

In this case, the tool-tip you mentioned will not be an apt tool-tip I think.

Comment 9 Ju Lim 2017-11-08 19:46:20 UTC
Try "The memory utilization threshold is set to >= 80% and is displayed as an orange area in the 80-100% range."

Comment 10 Ankush Behl 2017-11-08 19:53:51 UTC
@rghatvis can you review the text suggested by @julim and provide your inputs on it.

Comment 11 Ju Lim 2017-11-08 19:55:39 UTC
Alternatively, try "Values in the 80-100% range exceed the memory utilization threshold" to be less wordy, but @rghatvis can probably provide better guidance on the tooltip.

Is it possible instead of doing an orange area for 80-100% to only show a orange line at 80% as the area does seem to be confusing.  It's not typical to see the area shaded (unless it's a dynamic threshold) especially since the threshold is set statically set to 80% (and it does not vary).

Comment 13 Rakesh 2017-11-08 22:11:27 UTC
(In reply to Ankush Behl from comment #10)
> @rghatvis can you review the text suggested by @julim
> and provide your inputs on it.

Ankush, 
Both the versions suggested by Ju looks good to me. I expanded a bit(not succinct this time):

"The memory utilization threshold is set to 80%. Reaching or exceeding this value displays an orange horizontal line indicating high memory usage."

We don't need to mention about the orange shaded area(not helpful to the user) and it would be nice if we can take it off from the UI as Ju suggested.

Ju, correct me if I am wrong, aren't tooltips used for supplementary information? Info that is not super important to the user? In my knowledge yes, it is. Then why are we considering a "tooltip" to warn or alert the user about high memory consumption? This is a critical alert so instead of a tooltip where the user needs to hover over the icon, how about a pop-up box popping out to warn the user?

Comment 14 Ankush Behl 2017-11-10 06:57:08 UTC
Added Tool-tip which is provided by @rghatvis

Comment 15 Ju Lim 2017-11-13 21:24:24 UTC
@rghatvis We have to work within the constraints of what the tool (i.e. Grafana) allows.  Ankush thinks it's possible to just do a line (vs. the shaded area) for the static threshold.  It is not uncommon that as you hover over different things in the chart that the tooltip will give you additional / supplementary information indicating what it is.

A pop-up box would be annoying to a user especially when you're looking at a "dashboard" which may contain multiple threshold breaches, and it will quickly very noisy and potentially annoying if lots of pop-ups sprung up all over, which require user to click to close out the pop-up.

There are other treatments that can be done, but probably beyond the scope for what is needed at present, e.g. remediation, correlation with the root cause of the event that caused the breach, etc.

Comment 17 Bala Konda Reddy M 2017-11-18 12:49:20 UTC
Verified with tendrl-monitoring-integration-1.5.4-3

Imported the cluster. stressed one storage node as mentioned in the description.

For Memory Utilization widget, the tool-tip says the following.
 
"The memory utilization threshold is set to 80%. Reaching or exceeding this value displays an orange horizontal line indicating high memory usage." as suggested from the previous comments.

The explanation for the line at 80% is given and it is appropriate. After 80% of used memory, there is a orange shaded region 80-100 for specifying for maximum memory utilization

Hence marking it as verified

Comment 20 errata-xmlrpc 2017-12-18 04:39:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3478