Bug 1531139

Summary: [RFE] Brick Utilization: threshold breached Alert needs to be generated for brick usage above 90%
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Annette Clewett <aclewett>
Component: web-admin-tendrl-monitoring-integrationAssignee: Nishanth Thomas <nthomas>
Status: CLOSED ERRATA QA Contact: Filip Balák <fbalak>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.3CC: aclewett, amukherj, fbalak, gshanmug, jcall, mbukatov, nthomas, rhs-bugs, suprasad
Target Milestone: ---Keywords: FutureFeature
Target Release: RHGS 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: tendrl-commons-1.6.1-2.el7rhgs tendrl-node-agent-1.6.1-2.el7rhgs tendrl-api-1.6.1-2.el7rhgs tendrl-gluster-integration-1.6.1-2.el7rhgs tendrl-ui-1.6.1-2.el7rhgs tendrl-monitoring-integration-1.6.1-2.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 07:00:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1503132    

Description Annette Clewett 2018-01-04 16:21:52 UTC 
Description of problem:
Currently when gluster volume usage (brick usage) goes above 75% an Alert is generated and in the dashboard the color changes to Amber. If the gluster volume (brick usage) goes above 90% the color in the dashboard changes to Red BUT there is no Alert for crossing the 90% usage.

Version-Release number of selected component (if applicable):
tendrl-monitoring-integration-1.5.4-14.el7rhgs.noarch

How reproducible:
Always

Steps to Reproduce:
1.Identify gluster volume and write data into volume to exceed 75% threshold. 
2.Tendrl Alert should be generated for all bricks in the volumes. Validate that in dashboard the color is now Amber for this volume/bricks.
3. In the case of CRS 3 Alerts will be generated and sent to specified email address, one for each brick (gluster hostname is now shown because this Alert was generated from customer tendrl installation). 

Brick utilization of <gluster_server_hostname>:|var|lib|heketi|mounts|vg_0bfd0da65ef15a9d75692a67b838cfc9|brick_c5e7ee1e0704c91888f04cfb4cb50017|brick in cluster 7bc6aa73-0c97-404b-88a2-077b5c77656a is 82.29 % which is above WARNING threshold (75 %)

4.Continue writing to gluster volume until 90% usage has been achieved. Validate that in the dashboard the color is now Red for this volume/bricks.

Actual results:
No additional Alert is sent for crossing the 90% usage value even though the color in dashboard is now Red.

Expected results:
Tendrl Alert should be sent for crossing the 75% usage AND for crossing the 90% usage (Green->Amber->Red). 

Additional info:
Comment 7 Filip Balák 2018-04-05 14:45:10 UTC 
There are generated alerts when brick utilization is over 90%. --> VERIFIED

Tested with:
tendrl-ansible-1.6.1-3.el7rhgs.noarch
tendrl-api-1.6.1-3.el7rhgs.noarch
tendrl-api-httpd-1.6.1-3.el7rhgs.noarch
tendrl-commons-1.6.1-3.el7rhgs.noarch
tendrl-grafana-plugins-1.6.1-3.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-monitoring-integration-1.6.1-3.el7rhgs.noarch
tendrl-node-agent-1.6.1-3.el7rhgs.noarch
tendrl-notifier-1.6.0-1.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-ui-1.6.1-3.el7rhgs.noarch
glusterfs-3.12.2-7.el7rhgs.x86_64
Comment 10 errata-xmlrpc 2018-09-04 07:00:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616