Bug 1613521

Summary: Top File Operations panel still provides data after gluster profiling has been disabled
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Martin Bukatovic <mbukatov>
Component: web-admin-tendrl-monitoring-integrationAssignee: gowtham <gshanmug>
Status: CLOSED WONTFIX QA Contact: sds-qe-bugs
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: mbukatov, nthomas, rhs-bugs, sankarshan
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-08 18:51:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Screenshot #1: "Top File Operations" panel shows data even though the profiling is disabled for few days, as can be seen on "File Operations For Locks" panel (which shown no values for most of time)
none
Screenshot #2: Top File Operations panel shown no data (as expected), when the profiling was disabled during the whole selected time range none

Description Martin Bukatovic 2018-08-07 17:48:38 UTC 
Description of problem
======================

Top File Operations panel still provides some values even after gluster
profiling has been disabled.

This is a problem because this panel uses data available only when the
profiling is enabled[1] and so showing values when the profiling
data are no longer available is a little misleading.

Note that this scenario is not synthetic, because the gluster profiling
feature has some performance impact and so customers are likely to have it
enabled only when they need to debug something.

[1] BZ 1590693 (doc BZ about profiling)

Version-Release number of selected component
============================================

tendrl-monitoring-integration-1.6.3-7.el7rhgs.noarch

How reproducible
================

100 %

Steps to Reproduce
==================

1. Prepare Gluster Trusted Storage Pool with at least one volume
2. Install RHGS WA via tendrl-ansible
3. Import Trusted Storage Pool with profiling enabled
4. Wait a day or two
5. Disable profiling on cluster level via RHGS WA
6. Wait a day or two
7. Check "Top File Operations" panel on Volume dashboard

Actual results
==============

Top File Operations panel provides value, even though the profiling is disabled
for few days already.

See screenshot #1 (when the profiling was enabled only for for a brief initial
section of a time range shown on the screenshot and then disabled for the rest
of the time range).

Expected results
================

Top File Operations panel provides no values when the profiling is disabled.

Additional info
===============

This seems to be caused by the way Grafana charts works.

When I select a time range during which the profiling was disabled the whole
time, no values are shown in Top File Operations panel as expected. See
screenshot #2.

But when I select a time range during which the profiling was enabled and then
disabled, the values are provided in the panel, no matter how large part of the
time range accounts for the time with profiling enabled. See screenshot #1.

This is a problem, because I don't think that it makes sense to compute some
stats for a time range when the underlying data are not available for
the whole time range.

Such approach would be similar to a report about weekly average number of
meteorites based on data from Monday only (because it was bad weather for the
rest of the week), but computed as a weekly average as if we had the data
anyway.
Comment 1 Martin Bukatovic 2018-08-07 17:52:13 UTC 
Created attachment 1474070 [details]
Screenshot #1: "Top File Operations" panel shows data even though the profiling is disabled for few days, as can be seen on "File Operations For Locks" panel (which shown no values for most of time)
Comment 2 Martin Bukatovic 2018-08-07 17:53:48 UTC 
Created attachment 1474071 [details]
Screenshot #2: Top File Operations panel shown no data (as expected), when the profiling was disabled during the whole selected time range
Comment 3 Martin Bukatovic 2018-08-07 18:14:18 UTC 
Full list of packages on WA server machine for reference:

[root@mbukatov-usm2-server ~]# rpm -qa | grep tendrl | sort 
tendrl-ansible-1.6.3-5.el7rhgs.noarch
tendrl-api-1.6.3-4.el7rhgs.noarch
tendrl-api-httpd-1.6.3-4.el7rhgs.noarch
tendrl-commons-1.6.3-9.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-7.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-7.el7rhgs.noarch
tendrl-node-agent-1.6.3-9.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-ui-1.6.3-8.el7rhgs.noarch
Comment 4 gowtham 2018-08-08 06:29:55 UTC 
Collectd stopped pushing volume profile data when profiling is disabled. I have checked graphite also its latest values in a time series are empty only. 
But grafana displays this value from last one-hour data. After one hour these values are removed from grafana. Default time range value in grafana is 1 hour.
Comment 6 Nishanth Thomas 2018-08-13 07:49:46 UTC 
@mbukatov, Can you take a look at https://bugzilla.redhat.com/show_bug.cgi?id=1613521#c4 ?
If grafana is showing only older data points(untill profiling is enabled), its not a bug. Please confirm
Comment 7 Martin Bukatovic 2018-08-13 20:05:57 UTC 
(In reply to Nishanth Thomas from comment #6)
> @mbukatov, Can you take a look at
> https://bugzilla.redhat.com/show_bug.cgi?id=1613521#c4 ?
> If grafana is showing only older data points(untill profiling is enabled),
> its not a bug. Please confirm

I understand this aspect of Grafana (it depends on what time range you select)
and think that it's a bit misleading in this particular case (when no data means
"no measurement/evidence"), see Additional info section of this BZ for details.