Description of problem ====================== Top File Operations panel still provides some values even after gluster profiling has been disabled. This is a problem because this panel uses data available only when the profiling is enabled[1] and so showing values when the profiling data are no longer available is a little misleading. Note that this scenario is not synthetic, because the gluster profiling feature has some performance impact and so customers are likely to have it enabled only when they need to debug something. [1] BZ 1590693 (doc BZ about profiling) Version-Release number of selected component ============================================ tendrl-monitoring-integration-1.6.3-7.el7rhgs.noarch How reproducible ================ 100 % Steps to Reproduce ================== 1. Prepare Gluster Trusted Storage Pool with at least one volume 2. Install RHGS WA via tendrl-ansible 3. Import Trusted Storage Pool with profiling enabled 4. Wait a day or two 5. Disable profiling on cluster level via RHGS WA 6. Wait a day or two 7. Check "Top File Operations" panel on Volume dashboard Actual results ============== Top File Operations panel provides value, even though the profiling is disabled for few days already. See screenshot #1 (when the profiling was enabled only for for a brief initial section of a time range shown on the screenshot and then disabled for the rest of the time range). Expected results ================ Top File Operations panel provides no values when the profiling is disabled. Additional info =============== This seems to be caused by the way Grafana charts works. When I select a time range during which the profiling was disabled the whole time, no values are shown in Top File Operations panel as expected. See screenshot #2. But when I select a time range during which the profiling was enabled and then disabled, the values are provided in the panel, no matter how large part of the time range accounts for the time with profiling enabled. See screenshot #1. This is a problem, because I don't think that it makes sense to compute some stats for a time range when the underlying data are not available for the whole time range. Such approach would be similar to a report about weekly average number of meteorites based on data from Monday only (because it was bad weather for the rest of the week), but computed as a weekly average as if we had the data anyway.
Created attachment 1474070 [details] Screenshot #1: "Top File Operations" panel shows data even though the profiling is disabled for few days, as can be seen on "File Operations For Locks" panel (which shown no values for most of time)
Created attachment 1474071 [details] Screenshot #2: Top File Operations panel shown no data (as expected), when the profiling was disabled during the whole selected time range
Full list of packages on WA server machine for reference: [root@mbukatov-usm2-server ~]# rpm -qa | grep tendrl | sort tendrl-ansible-1.6.3-5.el7rhgs.noarch tendrl-api-1.6.3-4.el7rhgs.noarch tendrl-api-httpd-1.6.3-4.el7rhgs.noarch tendrl-commons-1.6.3-9.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-7.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-7.el7rhgs.noarch tendrl-node-agent-1.6.3-9.el7rhgs.noarch tendrl-notifier-1.6.3-4.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch tendrl-ui-1.6.3-8.el7rhgs.noarch
Collectd stopped pushing volume profile data when profiling is disabled. I have checked graphite also its latest values in a time series are empty only. But grafana displays this value from last one-hour data. After one hour these values are removed from grafana. Default time range value in grafana is 1 hour.
@mbukatov, Can you take a look at https://bugzilla.redhat.com/show_bug.cgi?id=1613521#c4 ? If grafana is showing only older data points(untill profiling is enabled), its not a bug. Please confirm
(In reply to Nishanth Thomas from comment #6) > @mbukatov, Can you take a look at > https://bugzilla.redhat.com/show_bug.cgi?id=1613521#c4 ? > If grafana is showing only older data points(untill profiling is enabled), > its not a bug. Please confirm I understand this aspect of Grafana (it depends on what time range you select) and think that it's a bit misleading in this particular case (when no data means "no measurement/evidence"), see Additional info section of this BZ for details.