Bug 1595020

Created attachment 1454526 [details]
File operations latency panel

Assuming this has been reported based on sandbox-usm1 instance, the             
full package version list follows:                                              
                                                                                
```                                                                             
[root@sandbox-usm1-server ~]# rpm -qa | grep tendrl | sort                      
tendrl-ansible-1.6.3-5.el7rhgs.noarch                                           
tendrl-api-1.6.3-3.el7rhgs.noarch                                               
tendrl-api-httpd-1.6.3-3.el7rhgs.noarch                                         
tendrl-commons-1.6.3-7.el7rhgs.noarch                                           
tendrl-grafana-plugins-1.6.3-5.el7rhgs.noarch                                   
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch                                   
tendrl-monitoring-integration-1.6.3-5.el7rhgs.noarch                            
tendrl-node-agent-1.6.3-7.el7rhgs.noarch                                        
tendrl-notifier-1.6.3-4.el7rhgs.noarch                                          
tendrl-selinux-1.5.4-2.el7rhgs.noarch                                           
tendrl-ui-1.6.3-4.el7rhgs.noarch                                                
```

We are showing the hits right now which is not percentage latency(% latency).

So changes we are going to make to a panel:

1. We are going to change the label "% latency"(row heading) to "Avg. Latency"
2. It will show the average latency value which will be in microseconds.

As of tendrl-monitoring-integration-1.6.3-7.el7rhgs.noarch, the unit of measure
are still unclear. Rechecked during dashboard overview.

Here is an example of output of `gluster volume profile` command, which this
WA panel uses to get data:

```
# gluster volume profile volume_gama_disperse_4_plus_2x2 info                   
                                                                                
Interval 2573 Stats:                                                            
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop 
 ---------   -----------   -----------   -----------   ------------        ---- 
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR 
      0.28      12.00 us      12.00 us      12.00 us              1     OPENDIR 
      2.07      90.00 us      90.00 us      90.00 us              1      STATFS 
      2.14      46.50 us      27.00 us      66.00 us              2     READDIR 
     28.95     629.50 us     296.00 us     963.00 us              2    GETXATTR 
     66.57     965.00 us     428.00 us    1816.00 us              3      LOOKUP 
```

I believe that to provide meaningful description and to improve representation
of data this BZ talks about, we need to:

 * locate and link gluster documentation of this feature
 * check with gluster eng. to provide additional insight

That said, the documentation I was able to find doesn't provide enough details
for me to understand meaning of %-latency:

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-monitoring_red_hat_storage_workload#profiling-volumes

Atin, could you help us here to understand the meaning of %-latency field so
that WA team can improve representation of panel in question and it's
description?

Pranith - Can you help Martin to explain the question asked for?

If % latency is showing the portion of latency of a particular file operation as a percentage of all operations, then this is not useful to customer.  What would customer do if the graph shows STATFS is responsible for 50% of latency ?

Even if we decide to show average latency for each operation, we still need inform customer of how to use this info.  If average latency for RELEASEDIR is higher than normal what would customer do ?  Does he then look at other resources like CPU and MEM and determine where the problem might be ?

(In reply to Anand Paladugu from comment #15)

"gluster volume profile" is created to do something similar to what "strace -Tc" does for an application/process. The way to use this tool is to run a particular workload while profile is enabled and try to get the statistics of what fops are triggered by the application on the bricks.

> If % latency is showing the portion of latency of a particular file
> operation as a percentage of all operations, then this is not useful to
> customer.  What would customer do if the graph shows STATFS is responsible
> for 50% of latency ?

If 50% of the latency is contributed by STATFS and the performance of the application is not met for the customer, there are 2 things that we need to answer:
1) Are these STATFS calls coming from DHT (or any other internal xlator) which keeps doing STATFS to figure out which subvolume it needs to create new files or is it the application which is doing all these STATFS calls.
    a) If it is DHT, then there seems to be a bug in DHT that needs to be fixed.
    b) If it is application, we need to have a conversation with the customer to find why the said application is doing so many statfs calls, based on that we should probably give a feature/fix which would do caching of STATFS to reduce the %-latency of STATFS for the workload enough to meet the latency/throughput requirements of the customer/user.

> 
> Even if we decide to show average latency for each operation, we still need
> inform customer of how to use this info.  If average latency for RELEASEDIR
> is higher than normal what would customer do ?  Does he then look at other
> resources like CPU and MEM and determine where the problem might be ?

It needs to be used exactly how the customer would use "strace -Tc" for explaining what the fs is behaving for the application workload. Just this output doesn't give all the information one may need to identify the bottlenecks in some cases but these metrics let us figure out which subsystem to look for the problems.

For example: In 3.4.0 we were working on a performance regression issue where we found a problem based on profile info (https://bugzilla.redhat.com/show_bug.cgi?id=1558989#c57).
We compared 3.3.1 profile info vs 3.4.0 profile info and the avg latency/%-latency were similar so we had to suspect the layer above it which is network and we found a regression introduced in the rpc layer. 

Pranith