This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1475721 - 'gluster volume profile' shouldn't cause negative impact on volume performance
'gluster volume profile' shouldn't cause negative impact on volume performance
Status: VERIFIED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core (Show other bugs)
unspecified
Unspecified Linux
unspecified Severity high
: ---
: RHGS 3.3.1
Assigned To: Krutika Dhananjay
Karan Sandha
: ZStream
Depends On:
Blocks: 1475687
  Show dependency treegraph
 
Reported: 2017-07-27 04:36 EDT by Amar Tumballi
Modified: 2017-10-12 07:18 EDT (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.8.4-45
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Amar Tumballi 2017-07-27 04:36:52 EDT
Description of problem:
Till now, `gluster volume profile` command is recommended to be used in a production system only when there are issues, and there is a need to understand the I/O pattern. The major reason was the impact on volume performance because of enabling the option.


Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
- this is an hypothesis - 


Actual results:
* Recommended only in few cases and not in actual production setup.

Expected results:
* Always have the option enabled, and use the metrics from filesystem to show the performance pattern over time.

Additional info:

Gluster volume profile can provide a lot of internal information on gluster's performance, and this can be used to plot metrics pretty well.
Comment 4 Krutika Dhananjay 2017-08-03 07:12:21 EDT
I don't have context on this issue except for knowing that we plan to enable profile by default and analyse the perf data. Do we know or have evidence to support the claim that enabling volume profiling does have a negative impact on performance?

-Krutika
Comment 5 Amar Tumballi 2017-08-03 07:37:09 EDT
>  we plan to enable profile by default and analyse the perf data.

Yes, above is the context for this bug to exist.

>  Do we know or have evidence to support the claim that enabling volume profiling does have a negative impact on performance?

Nothing where it was 'analyzed' before. Mostly hypothesis:

1. We don't turn it on by default, but ask customers to give the output when they get into perf issues. 
2. Inside code, if 'measure_latency' flag is set, it is a gettimeofday() syscall for every layer of translator for every fop().
3. Earlier in 'debug/io-stats' translator, we had xlator->lock which was used to update all the counters, which surely caused locking contentions at io-stats.

Hence, we want a performance run with profiling on, so we can see how much we regress, and see if there are anything more to enhance it further.
Comment 7 Atin Mukherjee 2017-09-04 05:54:49 EDT
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/117185

Note You need to log in before you can comment on or make changes to this bug.