Bug 1513692

Summary:	io-stats appends now instead of overwriting which floods filesystem with logs
Product:	[Community] GlusterFS	Reporter:	Chris Holcombe <xfactor973>
Component:	io-stats	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	amukherj, bugs, joe
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-4.0.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-15 11:20:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chris Holcombe 2017-11-15 19:11:02 UTC

Description of problem:
The io-stats xlator used to have a behavior where it constantly overwrites the files every x seconds.  This behavior has changed in 3.12.x to now append to the file.  After enabling the xlator the filesystem on my cluster filled up almost immediately and crashed.  

Version-Release number of selected component (if applicable):
3.12.0 and newer

How reproducible:
Always.  

Steps to Reproduce:
1. gluster volume set gv0 diagnostics.latency-measurement on
2. gluster volume set gv0 diagnostics.count-fop-hits on
3. gluster volume set gv0 diagnostics.stats-dump-interval 10
4. gluster volume set gv0 diagnostics.fop-sample-interval 5

Actual results:
Stats files in /var/lib/glusterd/stats fill up at an alarming rate which fills up the disk and crashes gluster.  I think gluster started to back up after awhile because I couldn't delete the files fast enough.  

Expected results:
I expected to see files in /var/lib/glusterd/stats/ that don't change in size and can be cat'd to find the latest stats.  

Additional info:
This is the patch that changed the behavior: https://github.com/gluster/glusterfs/commit/fc73ae5f81ef5926e3dc2311db116250d0f2a321.  Could you please revert it and backport it to the 3.12 series?

Comment 1 Chris Holcombe 2017-11-15 19:32:29 UTC

I added a revert change to review.gluster.org: https://review.gluster.org/#/c/18788/

Comment 2 Worker Ant 2017-11-17 22:36:05 UTC

REVIEW: https://review.gluster.org/18788 (Revert \"debug/io-stats: Append stats for each interval in the same file\") posted (#3) for review on master by

Comment 3 Worker Ant 2017-11-20 18:34:20 UTC

COMMIT: https://review.gluster.org/18788 committed in master by  with a commit message- Revert "debug/io-stats: Append stats for each interval in the same file"

This reverts commit fc73ae5f81ef5926e3dc2311db116250d0f2a321.
See bug: https://bugzilla.redhat.com/show_bug.cgi?id=1513692

Change-Id: I00d5989b042d4e345621c596d5370d324948557f
Bug: 1513692
Signed-off-by: Chris Holcombe <xfactor973>

Comment 4 Shyamsundar 2018-03-15 11:20:54 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report.

glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html
[2] https://www.gluster.org/pipermail/gluster-users/