Bug 1725552

Summary: Auto rotate shd log file
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: replicateAssignee: Karthik U S <ksubrahm>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.5CC: amukherj, rhs-bugs, sheggodu, storage-qa-internal, vdas
Target Milestone: ---   
Target Release: RHGS 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-7 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-30 12:22:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696809    

Description Nag Pavan Chilakam 2019-07-01 05:45:55 UTC
Description of problem:
========================
seems like with the new shd changes(shd multiplexing), the auto rotation of log files is not happening.
We need to auto rotate the shd log file as we do other log files, for easier debugging, parsing of the log file.

for example in my cluster the shd log file has grown to close to 45GB on most of the nodes and no rotation is observed

the reason of the log file growing in my setup could be with the reason of multiple heals pending, however, irrespectively, we must be rotating based on whatever is the existing policy of other log file rotations like bricklogs/glusterdlogs etc

Version-Release number of selected component (if applicable):
============================
6.0.6


Steps to Reproduce:
1. created a 6 node brickmux setup
2. created 2 1x3 volumes
3. created a  144x(4+2) ec volume
4. triggered IOs from all 6 nodes(they act as clients too)
5. observed that heals were required due to some bricks going down
6. the shd log file only keeps growing with auto rotation

[root@e24-h35-740xd glusterfs]# du -sh shd/cvlt-ecv/shd.log
44G	shd/cvlt-ecv/shd.log

Expected results:
================
autorotation of shd log files is required to help consuming the logs incase of debugging

Comment 7 Nag Pavan Chilakam 2019-07-17 09:54:05 UTC
Have rechecked on 6.0.7 and saw that auto rotation of shd log is happening as mentioned in c#4
Also, as shd multiplexing changes are reverted in 6.0.8 , I have rechecked to see if shd logs are rotating and I do see shd logs rotation happening.
Hence moving bz to verified

on 6.0.8:
===========
[root@rhs-gp-srv1 glusterfs]# ls
bricks           cmd_history.log-20190717  gfproxy                glustershd.log
cli.log          geo-replication           glusterd.log           glustershd.log-20190717
cmd_history.log  geo-replication-slaves    glusterd.log-20190717  snaps
[root@rhs-gp-srv1 glusterfs]#

Comment 9 errata-xmlrpc 2019-10-30 12:22:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249