Bug 1725552 - Auto rotate shd log file
Summary: Auto rotate shd log file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: replicate
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: RHGS 3.5.0
Assignee: Karthik U S
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On:
Blocks: 1696809
TreeView+ depends on / blocked
 
Reported: 2019-07-01 05:45 UTC by Nag Pavan Chilakam
Modified: 2019-10-31 09:04 UTC (History)
5 users (show)

Fixed In Version: glusterfs-6.0-7
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-30 12:22:15 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:3249 0 None None None 2019-10-30 12:22:51 UTC

Description Nag Pavan Chilakam 2019-07-01 05:45:55 UTC
Description of problem:
========================
seems like with the new shd changes(shd multiplexing), the auto rotation of log files is not happening.
We need to auto rotate the shd log file as we do other log files, for easier debugging, parsing of the log file.

for example in my cluster the shd log file has grown to close to 45GB on most of the nodes and no rotation is observed

the reason of the log file growing in my setup could be with the reason of multiple heals pending, however, irrespectively, we must be rotating based on whatever is the existing policy of other log file rotations like bricklogs/glusterdlogs etc

Version-Release number of selected component (if applicable):
============================
6.0.6


Steps to Reproduce:
1. created a 6 node brickmux setup
2. created 2 1x3 volumes
3. created a  144x(4+2) ec volume
4. triggered IOs from all 6 nodes(they act as clients too)
5. observed that heals were required due to some bricks going down
6. the shd log file only keeps growing with auto rotation

[root@e24-h35-740xd glusterfs]# du -sh shd/cvlt-ecv/shd.log
44G	shd/cvlt-ecv/shd.log

Expected results:
================
autorotation of shd log files is required to help consuming the logs incase of debugging

Comment 7 Nag Pavan Chilakam 2019-07-17 09:54:05 UTC
Have rechecked on 6.0.7 and saw that auto rotation of shd log is happening as mentioned in c#4
Also, as shd multiplexing changes are reverted in 6.0.8 , I have rechecked to see if shd logs are rotating and I do see shd logs rotation happening.
Hence moving bz to verified

on 6.0.8:
===========
[root@rhs-gp-srv1 glusterfs]# ls
bricks           cmd_history.log-20190717  gfproxy                glustershd.log
cli.log          geo-replication           glusterd.log           glustershd.log-20190717
cmd_history.log  geo-replication-slaves    glusterd.log-20190717  snaps
[root@rhs-gp-srv1 glusterfs]#

Comment 9 errata-xmlrpc 2019-10-30 12:22:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249


Note You need to log in before you can comment on or make changes to this bug.