Description of problem: ======================== seems like with the new shd changes(shd multiplexing), the auto rotation of log files is not happening. We need to auto rotate the shd log file as we do other log files, for easier debugging, parsing of the log file. for example in my cluster the shd log file has grown to close to 45GB on most of the nodes and no rotation is observed the reason of the log file growing in my setup could be with the reason of multiple heals pending, however, irrespectively, we must be rotating based on whatever is the existing policy of other log file rotations like bricklogs/glusterdlogs etc Version-Release number of selected component (if applicable): ============================ 6.0.6 Steps to Reproduce: 1. created a 6 node brickmux setup 2. created 2 1x3 volumes 3. created a 144x(4+2) ec volume 4. triggered IOs from all 6 nodes(they act as clients too) 5. observed that heals were required due to some bricks going down 6. the shd log file only keeps growing with auto rotation [root@e24-h35-740xd glusterfs]# du -sh shd/cvlt-ecv/shd.log 44G shd/cvlt-ecv/shd.log Expected results: ================ autorotation of shd log files is required to help consuming the logs incase of debugging
Have rechecked on 6.0.7 and saw that auto rotation of shd log is happening as mentioned in c#4 Also, as shd multiplexing changes are reverted in 6.0.8 , I have rechecked to see if shd logs are rotating and I do see shd logs rotation happening. Hence moving bz to verified on 6.0.8: =========== [root@rhs-gp-srv1 glusterfs]# ls bricks cmd_history.log-20190717 gfproxy glustershd.log cli.log geo-replication glusterd.log glustershd.log-20190717 cmd_history.log geo-replication-slaves glusterd.log-20190717 snaps [root@rhs-gp-srv1 glusterfs]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249