Bug 1267876

Summary: Log rotate is not happening on RHGS3.1 nodes where longevity test is going on.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Triveni Rao <trao>
Component: coreAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED WORKSFORME QA Contact: storage-qa-internal <storage-qa-internal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: atumball, mzywusko, rhs-bugs
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-29 17:45:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Triveni Rao 2015-10-01 09:31:19 UTC
Description of problem:
============================
Log rotate is not happening on RHGS3.1 nodes where longevity test is going on.
I have a longevity test setup running continuous IO from past 4 weeks, i see that log rotate doesnot happen on the /var/log/glusterfs/bricks and logs are filling up the /root space. I also see that some bricks logs are being deleted and fd s are not released causing the /root space 100% full.


Version-Release number of selected component (if applicable):
=============================================================
[root@pandora glusterfs]# glusterfs --version
glusterfs 3.7.1 built on Jul 19 2015 02:16:04
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@pandora glusterfs]# 

[root@pandora glusterfs]# rpm -qa | grep gluster
glusterfs-api-3.7.1-11.el7rhgs.x86_64
glusterfs-cli-3.7.1-11.el7rhgs.x86_64
vdsm-gluster-4.16.20-1.2.el7rhgs.noarch
glusterfs-libs-3.7.1-11.el7rhgs.x86_64
glusterfs-client-xlators-3.7.1-11.el7rhgs.x86_64
glusterfs-server-3.7.1-11.el7rhgs.x86_64
glusterfs-rdma-3.7.1-11.el7rhgs.x86_64
gluster-nagios-common-0.2.0-2.el7rhgs.noarch
gluster-nagios-addons-0.2.4-4.el7rhgs.x86_64
glusterfs-3.7.1-11.el7rhgs.x86_64
glusterfs-fuse-3.7.1-11.el7rhgs.x86_64
glusterfs-geo-replication-3.7.1-11.el7rhgs.x86_64
[root@pandora glusterfs]# 

How reproducible:
=================
easily.

Steps to Reproduce:
====================
1. prepare 4 nodes cluster gluster with rhgs3.1 installed.
2. run some continuous IOs
3. check the brick logs and /root space

Actual results:
===============
1. Log rotate is not happening.
2. some logs are being deleted automatically and opn fds filling the /root space.
3. sometimes i have to rotate the log manually 

Expected results:
================

Log rotate should happen and no logs to be deleted.

Additional info:
================

[root@pandora glusterfs]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/rhgs_pandora-root   50G   48G  2.4G  96% /
devtmpfs                        16G     0   16G   0% /dev
tmpfs                           16G     0   16G   0% /dev/shm
tmpfs                           16G   82M   16G   1% /run
tmpfs                           16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/rhgs_pandora-home  400G  430M  399G   1% /home
/dev/sda1                      497M   92M  406M  19% /boot
/dev/mapper/RHS_vg1-RHS_lv1    1.9T  779M  1.9T   1% /rhs/brick1
/dev/mapper/RHS_vg2-RHS_lv2    1.9T  1.5G  1.9T   1% /rhs/brick2
/dev/mapper/RHS_vg3-RHS_lv3    1.9T  779M  1.9T   1% /rhs/brick3
/dev/mapper/RHS_vg4-RHS_lv4    1.9T 1020M  1.9T   1% /rhs/brick4
/dev/mapper/RHS_vg5-RHS_lv5    1.9T   34M  1.9T   1% /rhs/brick5
/dev/mapper/RHS_vg6-RHS_lv6    1.9T   34M  1.9T   1% /rhs/brick6
/dev/mapper/RHS_vg7-RHS_lv7    1.9T   34M  1.9T   1% /rhs/brick7
/dev/mapper/RHS_vg8-RHS_lv8    1.9T   34M  1.9T   1% /rhs/brick8
/dev/mapper/RHS_vg9-RHS_lv9    1.9T   34M  1.9T   1% /rhs/brick9
/dev/mapper/RHS_vg10-RHS_lv10  1.9T   34M  1.9T   1% /rhs/brick10
/dev/mapper/RHS_vg11-RHS_lv11  1.9T   34M  1.9T   1% /rhs/brick11
[root@pandora glusterfs]#


glusterfs  3346  3359           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3447           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3457           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3459           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3470           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3471           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3472           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3473           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3480           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3481           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3482           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346 11611           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346 20369           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346 23907           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
[root@rhs-client2 ~]#

Comment 5 Amar Tumballi 2018-01-29 17:45:39 UTC
Not seen in RHGS-3.3.x releases. Feel free to reopen if seen again.