Bug 1267876 - Log rotate is not happening on RHGS3.1 nodes where longevity test is going on.
Log rotate is not happening on RHGS3.1 nodes where longevity test is going on.
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core (Show other bugs)
3.1
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
storage-qa-internal@redhat.com
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-01 05:31 EDT by Triveni Rao
Modified: 2017-03-25 12:25 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Triveni Rao 2015-10-01 05:31:19 EDT
Description of problem:
============================
Log rotate is not happening on RHGS3.1 nodes where longevity test is going on.
I have a longevity test setup running continuous IO from past 4 weeks, i see that log rotate doesnot happen on the /var/log/glusterfs/bricks and logs are filling up the /root space. I also see that some bricks logs are being deleted and fd s are not released causing the /root space 100% full.


Version-Release number of selected component (if applicable):
=============================================================
[root@pandora glusterfs]# glusterfs --version
glusterfs 3.7.1 built on Jul 19 2015 02:16:04
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@pandora glusterfs]# 

[root@pandora glusterfs]# rpm -qa | grep gluster
glusterfs-api-3.7.1-11.el7rhgs.x86_64
glusterfs-cli-3.7.1-11.el7rhgs.x86_64
vdsm-gluster-4.16.20-1.2.el7rhgs.noarch
glusterfs-libs-3.7.1-11.el7rhgs.x86_64
glusterfs-client-xlators-3.7.1-11.el7rhgs.x86_64
glusterfs-server-3.7.1-11.el7rhgs.x86_64
glusterfs-rdma-3.7.1-11.el7rhgs.x86_64
gluster-nagios-common-0.2.0-2.el7rhgs.noarch
gluster-nagios-addons-0.2.4-4.el7rhgs.x86_64
glusterfs-3.7.1-11.el7rhgs.x86_64
glusterfs-fuse-3.7.1-11.el7rhgs.x86_64
glusterfs-geo-replication-3.7.1-11.el7rhgs.x86_64
[root@pandora glusterfs]# 

How reproducible:
=================
easily.

Steps to Reproduce:
====================
1. prepare 4 nodes cluster gluster with rhgs3.1 installed.
2. run some continuous IOs
3. check the brick logs and /root space

Actual results:
===============
1. Log rotate is not happening.
2. some logs are being deleted automatically and opn fds filling the /root space.
3. sometimes i have to rotate the log manually 

Expected results:
================

Log rotate should happen and no logs to be deleted.

Additional info:
================

[root@pandora glusterfs]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/rhgs_pandora-root   50G   48G  2.4G  96% /
devtmpfs                        16G     0   16G   0% /dev
tmpfs                           16G     0   16G   0% /dev/shm
tmpfs                           16G   82M   16G   1% /run
tmpfs                           16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/rhgs_pandora-home  400G  430M  399G   1% /home
/dev/sda1                      497M   92M  406M  19% /boot
/dev/mapper/RHS_vg1-RHS_lv1    1.9T  779M  1.9T   1% /rhs/brick1
/dev/mapper/RHS_vg2-RHS_lv2    1.9T  1.5G  1.9T   1% /rhs/brick2
/dev/mapper/RHS_vg3-RHS_lv3    1.9T  779M  1.9T   1% /rhs/brick3
/dev/mapper/RHS_vg4-RHS_lv4    1.9T 1020M  1.9T   1% /rhs/brick4
/dev/mapper/RHS_vg5-RHS_lv5    1.9T   34M  1.9T   1% /rhs/brick5
/dev/mapper/RHS_vg6-RHS_lv6    1.9T   34M  1.9T   1% /rhs/brick6
/dev/mapper/RHS_vg7-RHS_lv7    1.9T   34M  1.9T   1% /rhs/brick7
/dev/mapper/RHS_vg8-RHS_lv8    1.9T   34M  1.9T   1% /rhs/brick8
/dev/mapper/RHS_vg9-RHS_lv9    1.9T   34M  1.9T   1% /rhs/brick9
/dev/mapper/RHS_vg10-RHS_lv10  1.9T   34M  1.9T   1% /rhs/brick10
/dev/mapper/RHS_vg11-RHS_lv11  1.9T   34M  1.9T   1% /rhs/brick11
[root@pandora glusterfs]#


glusterfs  3346  3359           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3447           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3457           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3459           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3470           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3471           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3472           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3473           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3480           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3481           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346  3482           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346 11611           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346 20369           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
glusterfs  3346 23907           root   22w      REG              253,0    105933  136057091 /var/log/glusterfs/bricks/rhs-brick1-ec1.log (deleted)
[root@rhs-client2 ~]#

Note You need to log in before you can comment on or make changes to this bug.