Bug 961271

Summary: Setting the client log level to DEBUG fills the client logs in GB's in minutes which after sometime leads to no space left for root
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rahul Hinduja <rhinduja>
Component: glusterfsAssignee: vsomyaju
Status: CLOSED ERRATA QA Contact: Rahul Hinduja <rhinduja>
Severity: urgent Docs Contact:
Priority: high    
Version: 2.1CC: amarts, nsathyan, rhs-bugs, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.4.0.10rhs Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-23 22:35:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rahul Hinduja 2013-05-09 10:09:38 UTC
Description of problem:
=======================

Setting the client log level to DEBUG fills the client logs in GB's in minutes which after sometime leads to no space left for root.

In 10 mins it filled the logs more than 2GB.

Note: 
Operations like cp and ls were performed from the fuse mount. Few of the bricks in replica pair were offlined.

[root@rahul-vm ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rahulvm-lv_root
                      5.5G  3.1G  2.1G  60% /
tmpfs                 499M     0  499M   0% /dev/shm
/dev/vda1             485M   55M  405M  12% /boot
10.70.36.35:/vol-dis-rep
                       11T  4.3G   11T   1% /mnt/fuse
[root@rahul-vm ~]# 
[root@rahul-vm ~]# 
[root@rahul-vm ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rahulvm-lv_root
                      5.5G  5.5G     0 100% /
tmpfs                 499M     0  499M   0% /dev/shm
/dev/vda1             485M   55M  405M  12% /boot
10.70.36.35:/vol-dis-rep
                       11T  5.0G   11T   1% /mnt/fuse
[root@rahul-vm ~]# 
[root@rahul-vm ~]# date
Thu May  9 09:05:04 IST 2013
[root@rahul-vm ~]# 

[root@rahul-vm ~]# du -sh /var/log/glusterfs/mnt-fuse.log 
3.5G	/var/log/glusterfs/mnt-fuse.log
[root@rahul-vm ~]# 

[root@rahul-vm ~]# mkdir test
mkdir: cannot create directory `test': No space left on device
[root@rahul-vm ~]# 


Version-Release number of selected component (if applicable):
=============================================================

[root@rahul-vm ~]# rpm -qa | grep gluster
glusterfs-fuse-3.4.0.4rhs-1.el6.x86_64
glusterfs-3.4.0.4rhs-1.el6.x86_64
glusterfs-devel-3.4.0.4rhs-1.el6.x86_64
glusterfs-debuginfo-3.4.0.4rhs-1.el6.x86_64
[root@rahul-vm ~]# 



How reproducible:
=================

1/1

Comment 3 vsomyaju 2013-05-24 08:57:59 UTC
Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/8017/
Upstream patch: http://review.gluster.com/#/c/5086/

Comment 4 Rahul Hinduja 2013-08-07 08:53:17 UTC
verified with: glusterfs-3.4.0.17rhs-1.el6.x86_64

Comment 5 Scott Haines 2013-09-23 22:35:29 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html