Bug 765410 (GLUSTER-3678)

Summary: 'glusterd' leaks file handles on .cmd_log_history upon SIGHUP.
Product: [Community] GlusterFS Reporter: Jeff Byers <jbyers>
Component: glusterdAssignee: Kaushal <kaushal>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: medium    
Version: 3.2.3CC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Jeff Byers 2011-10-02 08:26:40 EDT
'glusterd' leaks file handles on .cmd_log_history upon SIGHUP.

The signal SIGHUP is used to rotate the log files.
'glusterd' 3.2.3 seems to handle the 'usr-local-etc-
glusterfs-glusterd.vol.log' logfile properly and closes and
reopens the file.

But with '.cmd_log_history', it does reopen the file and
seems to be using the new file handle for logging, but it
also keeps the old file handle open. With the 'ulimit'
'nfiles' limit of 1024, 'glusterd' could eventually run out
file handles and fail with periodic log rotations.

# ls -la /usr/local/var/log/glusterfs
-rw-r--r-- 1 root root   824 Sep 28 16:52 .cmd_log_history
-rw-r--r-- 1 root root   106 Oct  1 16:54 usr-local-etc-glusterfs-glusterd.vol.log

# lsof +D /usr/local/var/log/glusterfs
COMMAND     PID USER   FD   TYPE DEVICE  SIZE   NODE NAME
glusterd  26864 root    6w   REG    8,4   824 174139 .cmd_log_history
glusterd  26864 root   13w   REG    8,4   106 174141 usr-local-etc-glusterfs-glusterd.vol.log
glusterd  26864 root   14w   REG    8,4   824 174139 .cmd_log_history

# mv .cmd_log_history .cmd_log_history.`date +%s`
# mv usr-local-etc-glusterfs-glusterd.vol.log usr-local-etc-glusterfs-glusterd.vol.log.`date +%s`

# kill -SIGHUP 26864

# lsof +D /usr/local/var/log/glusterfs
COMMAND     PID USER   FD   TYPE DEVICE  SIZE   NODE NAME
glusterd  26864 root    4w   REG    8,4   106 172117 usr-local-etc-glusterfs-glusterd.vol.log
glusterd  26864 root    6w   REG    8,4   824 174139 .cmd_log_history.1317557304
glusterd  26864 root   14w   REG    8,4   824 174139 .cmd_log_history.1317557304
glusterd  26864 root   17w   REG    8,4     0 171761 .cmd_log_history

# kill -SIGHUP 26864

# lsof +D /usr/local/var/log/glusterfs
COMMAND     PID USER   FD   TYPE DEVICE  SIZE   NODE NAME
glusterd  26864 root    6w   REG    8,4   824 174139 .cmd_log_history.1317557304
glusterd  26864 root   13w   REG    8,4  1218 172117 usr-local-etc-glusterfs-glusterd.vol.log
glusterd  26864 root   14w   REG    8,4   824 174139 .cmd_log_history.1317557304
glusterd  26864 root   17w   REG    8,4     0 171761 .cmd_log_history
glusterd  26864 root   18w   REG    8,4     0 171761 .cmd_log_history
Comment 1 Anand Avati 2011-10-28 05:03:21 EDT
CHANGE: http://review.gluster.com/584 (Closes the .cmd_log_history file and reopens it to prevent leakage of file) merged in master by Vijay Bellur (vijay@gluster.com)