Hide Forgot
'glusterd' leaks file handles on .cmd_log_history upon SIGHUP. The signal SIGHUP is used to rotate the log files. 'glusterd' 3.2.3 seems to handle the 'usr-local-etc- glusterfs-glusterd.vol.log' logfile properly and closes and reopens the file. But with '.cmd_log_history', it does reopen the file and seems to be using the new file handle for logging, but it also keeps the old file handle open. With the 'ulimit' 'nfiles' limit of 1024, 'glusterd' could eventually run out file handles and fail with periodic log rotations. # ls -la /usr/local/var/log/glusterfs -rw-r--r-- 1 root root 824 Sep 28 16:52 .cmd_log_history -rw-r--r-- 1 root root 106 Oct 1 16:54 usr-local-etc-glusterfs-glusterd.vol.log # lsof +D /usr/local/var/log/glusterfs COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME glusterd 26864 root 6w REG 8,4 824 174139 .cmd_log_history glusterd 26864 root 13w REG 8,4 106 174141 usr-local-etc-glusterfs-glusterd.vol.log glusterd 26864 root 14w REG 8,4 824 174139 .cmd_log_history # mv .cmd_log_history .cmd_log_history.`date +%s` # mv usr-local-etc-glusterfs-glusterd.vol.log usr-local-etc-glusterfs-glusterd.vol.log.`date +%s` # kill -SIGHUP 26864 # lsof +D /usr/local/var/log/glusterfs COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME glusterd 26864 root 4w REG 8,4 106 172117 usr-local-etc-glusterfs-glusterd.vol.log glusterd 26864 root 6w REG 8,4 824 174139 .cmd_log_history.1317557304 glusterd 26864 root 14w REG 8,4 824 174139 .cmd_log_history.1317557304 glusterd 26864 root 17w REG 8,4 0 171761 .cmd_log_history # kill -SIGHUP 26864 # lsof +D /usr/local/var/log/glusterfs COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME glusterd 26864 root 6w REG 8,4 824 174139 .cmd_log_history.1317557304 glusterd 26864 root 13w REG 8,4 1218 172117 usr-local-etc-glusterfs-glusterd.vol.log glusterd 26864 root 14w REG 8,4 824 174139 .cmd_log_history.1317557304 glusterd 26864 root 17w REG 8,4 0 171761 .cmd_log_history glusterd 26864 root 18w REG 8,4 0 171761 .cmd_log_history
CHANGE: http://review.gluster.com/584 (Closes the .cmd_log_history file and reopens it to prevent leakage of file) merged in master by Vijay Bellur (vijay)