Hide Forgot
Typically when the backend FS hangs (say during kernel soft lockups) the mount point will hang for processes accessing files from that backend. GlusterFS should recover from such a situation and not allow apps to hang at all.
PATCH: http://patches.gluster.com/patch/2676 in master (core: fix initialization of disjoint xlator graph)
PATCH: http://patches.gluster.com/patch/2677 in master (Server backend storage hang should not cause the mount point to hang.)
PATCH: http://patches.gluster.com/patch/2678 in master (protocol/server: cleanup whitespaces)
PATCH: http://patches.gluster.com/patch/2708 in master (Revert "Server backend storage hang should not cause the mount point to hang.")
PATCH: http://patches.gluster.com/patch/2707 in release-3.0 (Revert "Server backend storage hang should not cause the mount point to hang.")
We are evaluating the need to bring this in FS, instead we can write some scripts to take care of this issue.
Its hard to handle this within filesystem code, instead, the below script by Sacchi <sac> fixes the issue. Will be adding this as extras/ in the codebase, and closing the bug. ------------------------------------------ #!/bin/sh # This script keeps checking in the background if the filesystem is alive. # If the filesystem does respond then kill the glusterfsd daemon using the # pid file generated. This test is performed by running `df' continuously # on the backend. EXPORTDIR=/mnt/dist-1 PIDFILE=/var/run/glusterfsd.pid INTERVAL=30 while : do # Test if glusterfs is running, before doing anything. If glusterfs is # not running we don't have to do anything. test -f $PIDFILE || continue df $EXPORTDIR & DF_PID=$! sleep $INTERVAL if ps ae | egrep "${DF_PID}.*df" | grep -v grep then # df is hung, kill glusterfs process and remove the pid file kill -9 $(cat $PIDFILE) if [ $? -eq 0 ]; then rm -f $PIDFILE; fi fi done -------------------------------------