Bug 832609

Maybe the criteria should just be "if operations have been pending for a 'long' time," without the added constraint that "no other operations have completed in that time."

If a brick is degraded, less than 100% healthy, I would rather see it taken out of service completely than kept around longer.  Fail fast.

I've had the same problem. Others have as well and filed a bug report, but it was closed as wontfix. This should be a high priority issue as it does stop all access to that volume indefinately.

Please also backport the fix to the 3.1 and 3.2 release branches and issue new patch releases of both.

I can reproduce exactly this on AWS EC2 by force-detaching an EBS volume.

Sorry about delay in working on this.. Will take up this soon.

I'm seeing a similar issue on my end as well...  in my case one of the glusterfsd processes jumps up to 100% and gluster reports the brick as not connected.  I don't think I have issues with the entire volume though, as the other bricks seem to be functioning fine.  I'm running a distributed replicate volume and the other replicate (of the disconnected brick) receives data just fine.  A 'volume heal info' just shows an increasing number of entries for the disconnected brick.

Problem #2 I'm seeing on my end as well.  If I stop the glusterd service, the "broken" glusterfsd process remains behind and killing it creates a zombie as mentioned previously.  Also, I only seem to notice this problem under load.

I'd also like to note on my end a "gluster volume status" shows the brick in question as online, but the "gluster volume heal info" shows it as not connected.

Is this targeted for any particular milestone or tree?

*** Bug 764756 has been marked as a duplicate of this bug. ***

Sorry for delay in updating the bug. We have been thinking about this bug, but it is not very simple mainly because we would not be getting any 'notification' from the backend for the hang. If we ever get such notification, then it becomes easy to trigger a 'exit()' from the process itself in such cases (thus closing all the socket connections too).

We could get rid of this by reducing the 'frame-timeout' option, but that won't solve the problem completely. One of the thoughts of solving this is by running a script (per brick process), which keeps doing a 'df -h ${brick-dir}' in background, and if it doesn't get a reply in some '${brick-timeout}' (thinking of 30second default for now? (or someone wants it 42?) with it being configurable), kill the brick process.

Let me know if that is fine.

Created attachment 685710 [details]
script for checking backend hang killing brick processes

Please execute the attached script when the problem arises and see if it works for you.

Need 'Verification' for this bug on script attached in comment #11, as it seems to work for us to solve the issue. Once verified will move it to appropriate status.

I tried but was not able to verify this solution.  Here's the test procedure I followed...

1. Launched EC2 instance with two EBS volumes attached
2. Installed glusterfs and created a replica 2 volume with the two local bricks
3. Mounted the volume locally
4. Started screen with two windows, one writing the date to a file on the client every second, the other with tail -f on the date file
5. In another session, started the script from comment #11
6. Did a "force detach" of one of the two EBS volumes (a brick)

At this point I observed the writer & reader hang.  The script however did not kill the bad brick process.

For some strange reason, after force detaching the EBS volume, df still returned quickly -- with information about the detached brick's mount point.  I could even still ls the detached brick's mount point and see the date file.  When I tried to tail the date file on the detached brick though the tail hung.

I did a kill and a kill -9 on the detached brick's glusterfsd process which sent it into zombie/defunct state.  After this the client did detect a ping timeout and continued with the remaining brick.

--- Last line from the brick log showing that I killed it ---

[2013-02-16 00:37:13.664909] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f17449bacbd] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1744c8de9a] (-->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xd5) [0x7f1745773a55]))) 0-: received signum (15), shutting down

--- Client log recognizing the brick process died ---

[2013-02-16 00:37:58.800634] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-foo-client-1: server 10.70.7.170:24010 has not responded in the last 42 seconds, disconnecting.

---

Notice that the brick log reports signal 15 at :13 seconds, but the client detected ping timeout 45 seconds later at :58.  That extra 3 seconds may have been how long it took me to check ps again after the first kill, and kill a second time with -9, but I'm not sure about that.

Thanks & I hope this feedback is helpful.

That test in previous comment was using glusterfs 3.3.1 on ubuntu precise (12.04) by the way.

Thanks Louis :-)

That one gave much better understanding of the problem. Seems like we have to check for read() [cat $filename] hang, and not statfs() [df -h $brick].  Will try to improve the script.

Meantime, process going to zombie state is because a systemcall from the process is not yet complete, and even -15 doesn't kill it all the time. We will also debug why even after closing main thread, we took 'ping-timeout' time to disconnect.

I ran into this bug with 3.3.1 when I started a replace-brick causing the read-from brick to be overloaded with read IOPS (about 2M directories in the volume). The entire gluster volume was inaccessible for several minutes.

Now //sbin/glusterfs -f/var/lib/glusterd/vols/vol01/rb_dst_brick.vol on the target node is eating up 100% CPU, using 0 IOPS and showing no strace output. At least the volume is accessible again, but the replace-brick is stuck.

If trying to reproduce the problem in a Amazon Web Services setup using EBS disks, one should be aware of the, kinda related, problem of failure handling of EBS disk on never kernels: https://forums.aws.amazon.com/thread.jspa?threadID=110756

Created attachment 745151 [details]
script for checking backend hang killing brick processes (modified)

Previous script bit modified. Please check whether this works or not.

Is this a duplicate of Bug 971774?

The version that this bug has been reported against, does not get any updates from the Gluster Community anymore. Please verify if this report is still valid against a current (3.4, 3.5 or 3.6) release and update the version, or close this bug.

If there has been no update before 9 December 2014, this bug will get automatocally closed.

This bug has been CLOSED, and there has not been a response to the requested NEEDINFO in more than 4 weeks. The NEEDINFO flag is now getting cleared so that our Bugzilla household is getting more in order.