Bug 761830 (GLUSTER-98)

I havent seen this before but there is a hang in glusterfs_init that has suddenly surfaced in booster. The client is waiting for a reply from the server.
This is happening on the head of release-2.0 branch.

IMP: I consider this a blocker because we need to fork off a 2.0.3 from here. Know that 2.0.3 is a unfs3booster specific release so this is a BLOCKER.

Below is the bt from gdb of a test program that hangs in glusterfs_init.

 
root@indus:glusterfs# /usr/local/sbin/glusterfsd  -f /home/shehjart/docs/volfiles/localdisk-iot.vol  -l /test/srvlog -L DEBUG  --no-daemon

root@indus:glusterfs# /usr/local/sbin/glusterfsd  -f /home/shehjart/docs/volfiles/localdisk-iot.vol  -l /test/srvlog -L DEBUG  
root@indus:glusterfs# ps ax|
root@indus:glusterfs# gdb -p $(pidof fstatcache)
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Attaching to process 28282
Reading symbols from /home/shehjart/code/tools-git/statcache/fstatcache...done.
Reading symbols from /usr/local/lib/glusterfs/glusterfs-booster.so...done.
Loaded symbols for /usr/local/lib/glusterfs/glusterfs-booster.so
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
Loaded symbols for /usr/local/lib/libglusterfs.so.0
Reading symbols from /usr/local/lib/libglusterfsclient.so.0...done.
Loaded symbols for /usr/local/lib/libglusterfsclient.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2aca71e1edd0 (LWP 28282)]
[New Thread 0x2aca73f51950 (LWP 28284)]
[New Thread 0x2aca73750950 (LWP 28283)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/glusterfs/2.0.0git/xlator/protocol/client.so...done.
Loaded symbols for /usr/local/lib/glusterfs/2.0.0git/xlator/protocol/client.so
Reading symbols from /usr/local/lib/glusterfs/2.0.0git/xlator/performance/write-behind.so...done.
Loaded symbols for /usr/local/lib/glusterfs/2.0.0git/xlator/performance/write-behind.so
Reading symbols from /usr/local/lib/glusterfs/2.0.0git/transport/socket.so...done.
Loaded symbols for /usr/local/lib/glusterfs/2.0.0git/transport/socket.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
0x00002aca71c0dc09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) c
Continuing.

Program received signal SIGINT, Interrupt.
[Switching to Thread 0x2aca71e1edd0 (LWP 28282)]
0x00002aca71c0dc09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) bt
#0  0x00002aca71c0dc09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x00002aca717e6e80 in glusterfs_init (init_ctx=0x7fff39ca4c40) at libglusterfsclient.c:1205
#2  0x00002aca717e7464 in glusterfs_mount (vmp=0x60351c "/testpath", ipars=0x7fff39ca4c40)
    at libglusterfsclient.c:1377
#3  0x00002aca7102b183 in booster_mount (ent=0x6038e8) at booster_fstab.c:405
#4  0x00002aca7102b1de in booster_configure (
    confpath=0x7fff39ca5e0a "/home/shehjart/code/tools-git/statcache/booster.fstab")
    at booster_fstab.c:424
#5  0x00002aca71027b70 in booster_init () at booster.c:1221
#6  0x00002aca71029df8 in booster_lib_init () at booster.c:2313
#7  0x00002aca710284c2 in opendir (path=0x7fff39ca5936 "/testpath") at booster.c:1550
#8  0x0000000000400aa1 in traverse_dir (path=0x7fff39ca5936 "/testpath") at fstatcache.c:27
#9  0x0000000000400c51 in stat_cache_test (path=0x7fff39ca5936 "/testpath") at fstatcache.c:71
#10 0x0000000000400dcf in main (argc=5, argv=0x7fff39ca53b8) at fstatcache.c:111