Bug 812830

Summary: "NFS server not responding still trying" under heavy load with Solaris client
Product: [Community] GlusterFS Reporter: Anthony <sokar6012>
Component: nfsAssignee: Vinayaga Raman <vraman>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 3.2.5CC: gluster-bugs, rwheeler, sokar6012, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Solaris   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-23 07:27:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Anthony 2012-04-16 11:20:45 UTC 
Description of problem:

On Solaris Client (SunOS 5.10), under heavy load (lot of folders and files) , the mount point seems to disconnect
mount -o proto=tcp,vers=3,llock,hard,bg,timeo=600,intr,rsize=32768,wsize=32768 nfs://ylal3510/poolsave /users/glusterfs_mnt

glusterfs_mnt>ls
yval8250  yval9000  yvasj200  yvasj410 

:/users/glusterfs_mnt>cp -pr yval9000 testnfs/
NFS server ylal3510 not responding still trying  <= after a while

glusterfs_mnt>ls
testnfs   yval8250  yval9000  yvasj200  yvasj410    <= strange because everything seems ok, mount point still here

/users/glusterfs_mnt>rm -r testnfs
NFS server ylal3510 not responding still trying    <= Instantly 


nfs server crash
nfs.log :

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-04-16 11:57:49
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2.5
/lib64/libc.so.6(+0x329e0)[0x7fa9f23cd9e0]
/usr/local/lib//glusterfs/3.2.5/xlator/nfs/server.so(nfs_fop_local_wipe+0x13)[0x7fa9f028c163]
/usr/local/lib//glusterfs/3.2.5/xlator/nfs/server.so(nfs_fop_lookup_cbk+0x6f)[0x7fa9f028c28f]
/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_lookup_cbk+0xe4)[0x7fa9f04ea1c4]
/usr/local/lib//glusterfs/3.2.5/xlator/performance/quick-read.so(qr_lookup_cbk+0x1cd)[0x7fa9f06003ad]
/usr/local/lib//glusterfs/3.2.5/xlator/performance/io-cache.so(ioc_lookup_cbk+0x32e)[0x7fa9f0712a7e]
/usr/local/lib/libglusterfs.so.0(default_lookup_cbk+0xaa)[0x7fa9f2e27bfa]
---------


log/glusterfs/bricks/users3-poolsave.log

[2012-04-16 11:57:50.135294] W [socket.c:1494:__socket_proto_state_machine] 0-tcp.poolsave-server: reading from socket failed. Error (Transport endpoint is not connected), peer (10.80.49.10:1018)
[2012-04-16 11:57:50.135677] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -1359956227: 25f12aab-4a34-4e78-9c59-1ba212979b5b
[2012-04-16 11:57:50.135735] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -1359956227 with gfid 25f12aab-4a34-4e78-9c59-1ba212979b5b
[2012-04-16 11:57:50.135854] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -2689468103: 26cdb4e9-decb-413f-8558-5dcbda599b6e
[2012-04-16 11:57:50.135905] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -2689468103 with gfid 26cdb4e9-decb-413f-8558-5dcbda599b6e
[2012-04-16 11:57:50.136010] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -327950645: b0a40e33-b6bd-43a9-9b76-d94f50fd057c
[2012-04-16 11:57:50.166144] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -327950645 with gfid b0a40e33-b6bd-43a9-9b76-d94f50fd057c
[2012-04-16 11:57:50.174292] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -482216456: 9714189f-9f68-4423-85a2-ee015d9f07f7
[2012-04-16 11:57:50.198087] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -482216456 with gfid 9714189f-9f68-4423-85a2-ee015d9f07f7
[2012-04-16 11:57:50.210125] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -2192790751: 8e186818-5b3c-4e5d-ac05-4b07d147601a
[2012-04-16 11:57:50.222081] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -2192790751 with gfid 8e186818-5b3c-4e5d-ac05-4b07d147601a
[2012-04-16 11:57:50.222165] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -3420884929: 7b3e9639-bbd3-4c1c-b23e-8d7e81dae2c1
[2012-04-16 11:57:50.258088] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -3420884929 with gfid 7b3e9639-bbd3-4c1c-b23e-8d7e81dae2c1
[2012-04-16 11:57:50.282111] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -2218088309: 3244d97a-a326-43cb-b48b-ab8ac73efd01
[2012-04-16 11:57:50.302253] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -2218088309 with gfid 3244d97a-a326-43cb-b48b-ab8ac73efd01
[2012-04-16 11:57:50.326113] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -1481572864: 132bb66c-4f8e-4a81-acef-c673fac1378d
[2012-04-16 11:57:50.350081] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -1481572864 with gfid 132bb66c-4f8e-4a81-acef-c673fac1378d
[2012-04-16 11:57:50.374143] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -3383438248: f62070f4-8788-4ff5-89f4-8106ac605352
[2012-04-16 11:57:50.386090] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -3383438248 with gfid f62070f4-8788-4ff5-89f4-8106ac605352
[2012-04-16 11:57:50.398124] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -2784310458: 8c73927e-aca7-4b81-ab42-bdebaec80d5b
[2012-04-16 11:57:50.446087] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -2784310458 with gfid 8c73927e-aca7-4b81-ab42-bdebaec80d5b
[2012-04-16 11:57:50.470112] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -1511133263: 6aa3609c-204d-452c-b0ad-00c5f540eeec
[2012-04-16 11:57:50.494081] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -1511133263 with gfid 6aa3609c-204d-452c-b0ad-00c5f540eeec
[2012-04-16 11:57:50.506137] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -1858839369: a166b06b-5ab1-4e7e-8d46-0f4e03ee4d69
[2012-04-16 11:57:50.518087] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -1858839369 with gfid a166b06b-5ab1-4e7e-8d46-0f4e03ee4d69
[2012-04-16 11:57:50.518128] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f1c0b2b4106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f1c0b2b3e09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f1c0b2b2b6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode 296113869: 57085bd6-9a39-4277-938b-36c17e1b200c
[2012-04-16 11:57:50.542086] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino 296113869 with gfid 57085bd6-9a39-4277-938b-36c17e1b200c
Comment 1 Anthony 2012-07-23 07:27:53 UTC 
Fixed in 3.3