Bug 832396
| Summary: | nfs server asserted in inode_path since gfid of the inode is NULL | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra Bhat <rabhat> | |
| Component: | nfs | Assignee: | Rajesh <rajesh> | |
| Status: | CLOSED DUPLICATE | QA Contact: | ||
| Severity: | unspecified | Docs Contact: | ||
| Priority: | high | |||
| Version: | 3.3.0 | CC: | gluster-bugs, rajesh, vagarwal, vbellur | |
| Target Milestone: | --- | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 852564 (view as bug list) | Environment: | ||
| Last Closed: | 2012-11-20 07:41:24 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 852564, 858481 | |||
*** This bug has been marked as a duplicate of bug 826080 *** |
Description of problem: 2 replica volume with quota enabled, gsync running. 4 fuse and 4 nfs clients running different tests such as rdd, sanity script, ping_pong, million file creation, large file creation (100GB) etc. added 2 more bricks (2x2 now). nfs server crashed with below bt. Cannot find new threads: generic error Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.5.x86_64 libgcc-4.4.6-3.el6.x86_64 openssl-1.0.0-20.el6_2.3.x86_64 valgrind-3.6.0-4.el6.x86_64 zlib-1.2.3-27.el6.x86_64 (gdb) bt #0 0x0000003c380dc273 in poll () from /lib64/libc.so.6 #1 0x0000003c38112d60 in svc_run () from /lib64/libc.so.6 #2 0x0000000009f3b933 in nsm_thread (argv=0x0) at ../../../../../xlators/nfs/server/src/nlmcbk_svc.c:118 #3 0x0000003c388077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003c380e592d in clone () from /lib64/libc.so.6 (gdb) t 2 [Switching to thread 2 (Thread 6695)]#0 0x0000003c3880eccd in nanosleep () from /lib64/libpthread.so.0 (gdb) t 3 [Switching to thread 3 (Thread 6693)]#0 0x0000003c3880b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) t 4 [Switching to thread 4 (Thread 6692)]#0 0x0000003c3880b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) t 5 [Switching to thread 5 (Thread 6691)]#0 0x0000003c3880f245 in sigwait () from /lib64/libpthread.so.0 (gdb) t 6 [Switching to thread 6 (Thread 6690)]#0 0x0000003c38032885 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003c38032885 in raise () from /lib64/libc.so.6 #1 0x0000003c38034065 in abort () from /lib64/libc.so.6 #2 0x0000003c3802b9fe in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003c3802bac0 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000004c40d6e in __inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1090 #5 0x0000000004c41156 in inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1191 #6 0x00000000093bacec in protocol_client_reopendir (this=0x66736e0, fdctx=0xda06c20) at ../../../../../xlators/protocol/client/src/client-handshake.c:1104 #7 0x00000000093bb527 in client_post_handshake (frame=0x542d234, this=0x66736e0) at ../../../../../xlators/protocol/client/src/client-handshake.c:1289 #8 0x00000000093bbd67 in client_setvolume_cbk (req=0xd3de488, iov=0xd3de4c8, count=1, myframe=0x542d234) at ../../../../../xlators/protocol/client/src/client-handshake.c:1447 #9 0x0000000004eafa48 in rpc_clnt_handle_reply (clnt=0xa4bc2f0, pollin=0xe6c28a0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:788 #10 0x0000000004eafde5 in rpc_clnt_notify (trans=0x672c5c0, mydata=0xa4bc320, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe6c28a0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:907 #11 0x0000000004eabec8 in rpc_transport_notify (this=0x672c5c0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe6c28a0) at ../../../../rpc/rpc-lib/src/rpc-transport.c:489 #12 0x0000000008555280 in socket_event_poll_in (this=0x672c5c0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1677 #13 0x0000000008555804 in socket_event_handler (fd=15, idx=11, data=0x672c5c0, poll_in=1, poll_out=0, poll_err=0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1792 #14 0x0000000004c59d9c in event_dispatch_epoll_handler (event_pool=0x5306e60, events=0x666c320, i=0) at ../../../libglusterfs/src/event.c:785 #15 0x0000000004c59fbf in event_dispatch_epoll (event_pool=0x5306e60) at ../../../libglusterfs/src/event.c:847 #16 0x0000000004c5a34a in event_dispatch (event_pool=0x5306e60) at ../../../libglusterfs/src/event.c:947 #17 0x0000000000408567 in main (argc=11, argv=0x7fefffed8) at ../../../glusterfsd/src/glusterfsd.c:1689 (gdb) f 4 #4 0x0000000004c40d6e in __inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1090 1090 GF_ASSERT (0); (gdb) l 1085 int64_t ret = 0; 1086 int len = 0; 1087 char *buf = NULL; 1088 1089 if (!inode || uuid_is_null (inode->gfid)) { 1090 GF_ASSERT (0); 1091 gf_log_callingfn (THIS->name, GF_LOG_WARNING, "invalid inode"); 1092 return -1; 1093 } 1094 (gdb) p *inode $1 = {table = 0x56e8580, gfid = '\000' <repeats 15 times>, lock = 1, nlookup = 0, ref = 12, ia_type = IA_IFDIR, fd_list = { next = 0xa552830, prev = 0xa552830}, dentry_list = {next = 0xa552840, prev = 0xa552840}, hash = {next = 0xa552850, prev = 0xa552850}, list = {next = 0xa5520dc, prev = 0x56e85e0}, _ctx = 0xda00390} (gdb) up #5 0x0000000004c41156 in inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1191 1191 ret = __inode_path (inode, name, bufp); (gdb) l 1186 1187 table = inode->table; 1188 1189 pthread_mutex_lock (&table->lock); 1190 { 1191 ret = __inode_path (inode, name, bufp); 1192 } 1193 pthread_mutex_unlock (&table->lock); 1194 1195 return ret; (gdb) p name $2 = 0x0 (gdb) up #6 0x00000000093bacec in protocol_client_reopendir (this=0x66736e0, fdctx=0xda06c20) at ../../../../../xlators/protocol/client/src/client-handshake.c:1104 1104 ret = inode_path (inode, NULL, &path); (gdb) l 1099 goto out; 1100 1101 inode = fdctx->inode; 1102 conf = this->private; 1103 1104 ret = inode_path (inode, NULL, &path); 1105 if (ret < 0) { 1106 gf_log (this->name, GF_LOG_WARNING, 1107 "couldn't build path from inode %s", 1108 uuid_utoa (inode->gfid)); (gdb) p *fdctx $3 = {sfd_pos = {next = 0xda06c20, prev = 0xda06c20}, remote_fd = -1, inode = 0xa552800, ino = 0, gen = 0, is_dir = 1 '\001', released = 0 '\000', flags = 0, wbflags = 0, lk_ctx = 0x0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, lk_heal_state = GF_LK_HEAL_IN_PROGRESS, lock_list = {next = 0xda06c98, prev = 0xda06c98}} (gdb) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: gluster volume info Volume Name: mirror Type: Distributed-Replicate Volume ID: 06827332-81d4-42f0-a81f-83518286ba3b Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.16.156.9:/export/mirror Brick2: 10.16.156.12:/export/mirror Brick3: 10.16.156.15:/export/mirror Brick4: 10.16.156.18:/export/mirror Options Reconfigured: network.ping-timeout: 88 geo-replication.indexing: on diagnostics.count-fop-hits: on diagnostics.latency-measurement: on features.limit-usage: /:1TB features.quota: on [root@gqas007 ~]# e-d9f2-4ded-8190-5a6f7dc088dc [2012-06-13 05:34:34.371543] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7126a341, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2012-06-13 05:34:34.399741] I [dht-layout.c:593:dht_layout_normalize] 0-mirror-dht: found anomalies in <gfid:6a351afe-d9f2-4ded-8190- 5a6f7dc088dc>. holes=1 overlaps=0 [2012-06-13 05:34:34.400185] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:6a351afe-d9f2- 4ded-8190-5a6f7dc088dc>: Invalid argument [2012-06-13 05:34:34.400589] E [nfs3.c:1513:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.16.157.21:908) mirror : 6a351af e-d9f2-4ded-8190-5a6f7dc088dc [2012-06-13 05:34:34.400955] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7226a341, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2012-06-13 05:34:34.429187] I [dht-layout.c:593:dht_layout_normalize] 0-mirror-dht: found anomalies in <gfid:6a351afe-d9f2-4ded-8190- 5a6f7dc088dc>. holes=1 overlaps=0 [2012-06-13 05:34:34.429581] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:6a351afe-d9f2- 4ded-8190-5a6f7dc088dc>: Invalid argument [2012-06-13 05:34:34.429989] E [nfs3.c:1513:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.16.157.21:908) mirror : 6a351af e-d9f2-4ded-8190-5a6f7dc088dc [2012-06-13 05:34:34.430375] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7326a341, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2012-06-13 05:34:34.439696] I [client-handshake.c:1636:select_server_supported_programs] 0-mirror-client-0: Using Program GlusterFS 3 .3.0, Num (1298437), Version (330) [2012-06-13 05:34:34.446028] I [client-handshake.c:1433:client_setvolume_cbk] 0-mirror-client-0: Connected to 10.16.156.9:24009, attac hed to remote volume '/export/mirror'. [2012-06-13 05:34:34.446219] I [client-handshake.c:1445:client_setvolume_cbk] 0-mirror-client-0: Server and Client lk-version numbers are not same, reopening the fds [2012-06-13 05:34:34.447715] I [client-handshake.c:1282:client_post_handshake] 0-mirror-client-0: 1 fds open - Delaying child_up until they are re-opened pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 6 time of crash: 2012-06-13 05:34:34 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 :