This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 832396 - nfs server asserted in inode_path since gfid of the inode is NULL
nfs server asserted in inode_path since gfid of the inode is NULL
Status: CLOSED DUPLICATE of bug 826080
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
3.3.0
Unspecified Unspecified
high Severity unspecified
: ---
: ---
Assigned To: Rajesh
: Triaged
Depends On:
Blocks: 852564 858481
  Show dependency treegraph
 
Reported: 2012-06-15 06:13 EDT by Raghavendra Bhat
Modified: 2013-07-04 18:44 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 852564 (view as bug list)
Environment:
Last Closed: 2012-11-20 02:41:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Raghavendra Bhat 2012-06-15 06:13:38 EDT
Description of problem:

2 replica volume with quota enabled, gsync running. 4 fuse and 4 nfs clients running different tests such as rdd, sanity script, ping_pong, million file creation, large file creation (100GB) etc.

added 2 more bricks (2x2 now).

nfs server crashed with below bt.

Cannot find new threads: generic error
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.5.x86_64 libgcc-4.4.6-3.el6.x86_64 openssl-1.0.0-20.el6_2.3.x86_64 valgrind-3.6.0-4.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb)  bt
#0  0x0000003c380dc273 in poll () from /lib64/libc.so.6
#1  0x0000003c38112d60 in svc_run () from /lib64/libc.so.6
#2  0x0000000009f3b933 in nsm_thread (argv=0x0) at ../../../../../xlators/nfs/server/src/nlmcbk_svc.c:118
#3  0x0000003c388077f1 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c380e592d in clone () from /lib64/libc.so.6
(gdb) t 2
[Switching to thread 2 (Thread 6695)]#0  0x0000003c3880eccd in nanosleep () from /lib64/libpthread.so.0
(gdb) t 3
[Switching to thread 3 (Thread 6693)]#0  0x0000003c3880b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) t 4
[Switching to thread 4 (Thread 6692)]#0  0x0000003c3880b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) t 5
[Switching to thread 5 (Thread 6691)]#0  0x0000003c3880f245 in sigwait () from /lib64/libpthread.so.0
(gdb) t 6
[Switching to thread 6 (Thread 6690)]#0  0x0000003c38032885 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003c38032885 in raise () from /lib64/libc.so.6
#1  0x0000003c38034065 in abort () from /lib64/libc.so.6
#2  0x0000003c3802b9fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x0000003c3802bac0 in __assert_fail () from /lib64/libc.so.6
#4  0x0000000004c40d6e in __inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1090
#5  0x0000000004c41156 in inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1191
#6  0x00000000093bacec in protocol_client_reopendir (this=0x66736e0, fdctx=0xda06c20)
    at ../../../../../xlators/protocol/client/src/client-handshake.c:1104
#7  0x00000000093bb527 in client_post_handshake (frame=0x542d234, this=0x66736e0)
    at ../../../../../xlators/protocol/client/src/client-handshake.c:1289
#8  0x00000000093bbd67 in client_setvolume_cbk (req=0xd3de488, iov=0xd3de4c8, count=1, myframe=0x542d234)
    at ../../../../../xlators/protocol/client/src/client-handshake.c:1447
#9  0x0000000004eafa48 in rpc_clnt_handle_reply (clnt=0xa4bc2f0, pollin=0xe6c28a0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:788
#10 0x0000000004eafde5 in rpc_clnt_notify (trans=0x672c5c0, mydata=0xa4bc320, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe6c28a0)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:907
#11 0x0000000004eabec8 in rpc_transport_notify (this=0x672c5c0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0xe6c28a0)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:489
#12 0x0000000008555280 in socket_event_poll_in (this=0x672c5c0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1677
#13 0x0000000008555804 in socket_event_handler (fd=15, idx=11, data=0x672c5c0, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1792
#14 0x0000000004c59d9c in event_dispatch_epoll_handler (event_pool=0x5306e60, events=0x666c320, i=0)
    at ../../../libglusterfs/src/event.c:785
#15 0x0000000004c59fbf in event_dispatch_epoll (event_pool=0x5306e60) at ../../../libglusterfs/src/event.c:847
#16 0x0000000004c5a34a in event_dispatch (event_pool=0x5306e60) at ../../../libglusterfs/src/event.c:947
#17 0x0000000000408567 in main (argc=11, argv=0x7fefffed8) at ../../../glusterfsd/src/glusterfsd.c:1689
(gdb) f 4
#4  0x0000000004c40d6e in __inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1090
1090                    GF_ASSERT (0);
(gdb) l
1085            int64_t        ret   = 0;
1086            int            len   = 0;
1087            char          *buf   = NULL;
1088
1089            if (!inode || uuid_is_null (inode->gfid)) {
1090                    GF_ASSERT (0);
1091                    gf_log_callingfn (THIS->name, GF_LOG_WARNING, "invalid inode");
1092                    return -1;
1093            }
1094
(gdb) p *inode
$1 = {table = 0x56e8580, gfid = '\000' <repeats 15 times>, lock = 1, nlookup = 0, ref = 12, ia_type = IA_IFDIR, fd_list = {
    next = 0xa552830, prev = 0xa552830}, dentry_list = {next = 0xa552840, prev = 0xa552840}, hash = {next = 0xa552850, 
    prev = 0xa552850}, list = {next = 0xa5520dc, prev = 0x56e85e0}, _ctx = 0xda00390}
(gdb) up
#5  0x0000000004c41156 in inode_path (inode=0xa552800, name=0x0, bufp=0x7fefff818) at ../../../libglusterfs/src/inode.c:1191
1191                    ret = __inode_path (inode, name, bufp);
(gdb) l
1186
1187            table = inode->table;
1188
1189            pthread_mutex_lock (&table->lock);
1190            {
1191                    ret = __inode_path (inode, name, bufp);
1192            }
1193            pthread_mutex_unlock (&table->lock);
1194
1195            return ret;
(gdb) p name
$2 = 0x0
(gdb) up
#6  0x00000000093bacec in protocol_client_reopendir (this=0x66736e0, fdctx=0xda06c20)
    at ../../../../../xlators/protocol/client/src/client-handshake.c:1104
1104            ret = inode_path (inode, NULL, &path);
(gdb) l
1099                    goto out;
1100
1101            inode = fdctx->inode;
1102            conf = this->private;
1103
1104            ret = inode_path (inode, NULL, &path);
1105            if (ret < 0) {
1106                    gf_log (this->name, GF_LOG_WARNING,
1107                            "couldn't build path from inode %s",
1108                            uuid_utoa (inode->gfid));
(gdb) p *fdctx
$3 = {sfd_pos = {next = 0xda06c20, prev = 0xda06c20}, remote_fd = -1, inode = 0xa552800, ino = 0, gen = 0, is_dir = 1 '\001', 
  released = 0 '\000', flags = 0, wbflags = 0, lk_ctx = 0x0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
      __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, 
  lk_heal_state = GF_LK_HEAL_IN_PROGRESS, lock_list = {next = 0xda06c98, prev = 0xda06c98}}
(gdb) 






Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

gluster volume info
 
Volume Name: mirror
Type: Distributed-Replicate
Volume ID: 06827332-81d4-42f0-a81f-83518286ba3b
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.16.156.9:/export/mirror
Brick2: 10.16.156.12:/export/mirror
Brick3: 10.16.156.15:/export/mirror
Brick4: 10.16.156.18:/export/mirror
Options Reconfigured:
network.ping-timeout: 88
geo-replication.indexing: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
features.limit-usage: /:1TB
features.quota: on
[root@gqas007 ~]# 


e-d9f2-4ded-8190-5a6f7dc088dc
[2012-06-13 05:34:34.371543] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7126a341, ACCESS: NFS: 22(Invalid argument 
for operation), POSIX: 14(Bad address)
[2012-06-13 05:34:34.399741] I [dht-layout.c:593:dht_layout_normalize] 0-mirror-dht: found anomalies in <gfid:6a351afe-d9f2-4ded-8190-
5a6f7dc088dc>. holes=1 overlaps=0
[2012-06-13 05:34:34.400185] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:6a351afe-d9f2-
4ded-8190-5a6f7dc088dc>: Invalid argument
[2012-06-13 05:34:34.400589] E [nfs3.c:1513:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.16.157.21:908) mirror : 6a351af
e-d9f2-4ded-8190-5a6f7dc088dc
[2012-06-13 05:34:34.400955] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7226a341, ACCESS: NFS: 22(Invalid argument 
for operation), POSIX: 14(Bad address)
[2012-06-13 05:34:34.429187] I [dht-layout.c:593:dht_layout_normalize] 0-mirror-dht: found anomalies in <gfid:6a351afe-d9f2-4ded-8190-
5a6f7dc088dc>. holes=1 overlaps=0
[2012-06-13 05:34:34.429581] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:6a351afe-d9f2-
4ded-8190-5a6f7dc088dc>: Invalid argument
[2012-06-13 05:34:34.429989] E [nfs3.c:1513:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (10.16.157.21:908) mirror : 6a351af
e-d9f2-4ded-8190-5a6f7dc088dc
[2012-06-13 05:34:34.430375] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7326a341, ACCESS: NFS: 22(Invalid argument 
for operation), POSIX: 14(Bad address)
[2012-06-13 05:34:34.439696] I [client-handshake.c:1636:select_server_supported_programs] 0-mirror-client-0: Using Program GlusterFS 3
.3.0, Num (1298437), Version (330)
[2012-06-13 05:34:34.446028] I [client-handshake.c:1433:client_setvolume_cbk] 0-mirror-client-0: Connected to 10.16.156.9:24009, attac
hed to remote volume '/export/mirror'.
[2012-06-13 05:34:34.446219] I [client-handshake.c:1445:client_setvolume_cbk] 0-mirror-client-0: Server and Client lk-version numbers 
are not same, reopening the fds
[2012-06-13 05:34:34.447715] I [client-handshake.c:1282:client_post_handshake] 0-mirror-client-0: 1 fds open - Delaying child_up until
 they are re-opened
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 2012-06-13 05:34:34
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
:
Comment 1 Vijay Bellur 2012-11-20 02:41:24 EST

*** This bug has been marked as a duplicate of bug 826080 ***

Note You need to log in before you can comment on or make changes to this bug.