762735 – (GLUSTER-1003) glusterfsd crash

Bug 762735 (GLUSTER-1003) - glusterfsd crash

Summary: glusterfsd crash

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	GLUSTER-1003
Product:	GlusterFS
Classification:	Community
Component:	core
Sub Component:
Version:	3.0.4
Hardware:	All
OS:	Linux
Priority:	low
Severity:	high
Target Milestone:	---
Assignee:	kaushik
QA Contact:
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	GLUSTER-1258 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2010-06-14 07:31 UTC by Dushyanth Harinath
Modified:	2015-12-01 16:45 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Dependent Products:

Attachments	(Terms of Use)
ZIP of vol files (3.23 KB, application/zip) 2010-06-14 04:31 UTC, Dushyanth Harinath	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Dushyanth Harinath 2010-06-14 04:31:53 UTC

Created attachment 232 [details]
Dutch keyboard definition w/dead keys for loadkeys

Comment 1 Dushyanth Harinath 2010-06-14 07:31:12 UTC

* Glusterfsd segfaulted and dropped a core file on / . Backtrace is at the END of this report
* This is a simple replication setup using ib-verbs 
* Volfiles are attached
* All performance translators are disabled
* Gluster mounts are mounted as below

# On mail1

/etc/glusterfs/glusterfs-mail1-mnt-home0.vol     /gluster/home0  glusterfs  defaults,_netdev,log-level=TRACE  0  0
/etc/glusterfs/glusterfs-mail1-mnt-althome0.vol  /gluster/alt-home0  glusterfs  defaults,_netdev,log-level=TRACE  0  0

# On mail2

/etc/glusterfs/glusterfs-mail2-mnt-home0.vol     /gluster/home0  glusterfs  defaults,_netdev,log-level=TRACE  0  0
/etc/glusterfs/glusterfs-mail2-mnt-althome0.vol  /gluster/alt-home0  glusterfs  defaults,_netdev,log-level=TRACE  0  0

* glusterfsd server log on mail2 only had the below


[2010-06-12 04:52:50] T [server-protocol.c:2471:server_lookup_cbk] mail2-ssd-server-ib-verbs: 676527: LOOKUP /d/di/directi.com/c/ch/chandresh.d/742/dbox/storage/dovecot.map.index.log.2 (0) ==> -1 (No such file or directory)
[2010-06-12 04:52:50] D [inode.c:927:inode_path] brick-ssd-mail2/inode: no dentry for non-root inode 3467616
pending frames:
frame : type(1) op(OPENDIR)
frame : type(1) op(OPENDIR)

* Client logs had the below

[2010-06-12 04:52:50] E [ib-verbs.c:2071:ib_verbs_event_handler] transport/ib-verbs: mail2-ssd-1: pollin received on tcp sock
et (peer: 122.182.1.153:6997) after handshake is complete
[2010-06-12 04:52:50] D [ib-verbs.c:1940:ib_verbs_handshake_pollerr] transport/ib-verbs: mail2-ssd-1: peer disconnected, clea
ning up
[2010-06-12 04:52:50] T [client-protocol.c:6374:protocol_client_cleanup] mail2-ssd-1: cleaning up state in transport object 0
x61f050
[2010-06-12 04:52:50] T [client-protocol.c:6319:client_protocol_reconnect] mail2-ssd-1: attempting reconnect
[2010-06-12 04:52:50] D [name.c:146:client_fill_address_family] mail2-ssd-1: address-family not specified, guessing it to be
inet/inet6
[2010-06-12 04:52:50] T [common-utils.c:100:gf_resolve_ip6] resolver: flushing DNS cache
[2010-06-12 04:52:50] T [common-utils.c:107:gf_resolve_ip6] resolver: DNS cache not present, freshly probing hostname: mail2
[2010-06-12 04:52:50] T [common-utils.c:148:gf_resolve_ip6] resolver: returning ip-122.182.1.153 (port-6997) for hostname: ma
il2 and port: 6997
[2010-06-12 04:52:50] T [ib-verbs.c:2153:ib_verbs_connect] mail2-ssd-1: socket fd = 14
[2010-06-12 04:52:50] E [ib-verbs.c:2071:ib_verbs_event_handler] transport/ib-verbs: mail2-ssd-1: pollin received on tcp sock
et (peer: 122.182.1.153:6997) after handshake is complete
[2010-06-12 04:52:50] D [ib-verbs.c:1940:ib_verbs_handshake_pollerr] transport/ib-verbs: mail2-ssd-1: peer disconnected, clea
ning up
[2010-06-12 04:52:50] T [client-protocol.c:6374:protocol_client_cleanup] mail2-ssd-1: cleaning up state in transport object 0
x61f550
[2010-06-12 04:52:50] E [saved-frames.c:165:saved_frames_unwind] mail2-ssd-1: forced unwinding frame type(1) op(LOOKUP)


GDB Backtrace 
--------------

[root@mail2 /]# gdb /usr/sbin/glusterfsd ./core.20185
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...
warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/glusterfsd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/glusterfsd.debug
done.
done.
[New Thread 20186]
[New Thread 20192]
[New Thread 20185]
[New Thread 20188]
[New Thread 20190]
[New Thread 20206]
[New Thread 20193]
[New Thread 20189]
[New Thread 20196]
[New Thread 20201]
[New Thread 20199]
[New Thread 20195]
[New Thread 20191]
[New Thread 20197]
[New Thread 20194]
[New Thread 20202]
[New Thread 20200]
[New Thread 20203]
[New Thread 20187]
[New Thread 20198]
[New Thread 20204]

warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/glusterfsd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/glusterfsd.debug
Reading symbols from /usr/lib64/libglusterfs.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libglusterfs.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libglusterfs.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/features/locks.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/features/locks.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/features/locks.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/transport/ib-verbs.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/transport/ib-verbs.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/transport/ib-verbs.so
Reading symbols from /usr/lib64/libibverbs.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libibverbs.so.1
Reading symbols from /usr/lib64/libnes-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnes-rdmav2.so
Reading symbols from /usr/lib64/libmlx4-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmlx4-rdmav2.so
Reading symbols from /usr/lib64/libcxgb3-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcxgb3-rdmav2.so
Reading symbols from /usr/lib64/libmthca-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmthca-rdmav2.so
Reading symbols from /usr/lib64/libipathverbs-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libipathverbs-rdmav2.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/glusterfs/3.0.4/auth/addr.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/auth/addr.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/auth/addr.so

warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/glusterfsd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/glusterfsd.debug
Core was generated by `/usr/sbin/glusterfsd -f /etc/glusterfs/glusterfsd.vol -L TRACE'.
Program terminated with signal 11, Segmentation fault.
#0  gf_print_trace (signum=11) at common-utils.c:413
413                             if ((tmp->root->type == GF_OP_TYPE_FOP_REQUEST) ||
(gdb) bt
#0  gf_print_trace (signum=11) at common-utils.c:413
#1  <signal handler called>
#2  0x00007ff6165233b3 in server_link_cbk (frame=0x7ff604e3edb8, cookie=<value optimized out>,
    this=<value optimized out>, op_ret=<value optimized out>, op_errno=0, inode=0x7ff5fc68e2f0, stbuf=0x437b0f90,
    preparent=0x437b0f00, postparent=0x437b0e70) at server-protocol.c:1599
#3  0x00007ff616734462 in iot_link_cbk (frame=0x7ff604726750, cookie=<value optimized out>, this=<value optimized out>,
    op_ret=0, op_errno=0, inode=0x7ff5fc68e2f0, buf=0x437b0f90, preparent=0x437b0f00, postparent=0x437b0e70)
    at io-threads.c:1538
#4  0x0000003cb921a3b5 in default_link_cbk (frame=0x7ff5f8912710, cookie=<value optimized out>,
    this=<value optimized out>, op_ret=0, op_errno=0, inode=0x7ff5fc68e2f0, buf=0x437b0f90, preparent=0x437b0f00,
    postparent=0x437b0e70) at defaults.c:436
#5  0x00007ff616b5ca29 in posix_link (frame=0x7ff5f88fff30, this=<value optimized out>, oldloc=0x7ff604e3fbb0,
    newloc=<value optimized out>) at posix.c:2096
#6  0x0000003cb921a2e6 in default_link (frame=<value optimized out>, this=0x60f340, oldloc=0x7ff604e3fbb0,
    newloc=0x7ff604e3fbd8) at defaults.c:447
#7  0x00007ff616736306 in iot_link_wrapper (frame=<value optimized out>, this=0x60fac0, old=0x7ff604e3fbb0,
    new=0x7ff604e3fbd8) at io-threads.c:1547
#8  0x0000003cb9222534 in call_resume_wind (stub=0x7ff604e3fb80) at call-stub.c:2449
#9  call_resume (stub=0x7ff604e3fb80) at call-stub.c:4304
#10 0x00007ff61673802e in iot_worker_unordered (arg=<value optimized out>) at io-threads.c:2453
#11 0x0000003327a06617 in start_thread () from /lib64/libpthread.so.0
#12 0x0000003326ed3c2d in clone () from /lib64/libc.so.6

Comment 2 Dushyanth Harinath 2010-06-28 06:47:34 UTC

Another recent crash with similar errors

# GDB Backtrace

[root@mail2 /]# gdb /usr/sbin/glusterfsd ./core.4093
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...
warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/glusterfsd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/glusterfsd.debug
done.
done.
[New Thread 4108]
[New Thread 4112]
[New Thread 4101]
[New Thread 4106]
[New Thread 4096]
[New Thread 4107]
[New Thread 4113]
[New Thread 4093]
[New Thread 4095]
[New Thread 4094]
[New Thread 4110]
[New Thread 4105]
[New Thread 4103]
[New Thread 4099]
[New Thread 4109]
[New Thread 4111]
[New Thread 4100]
[New Thread 4098]
[New Thread 4114]
[New Thread 4097]
[New Thread 4102]

warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/glusterfsd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/glusterfsd.debug
Reading symbols from /usr/lib64/libglusterfs.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libglusterfs.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libglusterfs.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/features/locks.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/features/locks.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/features/locks.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so
Reading symbols from /usr/lib64/glusterfs/3.0.4/transport/ib-verbs.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/transport/ib-verbs.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/transport/ib-verbs.so
Reading symbols from /usr/lib64/libibverbs.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libibverbs.so.1
Reading symbols from /usr/lib64/libnes-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnes-rdmav2.so
Reading symbols from /usr/lib64/libmlx4-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmlx4-rdmav2.so
Reading symbols from /usr/lib64/libcxgb3-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcxgb3-rdmav2.so
Reading symbols from /usr/lib64/libmthca-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmthca-rdmav2.so
Reading symbols from /usr/lib64/libipathverbs-rdmav2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libipathverbs-rdmav2.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/glusterfs/3.0.4/auth/addr.so...Reading symbols from /usr/lib/debug/usr/lib64/glusterfs/3.0.4/auth/addr.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/glusterfs/3.0.4/auth/addr.so
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1

warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/glusterfsd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/glusterfsd.debug
Core was generated by `/usr/sbin/glusterfsd -f /etc/glusterfs/glusterfsd.vol -L TRACE'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f4be5a163b3 in server_link_cbk (frame=0xb25158, cookie=<value optimized out>, this=<value optimized out>,
    op_ret=<value optimized out>, op_errno=0, inode=0x7f4bdc2e5be0, stbuf=0x42caff90, preparent=0x42caff00,
    postparent=0x42cafe70) at server-protocol.c:1599
1599                    gf_log (state->bound_xl->name, GF_LOG_TRACE,
(gdb) bt
#0  0x00007f4be5a163b3 in server_link_cbk (frame=0xb25158, cookie=<value optimized out>, this=<value optimized out>,
    op_ret=<value optimized out>, op_errno=0, inode=0x7f4bdc2e5be0, stbuf=0x42caff90, preparent=0x42caff00,
    postparent=0x42cafe70) at server-protocol.c:1599
#1  0x00007f4be5c27462 in iot_link_cbk (frame=0xa44770, cookie=<value optimized out>, this=<value optimized out>,
    op_ret=0, op_errno=0, inode=0x7f4bdc2e5be0, buf=0x42caff90, preparent=0x42caff00, postparent=0x42cafe70)
    at io-threads.c:1538
#2  0x0000003cb921a3b5 in default_link_cbk (frame=0x7f4bcc6e2820, cookie=<value optimized out>,
    this=<value optimized out>, op_ret=0, op_errno=0, inode=0x7f4bdc2e5be0, buf=0x42caff90, preparent=0x42caff00,
    postparent=0x42cafe70) at defaults.c:436
#3  0x00007f4be604fa29 in posix_link (frame=0x7f4bcc6f3500, this=<value optimized out>, oldloc=0xb2c720,
    newloc=<value optimized out>) at posix.c:2096
#4  0x0000003cb921a2e6 in default_link (frame=<value optimized out>, this=0x6112a0, oldloc=0xb2c720, newloc=0xb2c748)
    at defaults.c:447
#5  0x00007f4be5c29306 in iot_link_wrapper (frame=<value optimized out>, this=0x6113e0, old=0xb2c720, new=0xb2c748)
    at io-threads.c:1547
#6  0x0000003cb9222534 in call_resume_wind (stub=0xb2c6f0) at call-stub.c:2449
#7  call_resume (stub=0xb2c6f0) at call-stub.c:4304
#8  0x00007f4be5c2b02e in iot_worker_unordered (arg=<value optimized out>) at io-threads.c:2453
#9  0x0000003327a06617 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003326ed3c2d in clone () from /lib64/libc.so.6
(gdb)                                                          

# From glusterfsd log files

[2010-06-25 22:43:32] T [server-protocol.c:1415:server_rename_cbk] threads-home0: 3415482: RENAME_CBK (50469) 1/foo.2 ==> 1/foo
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] T [common.c:429:pl_inode_get] locks-home0: Allocating new pl inode
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=497
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bcc7f9c60
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=510
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bcc7edd10
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bdc26b430
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=494
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bcc7f9370
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=497
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] T [server-protocol.c:2471:server_lookup_cbk] brick-home0: 3415489: LOOKUP /foo.2 (0) ==> -1 (No suchfile or directory)
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bd89f1200
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=494
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bd89f04c0
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=497
[2010-06-25 22:43:32] T [server-protocol.c:2092:server_create_cbk] threads-home0: 3415490: CREATE 1/foo.2 (50468)
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=497
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bd89ef890
[2010-06-25 22:43:32] T [server-protocol.c:1465:server_unlink_cbk] threads-home0: 7386101: UNLINK_CBK 1/foo.3 (50471)
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0x7f4bdc2e4e80
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=510
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] T [server-protocol.c:2471:server_lookup_cbk] brick-home0: 3415495: LOOKUP /foo.3 (0) ==> -1 (No suchfile or directory)
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0xb2a170
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=497
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
pending frames:
frame : type(1) op(OPEN)
frame : type(1) op(LINK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0xa9a840
frame : type(1) op(LK)
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=510
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
frame : type(1) op(LK)
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0xb19c30
frame : type(1) op(LK)
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=512
frame : type(1) op(LK)

patchset: v3.0.4
signal received: 11
time of crash: 2010-06-25 22:43:32
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.0.4
[2010-06-25 22:43:32] T [posix.c:982:pl_release] locks-home0: Releasing all locks with fd 0xb2a170
[2010-06-25 22:43:32] T [posix.c:1454:posix_janitor_thread_proc] posix-home0: janitor: closing file fd=497
[2010-06-25 22:43:32] D [inode.c:927:inode_path] threads-home0/inode: no dentry for non-root inode 50469
/lib64/libc.so.6[0x3326e302d0]
/usr/lib64/glusterfs/3.0.4/xlator/protocol/server.so(server_link_cbk+0x4a3)[0x7f4be5a163b3]
/usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so(iot_link_cbk+0x92)[0x7f4be5c27462]
/usr/lib64/libglusterfs.so.0[0x3cb921a3b5]
/usr/lib64/glusterfs/3.0.4/xlator/storage/posix.so(posix_link+0x369)[0x7f4be604fa29]
/usr/lib64/libglusterfs.so.0(default_link+0xd6)[0x3cb921a2e6]
/usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so(iot_link_wrapper+0xd6)[0x7f4be5c29306]
/usr/lib64/libglusterfs.so.0(call_resume+0x254)[0x3cb9222534]
/usr/lib64/glusterfs/3.0.4/xlator/performance/io-threads.so(iot_worker_unordered+0xe)[0x7f4be5c2b02e]
/lib64/libpthread.so.0[0x3327a06617]
/lib64/libc.so.6(clone+0x6d)[0x3326ed3c2d]
---------

Comment 3 Amar Tumballi 2010-09-13 04:06:58 UTC

*** Bug 1258 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.