Bug 762340 (GLUSTER-608)

The glusterfs server crashed while doing read opearation. The setup was distribute. error-gen was used with ENOMEM. It was observed in 2.0.10rc1 on my local machine. This is the backtrace of the core generated.


[?1034hGNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-slackware-linux"...

warning: core file may not match specified executable file.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/libglusterfs.so.0...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/libglusterfs.so.0
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/storage/posix.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/storage/posix.so
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/debug/error-gen.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/debug/error-gen.so
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/features/locks.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/features/locks.so
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/performance/io-threads.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/performance/io-threads.so
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/protocol/server.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/xlator/protocol/server.so
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/transport/socket.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/transport/socket.so
Reading symbols from /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/auth/addr.so...done.
Loaded symbols for /opt/glusterfs/2.0.10rc1/lib/glusterfs/2.0.10rc1/auth/addr.so
Reading symbols from /usr/lib64/libgcc_s.so.1...done.
Loaded symbols for /usr/lib64/libgcc_s.so.1
Core was generated by `/opt/glusterfs/2.0.10rc1/sbin/glusterfsd -f error-export.vol -l /tmp/error.log'.
Program terminated with signal 11, Segmentation fault.
[New process 5200]
[New process 5199]
[New process 5201]
[New process 5202]
[New process 5204]
[New process 5205]
[New process 5206]
[New process 5207]
[New process 5208]
[New process 5209]
[New process 5210]
[New process 5212]
[New process 5213]
[New process 5214]
[New process 5215]
[New process 5216]
[New process 5217]
[New process 5219]
[New process 5220]
[New process 5221]
[New process 5222]
[New process 5223]
[New process 5190]
[New process 5194]
[New process 5195]
[New process 5196]
[New process 5197]
[New process 5198]
[New process 5203]
[New process 5211]
[New process 5218]
[New process 5191]
[New process 5192]
[New process 5193]
#0  0x00007f1f0d275aa0 in pthread_spin_lock () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f1f0d275aa0 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007f1f0d6b4fe7 in iobuf_unref (iobuf=0x20000) at iobuf.c:432
#2  0x00007f1f0d6b5068 in iobref_destroy (iobref=0x7f1f0ad97fd0) at iobuf.c:507
#3  0x00007f1f0937a02d in __socket_ioq_entry_free (entry=0x62a540) at socket.c:329
#4  0x00007f1f0937b839 in __socket_ioq_churn_entry (this=<value optimized out>, entry=0x62a540) at socket.c:369
#5  0x00007f1f0937b98b in socket_submit (this=0x61dd50, buf=0x62af30 "", len=108, vector=<value optimized out>, count=0, 
    iobref=<value optimized out>) at socket.c:1305
#6  0x00007f1f0bca8a7c in protocol_server_reply (frame=0x7f1efc00b910, type=4, op=13, hdr=0x62af30, hdrlen=108, vector=0x0, count=0, 
    iobref=0x7f1f0ad97fd0) at server-protocol.c:64
#7  0x00007f1f0bcb4bd7 in server_readv_cbk (frame=0x7f1efc00b910, cookie=<value optimized out>, this=0x611930, op_ret=-1, 
    op_errno=107, vector=0x0, count=0, stbuf=0x0, iobref=0x7f1f0ad97fd0) at server-protocol.c:1974
#8  0x00007f1f0bec7b2c in iot_readv_cbk (frame=<value optimized out>, cookie=<value optimized out>, this=<value optimized out>, 
    op_ret=0, op_errno=6465136, vector=0x1, count=0, stbuf=0x0, iobref=0x7f1f0ad97fd0) at io-threads.c:784
#9  0x00007f1f0c0d51fc in pl_readv_cbk (frame=<value optimized out>, cookie=<value optimized out>, this=<value optimized out>, 
    op_ret=0, op_errno=6465136, vector=0x1, count=0, stbuf=0x0, iobref=0x7f1f0ad97fd0) at posix.c:360
#10 0x00007f1f0c2e0ec9 in error_gen_readv (frame=0x62a1f0, this=0x60fea0, fd=0x7f1efc00be40, size=131072, offset=524288)
    at error-gen.c:821
#11 0x00007f1f0c0d6bbd in pl_readv (frame=0x620b30, this=0x6101b0, fd=0x7f1efc00be40, size=0, offset=1) at posix.c:506
#12 0x00007f1f0beca120 in iot_readv_wrapper (frame=0x7f1efc008210, this=0x610380, fd=0x7f1efc00be40, size=131072, offset=524288)
    at io-threads.c:794
#13 0x00007f1f0d6a89bf in call_resume (stub=0x7f1efc00f610) at call-stub.c:2399
#14 0x00007f1f0bec8b08 in iot_worker_ordered (arg=<value optimized out>) at io-threads.c:1943
#15 0x00007f1f0d2713a0 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f1f0cfdebdd in clone () from /lib64/libc.so.6
(gdb) 



script containing the tests was running during the time of crash.