Hide Forgot
The test was done with ENOSPC, ENOENT and EIO. During those tests one of the servers crashed. With ENOMEM all the servers crashed.
(In reply to comment #1) > The test was done with ENOSPC, ENOENT and EIO. During those tests one of the > servers crashed. With ENOMEM all the servers crashed. With EIO all the servers crashed.
All the server processes were crashed due to ENOMEM generated using error-gen. This is the backtrace of the core generated. Core was generated by `/opt/glusterfs/3.0.1rc1/sbin/glusterfsd -f dist3.vol -l /d/glusterfs/logs/distr'. Program terminated with signal 11, Segmentation fault. [New process 10727] [New process 10726] [New process 10725] [New process 10729] [New process 10728] [New process 10733] [New process 10732] [New process 10730] [New process 10731] [New process 10724] #0 0x00007f2cbc2c0280 in gf_stat_from_stat (gf_stat=0x61fd1c, stat=0x0) at ../../../../libglusterfs/src/protocol.h:87 87 gf_stat->dev = hton64 (stat->st_dev); (gdb) bt #0 0x00007f2cbc2c0280 in gf_stat_from_stat (gf_stat=0x61fd1c, stat=0x0) at ../../../../libglusterfs/src/protocol.h:87 #1 0x00007f2cbc2c2ccc in server_fsync_cbk (frame=0x7f2cb40009d8, cookie=0x7f2cb4001160, this=0x611ca0, op_ret=0, op_errno=0, prebuf=0x7f2cbdce334c, postbuf=0x0) at server-protocol.c:1857 #2 0x00007f2cbc4e6e7b in iot_fsync_cbk (frame=0x7f2cb4001160, cookie=0x61fbc0, this=0x6113b0, op_ret=0, op_errno=0, prebuf=0x7f2cbdce334c, postbuf=0x0) at io-threads.c:1047 #3 0x00007f2cbdce3424 in default_fsync_cbk (frame=0x61fbc0, cookie=0x621250, this=0x610b10, op_ret=0, op_errno=0, prebuf=0x7f2cbdce334c, postbuf=0x0) at defaults.c:630 #4 0x00007f2cbc90e817 in error_gen_fsync_cbk (frame=0x621250, cookie=0x620e00, this=0x610220, op_ret=0, op_errno=0, prebuf=0x7f2cbb6a5e80, postbuf=0x7f2cbb6a5df0) at error-gen.c:1333 #5 0x00007f2cbcb2844f in posix_fsync (frame=0x620e00, this=0x60f830, fd=0x7f2cb4000e00, datasync=0) at posix.c:2850 #6 0x00007f2cbc90eb0e in error_gen_fsync (frame=0x621250, this=0x610220, fd=0x7f2cb4000e00, flags=0) at error-gen.c:1361 #7 0x00007f2cbdce35bf in default_fsync (frame=0x61fbc0, this=0x610b10, fd=0x7f2cb4000e00, flags=0) at defaults.c:644 #8 0x00007f2cbc4e7011 in iot_fsync_wrapper (frame=0x7f2cb4001160, this=0x6113b0, fd=0x7f2cb4000e00, datasync=0) at io-threads.c:1056 #9 0x00007f2cbdcf836e in call_resume_wind (stub=0x7f2cb40011c0) at call-stub.c:2504 #10 0x00007f2cbdcfe312 in call_resume (stub=0x7f2cb40011c0) at call-stub.c:4304 #11 0x00007f2cbc4ecc8d in iot_worker_ordered (arg=0x6170e0) at io-threads.c:2338 #12 0x00007f2cbd8ac3a0 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f2cbd619bdd in clone () from /lib64/libc.so.6 (gdb) p stat $1 = (struct stat *) 0x0 (gdb) The server log file is attached. LTP tests were being run during the time of the crash.
PATCH: http://patches.gluster.com/patch/2679 in master (error-gen: unwind fsync with proper parameters)
*** Bug 372 has been marked as a duplicate of this bug. ***