Description of problem:

glusterfs process crashed while running some tests. Was not able to reproduce the bug with same tests and valgrind also did not point out to anything.


This is the backtrace.

GNU gdb (GDB) Fedora (7.4.50.20120120-49.fc17)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/glusterfs...done.
[New LWP 17927]
[New LWP 15628]
[New LWP 6871]
[New LWP 17928]
[New LWP 6884]
[New LWP 15629]
[New LWP 17929]
[New LWP 6883]
[New LWP 17931]
[New LWP 1376]
[New LWP 17934]
[New LWP 15631]
[New LWP 6881]
[New LWP 15634]
[New LWP 15632]
[New LWP 536]
[New LWP 1235]
[New LWP 17932]
[New LWP 17930]
[New LWP 15633]
[New LWP 6872]
[New LWP 15630]
[New LWP 6882]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/glusterfsd -s localhost --volfile-id repl.thinkpad.export1-repl'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003a99c7bcfb in malloc_consolidate () from /lib64/libc.so.6

Thread 23 (Thread 0x7f2672cec700 (LWP 6882)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a2eaac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a2eaac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a2eaac, proghdr=0x7f2672ceadf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26780dec60) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a30b1c, req=0x2a2eaac, arg=0x7f2672ceae90, payload=0x0, 
    payloadcount=0, iobref=0x7f26780dec60, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a30b1c, cookie=0x2a313dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672ceb290, postbuf=0x7f2672ceb220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a313dc, cookie=0x2a3149c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672ceb290, postbuf=0x7f2672ceb220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a3149c, cookie=0x2a3155c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672ceb290, postbuf=0x7f2672ceb220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a3155c, cookie=0x7f268d258140, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672ceb290, postbuf=0x7f2672ceb220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d258140, cookie=0x7f268d2575d4, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672ceb290, postbuf=0x7f2672ceb220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2575d4, this=0x1974bf0, fd=0x1f58a8c, vector=0x2a31ae0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a2ea00, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2575d4, this=0x19763a0, fd=0x1f58a8c, vector=0x2a31ae0, count=1, 
    off=2408448, flags=32769, iobref=0x2a2ea00, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d258140, this=0x1977510, fd=0x1f58a8c, vector=0x2a31ae0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a2ea00, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a3155c, this=0x19785a0, fd=0x1f58a8c, vector=0x2a31ae0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a2ea00, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a3161c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a3161c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f26727e7700 (LWP 15630)):
#0  0x0000003a9a00c0e2 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a4a3ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a4a3ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a4a3ac, proghdr=0x7f26727e5df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26800970f0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a4c41c, req=0x2a4a3ac, arg=0x7f26727e5e90, payload=0x0, 
    payloadcount=0, iobref=0x7f26800970f0, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a4c41c, cookie=0x2a4ccdc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26727e6290, postbuf=0x7f26727e6220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a4ccdc, cookie=0x2a4cd9c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26727e6290, postbuf=0x7f26727e6220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a4cd9c, cookie=0x2a4ce5c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26727e6290, postbuf=0x7f26727e6220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a4ce5c, cookie=0x7f268d21ec24, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26727e6290, postbuf=0x7f26727e6220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d21ec24, cookie=0x7f268d2606f4, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26727e6290, postbuf=0x7f26727e6220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2606f4, this=0x1974bf0, fd=0x1f5e6ac, vector=0x2a4d3e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a4a300, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2606f4, this=0x19763a0, fd=0x1f5e6ac, vector=0x2a4d3e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a4a300, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d21ec24, this=0x1977510, fd=0x1f5e6ac, vector=0x2a4d3e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a4a300, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a4ce5c, this=0x19785a0, fd=0x1f5e6ac, vector=0x2a4d3e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a4a300, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a4cf1c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a4cf1c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f2672eee700 (LWP 6872)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a50c8c) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a50c8c) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a50c8c, proghdr=0x7f2672eecdf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f265c09d900) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a52cfc, req=0x2a50c8c, arg=0x7f2672eece90, payload=0x0, 
    payloadcount=0, iobref=0x7f265c09d900, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a52cfc, cookie=0x2a535bc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672eed290, postbuf=0x7f2672eed220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a535bc, cookie=0x2a5367c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672eed290, postbuf=0x7f2672eed220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a5367c, cookie=0x2a5373c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672eed290, postbuf=0x7f2672eed220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a5373c, cookie=0x7f268d23355c, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672eed290, postbuf=0x7f2672eed220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d23355c, cookie=0x7f268d25fee4, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672eed290, postbuf=0x7f2672eed220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d25fee4, this=0x1974bf0, fd=0x1f5ee8c, vector=0x2a53cc0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a50be0, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d25fee4, this=0x19763a0, fd=0x1f5ee8c, vector=0x2a53cc0, count=1, 
    off=2408448, flags=32769, iobref=0x2a50be0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d23355c, this=0x1977510, fd=0x1f5ee8c, vector=0x2a53cc0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a50be0, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a5373c, this=0x19785a0, fd=0x1f5ee8c, vector=0x2a53cc0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a50be0, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a537fc) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a537fc) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f26724e4700 (LWP 15633)):
#0  0x0000003a9a00c0e2 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a472ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a472ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a472ac, proghdr=0x7f26724e2df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f264c025610) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a4931c, req=0x2a472ac, arg=0x7f26724e2e90, payload=0x0, 
    payloadcount=0, iobref=0x7f264c025610, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a4931c, cookie=0x2a49bdc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26724e3290, postbuf=0x7f26724e3220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a49bdc, cookie=0x2a49c9c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26724e3290, postbuf=0x7f26724e3220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a49c9c, cookie=0x2a49d5c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26724e3290, postbuf=0x7f26724e3220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a49d5c, cookie=0x7f268d257120, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26724e3290, postbuf=0x7f26724e3220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d257120, cookie=0x7f268d255798, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26724e3290, postbuf=0x7f26724e3220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d255798, this=0x1974bf0, fd=0x1f5e28c, vector=0x2a4a2e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a47200, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d255798, this=0x19763a0, fd=0x1f5e28c, vector=0x2a4a2e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a47200, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d257120, this=0x1977510, fd=0x1f5e28c, vector=0x2a4a2e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a47200, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a49d5c, this=0x19785a0, fd=0x1f5e28c, vector=0x2a4a2e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a47200, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a49e1c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a49e1c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f268bc43700 (LWP 17930)):
#0  0x0000003a9a00b902 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f268e40fa27 in syncenv_task (proc=0x1953100) at ../../../libglusterfs/src/syncop.c:282
#2  0x00007f268e40fc79 in syncenv_processor (thdata=0x1953100) at ../../../libglusterfs/src/syncop.c:360
#3  0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f2688fe4700 (LWP 17932)):
#0  0x0000003a9a00b595 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2689d839dd in index_worker (data=0x1979680) at ../../../../../xlators/features/index/src/index.c:70
#2  0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f26731f1700 (LWP 1235)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a226ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a226ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a226ac, proghdr=0x7f26731efdf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f263c0e22c0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a2471c, req=0x2a226ac, arg=0x7f26731efe90, payload=0x0, 
    payloadcount=0, iobref=0x7f263c0e22c0, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a2471c, cookie=0x2a24fdc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26731f0290, postbuf=0x7f26731f0220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a24fdc, cookie=0x2a2509c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26731f0290, postbuf=0x7f26731f0220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a2509c, cookie=0x2a2515c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26731f0290, postbuf=0x7f26731f0220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a2515c, cookie=0x7f268d23f98c, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26731f0290, postbuf=0x7f26731f0220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d23f98c, cookie=0x7f268d206fdc, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26731f0290, postbuf=0x7f26731f0220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d206fdc, this=0x1974bf0, fd=0x1f1e8fc, vector=0x2a256e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a22600, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d206fdc, this=0x19763a0, fd=0x1f1e8fc, vector=0x2a256e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a22600, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d23f98c, this=0x1977510, fd=0x1f1e8fc, vector=0x2a256e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a22600, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a2515c, this=0x19785a0, fd=0x1f1e8fc, vector=0x2a256e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a22600, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a2521c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a2521c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f2673cfc700 (LWP 536)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a4d4ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a4d4ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a4d4ac, proghdr=0x7f2673cfadf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26680cc350) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a4f51c, req=0x2a4d4ac, arg=0x7f2673cfae90, payload=0x0, 
    payloadcount=0, iobref=0x7f26680cc350, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a4f51c, cookie=0x2a4fddc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2673cfb290, postbuf=0x7f2673cfb220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a4fddc, cookie=0x2a4fe9c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2673cfb290, postbuf=0x7f2673cfb220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a4fe9c, cookie=0x2a4ff5c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2673cfb290, postbuf=0x7f2673cfb220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a4ff5c, cookie=0x7f268d259568, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2673cfb290, postbuf=0x7f2673cfb220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d259568, cookie=0x7f268d2563b0, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2673cfb290, postbuf=0x7f2673cfb220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2563b0, this=0x1974bf0, fd=0x1f5ec8c, vector=0x2a504e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a4d400, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2563b0, this=0x19763a0, fd=0x1f5ec8c, vector=0x2a504e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a4d400, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d259568, this=0x1977510, fd=0x1f5ec8c, vector=0x2a504e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a4d400, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a4ff5c, this=0x19785a0, fd=0x1f5ec8c, vector=0x2a504e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a4d400, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a5001c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a5001c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f26725e5700 (LWP 15632)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a441ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a441ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a441ac, proghdr=0x7f26725e3df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f2634003550) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a4621c, req=0x2a441ac, arg=0x7f26725e3e90, payload=0x0, 
    payloadcount=0, iobref=0x7f2634003550, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a4621c, cookie=0x2a46adc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26725e4290, postbuf=0x7f26725e4220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a46adc, cookie=0x2a46b9c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26725e4290, postbuf=0x7f26725e4220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a46b9c, cookie=0x2a46c5c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26725e4290, postbuf=0x7f26725e4220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a46c5c, cookie=0x7f268d26288c, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26725e4290, postbuf=0x7f26725e4220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d26288c, cookie=0x7f268d228f68, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26725e4290, postbuf=0x7f26725e4220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d228f68, this=0x1974bf0, fd=0x1f5dfcc, vector=0x2a471e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a44100, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d228f68, this=0x19763a0, fd=0x1f5dfcc, vector=0x2a471e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a44100, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d26288c, this=0x1977510, fd=0x1f5dfcc, vector=0x2a471e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a44100, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a46c5c, this=0x19785a0, fd=0x1f5dfcc, vector=0x2a471e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a44100, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a46d1c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a46d1c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f26723e3700 (LWP 15634)):
#0  0x0000003a9a00c0e2 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a31bac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a31bac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a31bac, proghdr=0x7f26723e1df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f2640094a90) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a33c1c, req=0x2a31bac, arg=0x7f26723e1e90, payload=0x0, 
    payloadcount=0, iobref=0x7f2640094a90, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a33c1c, cookie=0x2a344dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26723e2290, postbuf=0x7f26723e2220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a344dc, cookie=0x2a3459c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26723e2290, postbuf=0x7f26723e2220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a3459c, cookie=0x2a3465c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26723e2290, postbuf=0x7f26723e2220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a3465c, cookie=0x7f268d258f5c, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26723e2290, postbuf=0x7f26723e2220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d258f5c, cookie=0x7f268d2561ac, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26723e2290, postbuf=0x7f26723e2220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2561ac, this=0x1974bf0, fd=0x1f59c6c, vector=0x2a34be0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a31b00, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2561ac, this=0x19763a0, fd=0x1f59c6c, vector=0x2a34be0, count=1, 
    off=2408448, flags=32769, iobref=0x2a31b00, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d258f5c, this=0x1977510, fd=0x1f59c6c, vector=0x2a34be0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a31b00, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a3465c, this=0x19785a0, fd=0x1f59c6c, vector=0x2a34be0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a31b00, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a3471c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a3471c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f2672ded700 (LWP 6881)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a3dfac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a3dfac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a3dfac, proghdr=0x7f2672debdf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26500ce150) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a4001c, req=0x2a3dfac, arg=0x7f2672debe90, payload=0x0, 
    payloadcount=0, iobref=0x7f26500ce150, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a4001c, cookie=0x2a408dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672dec290, postbuf=0x7f2672dec220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a408dc, cookie=0x2a4099c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672dec290, postbuf=0x7f2672dec220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a4099c, cookie=0x2a40a5c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672dec290, postbuf=0x7f2672dec220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a40a5c, cookie=0x7f268d23ac48, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672dec290, postbuf=0x7f2672dec220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d23ac48, cookie=0x7f268d2608f8, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672dec290, postbuf=0x7f2672dec220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2608f8, this=0x1974bf0, fd=0x1f5becc, vector=0x2a40fe0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a3df00, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2608f8, this=0x19763a0, fd=0x1f5becc, vector=0x2a40fe0, count=1, 
    off=2408448, flags=32769, iobref=0x2a3df00, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d23ac48, this=0x1977510, fd=0x1f5becc, vector=0x2a40fe0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a3df00, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a40a5c, this=0x19785a0, fd=0x1f5becc, vector=0x2a40fe0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a3df00, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a40b1c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a40b1c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f26726e6700 (LWP 15631)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a288ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a288ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a288ac, proghdr=0x7f26726e4df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f264409ed20) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a2a91c, req=0x2a288ac, arg=0x7f26726e4e90, payload=0x0, 
    payloadcount=0, iobref=0x7f264409ed20, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a2a91c, cookie=0x2a2b1dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26726e5290, postbuf=0x7f26726e5220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a2b1dc, cookie=0x2a2b29c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26726e5290, postbuf=0x7f26726e5220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a2b29c, cookie=0x2a2b35c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26726e5290, postbuf=0x7f26726e5220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a2b35c, cookie=0x7f268d257be0, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26726e5290, postbuf=0x7f26726e5220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d257be0, cookie=0x7f268d20aba8, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26726e5290, postbuf=0x7f26726e5220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d20aba8, this=0x1974bf0, fd=0x1f2903c, vector=0x2a2b8e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a28800, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d20aba8, this=0x19763a0, fd=0x1f2903c, vector=0x2a2b8e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a28800, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d257be0, this=0x1977510, fd=0x1f2903c, vector=0x2a2b8e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a28800, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a2b35c, this=0x19785a0, fd=0x1f2903c, vector=0x2a2b8e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a28800, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a2b41c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a2b41c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f2688de2700 (LWP 17934)):
#0  0x0000003a9a00b902 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f268a5f21a6 in janitor_get_next_fd (this=0x1974bf0)
    at ../../../../../xlators/storage/posix/src/posix-helpers.c:822
#2  0x00007f268a5f22cf in posix_janitor_thread_proc (data=0x1974bf0)
    at ../../../../../xlators/storage/posix/src/posix-helpers.c:868
#3  0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f26730f0700 (LWP 1376)):
#0  0x0000003a9a00c0e2 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a3aeac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a3aeac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a3aeac, proghdr=0x7f26730eedf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26640ab550) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a3cf1c, req=0x2a3aeac, arg=0x7f26730eee90, payload=0x0, 
    payloadcount=0, iobref=0x7f26640ab550, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a3cf1c, cookie=0x2a3d7dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26730ef290, postbuf=0x7f26730ef220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a3d7dc, cookie=0x2a3d89c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26730ef290, postbuf=0x7f26730ef220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a3d89c, cookie=0x2a3d95c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26730ef290, postbuf=0x7f26730ef220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a3d95c, cookie=0x7f268d258b54, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26730ef290, postbuf=0x7f26730ef220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d258b54, cookie=0x7f268d2583f0, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26730ef290, postbuf=0x7f26730ef220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2583f0, this=0x1974bf0, fd=0x1f5bc0c, vector=0x2a3dee0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a3ae00, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2583f0, this=0x19763a0, fd=0x1f5bc0c, vector=0x2a3dee0, count=1, 
    off=2408448, flags=32769, iobref=0x2a3ae00, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d258b54, this=0x1977510, fd=0x1f5bc0c, vector=0x2a3dee0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a3ae00, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a3d95c, this=0x19785a0, fd=0x1f5bc0c, vector=0x2a3dee0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a3ae00, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a3da1c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a3da1c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f268affe700 (LWP 17931)):
#0  0x0000003a9a00e80d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f268e3e8ba0 in gf_timer_proc (ctx=0x1936010) at ../../../libglusterfs/src/timer.c:173
#2  0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f2672beb700 (LWP 6883)):
#0  0x0000003a9a00c0e2 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a410ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a410ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a410ac, proghdr=0x7f2672be9df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26580d7350) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a4311c, req=0x2a410ac, arg=0x7f2672be9e90, payload=0x0, 
    payloadcount=0, iobref=0x7f26580d7350, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a4311c, cookie=0x2a439dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672bea290, postbuf=0x7f2672bea220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a439dc, cookie=0x2a43a9c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672bea290, postbuf=0x7f2672bea220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a43a9c, cookie=0x2a43b5c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672bea290, postbuf=0x7f2672bea220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a43b5c, cookie=0x7f268d259970, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672bea290, postbuf=0x7f2672bea220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d259970, cookie=0x7f268d259364, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672bea290, postbuf=0x7f2672bea220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d259364, this=0x1974bf0, fd=0x1f5dbac, vector=0x2a440e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a41000, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d259364, this=0x19763a0, fd=0x1f5dbac, vector=0x2a440e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a41000, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d259970, this=0x1977510, fd=0x1f5dbac, vector=0x2a440e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a41000, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a43b5c, this=0x19785a0, fd=0x1f5dbac, vector=0x2a440e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a41000, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a43c1c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a43c1c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f268c444700 (LWP 17929)):
#0  0x0000003a9a00b902 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f268e40fa27 in syncenv_task (proc=0x1952d40) at ../../../libglusterfs/src/syncop.c:282
#2  0x00007f268e40fc79 in syncenv_processor (thdata=0x1952d40) at ../../../libglusterfs/src/syncop.c:360
#3  0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f26728e8700 (LWP 15629)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a257ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a257ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a257ac, proghdr=0x7f26728e6df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f2654004df0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a2781c, req=0x2a257ac, arg=0x7f26728e6e90, payload=0x0, 
    payloadcount=0, iobref=0x7f2654004df0, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a2781c, cookie=0x2a280dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26728e7290, postbuf=0x7f26728e7220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a280dc, cookie=0x2a2819c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26728e7290, postbuf=0x7f26728e7220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a2819c, cookie=0x2a2825c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26728e7290, postbuf=0x7f26728e7220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a2825c, cookie=0x7f268d2567b8, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26728e7290, postbuf=0x7f26728e7220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d2567b8, cookie=0x7f268d2573d0, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26728e7290, postbuf=0x7f26728e7220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2573d0, this=0x1974bf0, fd=0x1f4f77c, vector=0x2a287e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a25700, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2573d0, this=0x19763a0, fd=0x1f4f77c, vector=0x2a287e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a25700, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d2567b8, this=0x1977510, fd=0x1f4f77c, vector=0x2a287e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a25700, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a2825c, this=0x19785a0, fd=0x1f4f77c, vector=0x2a287e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a25700, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a2831c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a2831c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f2672aea700 (LWP 6884)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a34cac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a34cac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a34cac, proghdr=0x7f2672ae8df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26600d9c40) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a36d1c, req=0x2a34cac, arg=0x7f2672ae8e90, payload=0x0, 
    payloadcount=0, iobref=0x7f26600d9c40, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a36d1c, cookie=0x2a375dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672ae9290, postbuf=0x7f2672ae9220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a375dc, cookie=0x2a3769c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672ae9290, postbuf=0x7f2672ae9220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a3769c, cookie=0x2a3775c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672ae9290, postbuf=0x7f2672ae9220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a3775c, cookie=0x7f268d258950, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672ae9290, postbuf=0x7f2672ae9220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d258950, cookie=0x7f268d2577d8, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672ae9290, postbuf=0x7f2672ae9220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2577d8, this=0x1974bf0, fd=0x1f5aa2c, vector=0x2a37ce0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a34c00, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2577d8, this=0x19763a0, fd=0x1f5aa2c, vector=0x2a37ce0, count=1, 
    off=2408448, flags=32769, iobref=0x2a34c00, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d258950, this=0x1977510, fd=0x1f5aa2c, vector=0x2a37ce0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a34c00, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a3775c, this=0x19785a0, fd=0x1f5aa2c, vector=0x2a37ce0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a34c00, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a3781c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a3781c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f268cc45700 (LWP 17928)):
#0  0x0000003a9a00ed70 in sigwait () from /lib64/libpthread.so.0
#1  0x0000000000407bc8 in glusterfs_sigwaiter (arg=0x7fff27635260) at ../../../glusterfsd/src/glusterfsd.c:1571
#2  0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f2672fef700 (LWP 6871)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a37dac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a37dac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a37dac, proghdr=0x7f2672feddf0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f26480ce730) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a39e1c, req=0x2a37dac, arg=0x7f2672fede90, payload=0x0, 
    payloadcount=0, iobref=0x7f26480ce730, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a39e1c, cookie=0x2a3a6dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672fee290, postbuf=0x7f2672fee220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a3a6dc, cookie=0x2a3a79c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672fee290, postbuf=0x7f2672fee220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a3a79c, cookie=0x2a3a85c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f2672fee290, postbuf=0x7f2672fee220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a3a85c, cookie=0x7f268d259f7c, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672fee290, postbuf=0x7f2672fee220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d259f7c, cookie=0x7f268d2579dc, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f2672fee290, postbuf=0x7f2672fee220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d2579dc, this=0x1974bf0, fd=0x1f5ae4c, vector=0x2a3ade0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a37d00, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d2579dc, this=0x19763a0, fd=0x1f5ae4c, vector=0x2a3ade0, count=1, 
    off=2408448, flags=32769, iobref=0x2a37d00, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d259f7c, this=0x1977510, fd=0x1f5ae4c, vector=0x2a3ade0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a37d00, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a3a85c, this=0x19785a0, fd=0x1f5ae4c, vector=0x2a3ade0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a37d00, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a3a91c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a3a91c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f26729e9700 (LWP 15628)):
#0  0x0000003a9a00c0e5 in ?? () from /lib64/libpthread.so.0
#1  0x00007f268e4010ed in mem_put (ptr=0x2a2b9ac) at ../../../libglusterfs/src/mem-pool.c:495
#2  0x00007f268e1a84dd in rpcsvc_request_destroy (req=0x2a2b9ac) at ../../../../rpc/rpc-lib/src/rpcsvc.c:284
#3  0x00007f268e1aa216 in rpcsvc_submit_generic (req=0x2a2b9ac, proghdr=0x7f26729e7df0, hdrcount=1, payload=0x0, 
    payloadcount=0, iobref=0x7f266c0a2300) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1131
#4  0x00007f26897131e2 in server_submit_reply (frame=0x2a2da1c, req=0x2a2b9ac, arg=0x7f26729e7e90, payload=0x0, 
    payloadcount=0, iobref=0x7f266c0a2300, xdrproc=0x7f268df90817 <xdr_gfs3_write_rsp>)
    at ../../../../../xlators/protocol/server/src/server.c:163
#5  0x00007f26897223a0 in server_writev_cbk (frame=0x2a2da1c, cookie=0x2a2e2dc, this=0x197cad0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26729e8290, postbuf=0x7f26729e8220, xdata=0x0)
    at ../../../../../xlators/protocol/server/src/server-rpc-fops.c:1433
#6  0x00007f268994a395 in io_stats_writev_cbk (frame=0x2a2e2dc, cookie=0x2a2e39c, this=0x197b9c0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26729e8290, postbuf=0x7f26729e8220, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:1371
#7  0x00007f2689b63b0e in marker_writev_cbk (frame=0x2a2e39c, cookie=0x2a2e45c, this=0x197a7d0, op_ret=131072, op_errno=0, 
    prebuf=0x7f26729e8290, postbuf=0x7f26729e8220, xdata=0x0) at ../../../../../xlators/features/marker/src/marker.c:643
#8  0x00007f2689f95706 in iot_writev_cbk (frame=0x2a2e45c, cookie=0x7f268d228804, this=0x19785a0, op_ret=131072, 
    op_errno=0, prebuf=0x7f26729e8290, postbuf=0x7f26729e8220, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1121
#9  0x00007f268a1b0e99 in pl_writev_cbk (frame=0x7f268d228804, cookie=0x7f268d261510, this=0x1977510, op_ret=131072, 
    op_errno=0, prebuf=0x7f26729e8290, postbuf=0x7f26729e8220, xdata=0x0)
    at ../../../../../xlators/features/locks/src/posix.c:1008
#10 0x00007f268a5e631c in posix_writev (frame=0x7f268d261510, this=0x1974bf0, fd=0x1f5472c, vector=0x2a2e9e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a2b900, xdata=0x0) at ../../../../../xlators/storage/posix/src/posix.c:2169
#11 0x00007f268e3e19ec in default_writev (frame=0x7f268d261510, this=0x19763a0, fd=0x1f5472c, vector=0x2a2e9e0, count=1, 
    off=2408448, flags=32769, iobref=0x2a2b900, xdata=0x0) at ../../../libglusterfs/src/defaults.c:949
#12 0x00007f268a1b210c in pl_writev (frame=0x7f268d228804, this=0x1977510, fd=0x1f5472c, vector=0x2a2e9e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a2b900, xdata=0x0) at ../../../../../xlators/features/locks/src/posix.c:1240
#13 0x00007f2689f95956 in iot_writev_wrapper (frame=0x2a2e45c, this=0x19785a0, fd=0x1f5472c, vector=0x2a2e9e0, count=1, 
    offset=2408448, flags=32769, iobref=0x2a2b900, xdata=0x0)
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1133
#14 0x00007f268e3f6022 in call_resume_wind (stub=0x2a2e51c) at ../../../libglusterfs/src/call-stub.c:2494
#15 0x00007f268e3fd280 in call_resume (stub=0x2a2e51c) at ../../../libglusterfs/src/call-stub.c:4142
#16 0x00007f2689f8ff1e in iot_worker (data=0x198a8a0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:191
#17 0x0000003a9a007d14 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a99cf199d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f268df62740 (LWP 17927)):
#0  0x0000003a99c7bcfb in malloc_consolidate () from /lib64/libc.so.6
#1  0x0000003a99c7ce1e in _int_malloc () from /lib64/libc.so.6
#2  0x0000003a99c810d8 in calloc () from /lib64/libc.so.6
#3  0x00007f268e3fff53 in __gf_default_calloc (cnt=1, size=6388) at ../../../libglusterfs/src/mem-pool.h:75
#4  0x00007f268e40035c in __gf_calloc (nmemb=1, size=6388, type=48) at ../../../libglusterfs/src/mem-pool.c:113
#5  0x00007f268e400ea5 in mem_get (mem_pool=0x197ea60) at ../../../libglusterfs/src/mem-pool.c:428
#6  0x00007f268e1a8768 in rpcsvc_request_create (svc=0x197e9b0, trans=0x1eea120, msg=0x20b90a0)
    at ../../../../rpc/rpc-lib/src/rpcsvc.c:361
#7  0x00007f268e1a8cb6 in rpcsvc_handle_rpc_call (svc=0x197e9b0, trans=0x1eea120, msg=0x20b90a0)
    at ../../../../rpc/rpc-lib/src/rpcsvc.c:504
#8  0x00007f268e1a91bd in rpcsvc_notify (trans=0x1eea120, mydata=0x197e9b0, event=RPC_TRANSPORT_MSG_RECEIVED, 
    data=0x20b90a0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:644
#9  0x00007f268e1ae001 in rpc_transport_notify (this=0x1eea120, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x20b90a0)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:495
#10 0x00007f268b23601a in socket_event_poll_in (this=0x1eea120)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:2088
#11 0x00007f268b2364b2 in socket_event_handler (fd=1713, idx=8, data=0x1eea120, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:2200
#12 0x00007f268e42835e in event_dispatch_epoll_handler (event_pool=0x1951ea0, events=0x196eb80, i=0)
    at ../../../libglusterfs/src/event-epoll.c:384
#13 0x00007f268e428543 in event_dispatch_epoll (event_pool=0x1951ea0) at ../../../libglusterfs/src/event-epoll.c:445
#14 0x00007f268e3ffde1 in event_dispatch (event_pool=0x1951ea0) at ../../../libglusterfs/src/event.c:113
#15 0x000000000040859f in main (argc=19, argv=0x7fff27636488) at ../../../glusterfsd/src/glusterfsd.c:1901
Missing separate debuginfos, use: debuginfo-install glibc-2.15-37.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-2.fc17.x86_64 libaio-0.3.109-5.fc17.x86_64 libcom_err-1.42.3-2.fc17.x86_64 libgcc-4.7.0-5.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 openssl-1.0.0j-1.fc17.x86_64 zlib-1.2.5-6.fc17.x86_64
[?1034h(gdb) quit


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info: