+++ This bug was initially created as a clone of Bug #1333266 +++ +++ This bug was initially created as a clone of Bug #1332156 +++ +++ This bug was initially created as a clone of Bug #1328411 +++ Description of problem: *************************** While running I/O's on cifs mount and changing the graph by disabling and enabling stat-prefetch , the cifs mount hangs and there are errors seen in the log. Version-Release number of selected component (if applicable): glusterfs-3.7.9-1.el7rhgs.x86_64 samba-4.4.0-2.el7rhgs.x86_64 How reproducible: tried once Steps to Reproduce: 1.Create a volume and mount it on linux cifs client 2.start dd on the mount point "dd if=/dev/zero of=/mnt/cifs/file.dd bs=1M count=1024" 3.On the server do stat-prefetch off , on and off 4. check the mount point, the i/o's are hung. Actual results: The i/o is hung and there are errors in the logs. Expected results: The i/o should not get hung. Additional info: [2016-04-19 07:37:01.717797] E [dht-helper.c:1592:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so(client3_3_lookup _cbk+0x707) [0x7ff26f259417] -->/usr/lib64/glusterfs/3.7.9/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7ff26efed269] -->/usr/lib64/glust erfs/3.7.9/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7ff26efca680] ) 76-batman-dht: invalid argument: inode [Invalid argument] [2016-04-19 07:33:22.400959] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 76-batman-client-0: remote operation failed. Path: / (000 00000-0000-0000-0000-000000000001) [Transport endpoint is not connected] --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-19 06:45:28 EDT --- This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Poornima G on 2016-04-19 08:18:11 EDT --- Here is the bt of all threads: Thread 21 (Thread 0x7ff272aec700 (LWP 4938)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f53d4c0, iovcnt=1, offset=25624576, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 20 (Thread 0x7ff2722eb700 (LWP 4939)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fea1f70, iovcnt=1, offset=26083328, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 19 (Thread 0x7ff271274700 (LWP 4940)): #0 0x00007ff28d39d96d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff2739979ec in gf_timer_proc (ctx=0x7ff28f540a90) at timer.c:193 #2 0x00007ff28d396dc5 in start_thread (arg=0x7ff271274700) at pthread_create.c:308 #3 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 18 (Thread 0x7ff270870700 (LWP 4941)): #0 0x00007ff28d397ef7 in pthread_join (threadid=140679236122368, thread_return=thread_return@entry=0x0) at pthread_join.c:92 #1 0x00007ff2739d9d28 in event_dispatch_epoll (event_pool=0x7ff28f55f550) at event-epoll.c:762 #2 0x00007ff274058614 in glfs_poller (data=<optimized out>) at glfs.c:587 #3 0x00007ff28d396dc5 in start_thread (arg=0x7ff270870700) at pthread_create.c:308 #4 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 17 (Thread 0x7ff26fe57700 (LWP 4942)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff27406747b in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_subvol_done (fs=fs@entry=0x7ff28f540930, subvol=subvol@entry=0x7ff2600866d0) at glfs-resolve.c:884 #3 0x00007ff270871de1 in graph_setup (fs=fs@entry=0x7ff28f540930, graph=graph@entry=0x7ff260075630) at glfs-master.c:75 #4 0x00007ff270871f1b in notify (this=<optimized out>, event=<optimized out>, data=0x7ff260075630) at glfs-master.c:103 #5 0x00007ff273975906 in xlator_notify (xl=0x7ff28f570b30, event=5, data=0x7ff260075630) at xlator.c:496 #6 0x00007ff27398c2fe in default_notify (this=0x7ff2602d2620, event=5, data=<optimized out>) at defaults.c:2316 #7 0x00007ff273975906 in xlator_notify (xl=0x7ff2602d2620, event=event@entry=5, data=data@entry=0x7ff2602d1820) at xlator.c:496 #8 0x00007ff27398c1f3 in default_notify (this=0x7ff2602d1820, event=5, data=<optimized out>) at defaults.c:2320 #9 0x00007ff26e34fbe9 in notify (this=0x7ff2602d1820, event=<optimized out>, data=0x7ff2602d0b40) at io-stats.c:3064 #10 0x00007ff273975906 in xlator_notify (xl=0x7ff2602d1820, event=event@entry=5, data=data@entry=0x7ff2602d0b40) at xlator.c:496 #11 0x00007ff27398c1f3 in default_notify (this=0x7ff2602d0b40, event=5, data=<optimized out>) at defaults.c:2320 #12 0x00007ff273975906 in xlator_notify (xl=0x7ff2602d0b40, event=event@entry=5, data=data@entry=0x7ff2602cfe70) at xlator.c:496 #13 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cfe70, event=5, data=<optimized out>) at defaults.c:2320 #14 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cfe70, event=event@entry=5, data=data@entry=0x7ff2602cf3d0) at xlator.c:496 #15 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cf3d0, event=5, data=<optimized out>) at defaults.c:2320 #16 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cf3d0, event=event@entry=5, data=data@entry=0x7ff2602ce930) at xlator.c:496 #17 0x00007ff27398c1f3 in default_notify (this=0x7ff2602ce930, event=5, data=<optimized out>) at defaults.c:2320 #18 0x00007ff273975906 in xlator_notify (xl=0x7ff2602ce930, event=event@entry=5, data=data@entry=0x7ff2602cde90) at xlator.c:496 #19 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cde90, event=5, data=<optimized out>) at defaults.c:2320 #20 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cde90, event=event@entry=5, data=data@entry=0x7ff2602cd3f0) at xlator.c:496 #21 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cd3f0, event=5, data=<optimized out>) at defaults.c:2320 #22 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cd3f0, event=event@entry=5, data=data@entry=0x7ff2602cc950) at xlator.c:496 #23 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cc950, event=event@entry=5, data=data@entry=0x7ff2602cbeb0) at defaults.c:2320 #24 0x00007ff26f1e1b3f in dht_notify (this=<optimized out>, event=5, data=0x7ff2602cbeb0) at dht-common.c:7844 #25 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cc950, event=event@entry=5, data=data@entry=0x7ff2602cbeb0) at xlator.c:496 #26 0x00007ff27398c1f3 in default_notify (this=this@entry=0x7ff2602cbeb0, event=event@entry=5, data=data@entry=0x0) at defaults.c:2320 #27 0x00007ff26f41f859 in client_notify_dispatch (this=this@entry=0x7ff2602cbeb0, event=event@entry=5, data=data@entry=0x0) at client.c:88 #28 0x00007ff26f444406 in client_notify_parents_child_up (this=0x7ff2602cbeb0) at client-handshake.c:138 #29 0x00007ff26f44615e in client_post_handshake (frame=frame@entry=0x7ff271a40560, this=0x7ff2602cbeb0) at client-handshake.c:1069 #30 0x00007ff26f446894 in client_setvolume_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7ff271a40560) at client-handshake.c:1226 #31 0x00007ff273e46a20 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7ff2604b2d00, pollin=pollin@entry=0x7ff2602c33d0) at rpc-clnt.c:764 #32 0x00007ff273e46cdf in rpc_clnt_notify (trans=<optimized out>, mydata=0x7ff2604b2d30, event=<optimized out>, data=0x7ff2602c33d0) at rpc-clnt.c:905 #33 0x00007ff273e42823 in rpc_transport_notify (this=this@entry=0x7ff26050c230, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7ff2602c33d0) at rpc-transport.c:546 #34 0x00007ff26fe5e666 in socket_event_poll_in (this=this@entry=0x7ff26050c230) at socket.c:2237 #35 0x00007ff26fe612c4 in socket_event_handler (fd=fd@entry=44, idx=idx@entry=4, data=0x7ff26050c230, poll_in=1, poll_out=0, poll_err=0) at socket.c:2350 #36 0x00007ff2739d99ca in event_dispatch_epoll_handler (event=0x7ff26fe56d80, event_pool=0x7ff28f55f550) at event-epoll.c:575 #37 event_dispatch_epoll_worker (data=0x7ff268000920) at event-epoll.c:678 #38 0x00007ff28d396dc5 in start_thread (arg=0x7ff26fe57700) at pthread_create.c:308 #39 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 16 (Thread 0x7ff26df7a700 (LWP 4943)): #0 0x00007ff2893bb863 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff2739d9820 in event_dispatch_epoll_worker (data=0x7ff26002f6a0) at event-epoll.c:668 #2 0x00007ff28d396dc5 in start_thread (arg=0x7ff26df7a700) at pthread_create.c:308 #3 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7ff25f57e700 (LWP 4970)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f53c5c0, iovcnt=1, offset=25427968, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 14 (Thread 0x7ff25ed7d700 (LWP 4971)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe44500, iovcnt=1, offset=25886720, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 13 (Thread 0x7ff25e57c700 (LWP 4974)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f59fe70, iovcnt=1, offset=25493504, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 12 (Thread 0x7ff25dd7b700 (LWP 4976)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28feb3990, iovcnt=1, offset=26411008, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 11 (Thread 0x7ff25d57a700 (LWP 4977)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5a1300, iovcnt=1, offset=25690112, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 10 (Thread 0x7ff25cd79700 (LWP 4978)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fea1e80, iovcnt=1, offset=26017792, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 9 (Thread 0x7ff23ffff700 (LWP 4981)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe29de0, iovcnt=1, offset=26279936, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 8 (Thread 0x7ff23f7fe700 (LWP 4985)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5f4e20, iovcnt=1, offset=25952256, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 7 (Thread 0x7ff23effd700 (LWP 4994)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5c92a0, iovcnt=1, offset=25362432, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 6 (Thread 0x7ff23e7fc700 (LWP 4995)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe29cf0, iovcnt=1, offset=26214400, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 5 (Thread 0x7ff23dffb700 (LWP 4996)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe29c00, iovcnt=1, offset=26148864, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 4 (Thread 0x7ff23d7fa700 (LWP 4997)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5a40f0, iovcnt=1, offset=25821184, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 3 (Thread 0x7ff23cff9700 (LWP 4998)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f572020, iovcnt=1, offset=25755648, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 2 (Thread 0x7ff22bfff700 (LWP 4999)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5a4590, iovcnt=1, offset=25559040, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 1 (Thread 0x7ff28d759880 (LWP 4937)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405d1cd in pub_glfs_close (glfd=0x7ff28f5a0110) at glfs-fops.c:251 #4 0x00007ff28ccce870 in fd_close (fsp=fsp@entry=0x7ff28f5a2b10) at ../source3/smbd/open.c:431 #5 0x00007ff28ccd7b65 in close_normal_file (close_type=SHUTDOWN_CLOSE, fsp=0x7ff28f5a2b10, req=0x0) at ../source3/smbd/close.c:753 #6 close_file (req=req@entry=0x0, fsp=0x7ff28f5a2b10, close_type=close_type@entry=SHUTDOWN_CLOSE) at ../source3/smbd/close.c:1220 #7 0x00007ff28cc75fca in file_close_conn (conn=conn@entry=0x7ff28f532a40) at ../source3/smbd/files.c:165 #8 0x00007ff28ccf5d59 in close_cnum (conn=0x7ff28f532a40, vuid=49446) at ../source3/smbd/service.c:1128 #9 0x00007ff28cd240a4 in smbXsrv_tcon_disconnect (tcon=0x7ff28f533a40, vuid=49446) at ../source3/smbd/smbXsrv_tcon.c:983 #10 0x00007ff28cd24372 in smbXsrv_tcon_disconnect_all_callback (local_rec=local_rec@entry=0x7ffd268981b0, private_data=private_data@entry=0x7ffd26898280) at ../source3/smbd/smbXsrv_tcon.c:1064 #11 0x00007ff28736cc2a in db_rbt_traverse_internal (db=db@entry=0x7ff28f52cc90, f=f@entry=0x7ff28cd24300 <smbXsrv_tcon_disconnect_all_callback>, private_data=private_data@entry=0x7ffd26898280, count=count@entry=0x7ffd2689823c, rw=rw@entry=true) at ../lib/dbwrap/dbwrap_rbt.c:434 #12 0x00007ff28736cd67 in db_rbt_traverse (db=0x7ff28f52cc90, f=0x7ff28cd24300 <smbXsrv_tcon_disconnect_all_callback>, private_data=0x7ffd26898280) at ../lib/dbwrap/dbwrap_rbt.c:492 #13 0x00007ff28736b20a in dbwrap_traverse (db=<optimized out>, f=f@entry=0x7ff28cd24300 <smbXsrv_tcon_disconnect_all_callback>, private_data=private_data@entry=0x7ffd26898280, count=count@entry=0x7ffd2689827c) at ../lib/dbwrap/dbwrap.c:343 #14 0x00007ff28cd230a9 in smbXsrv_tcon_disconnect_all (table=<optimized out>, vuid=vuid@entry=0) at ../source3/smbd/smbXsrv_tcon.c:1013 #15 0x00007ff28cd24463 in smb1srv_tcon_disconnect_all (conn=conn@entry=0x7ff28f514d50) at ../source3/smbd/smbXsrv_tcon.c:1132 #16 0x00007ff28cd27202 in exit_server_common (how=how@entry=SERVER_EXIT_NORMAL, reason=0x7ff28ce3a214 "failed to receive smb request") at ../source3/smbd/server_exit.c:145 #17 0x00007ff28cd2777e in smbd_exit_server_cleanly (explanation=<optimized out>) at ../source3/smbd/server_exit.c:263 #18 0x00007ff28a603ba2 in exit_server_cleanly (reason=reason@entry=0x7ff28ce3a214 "failed to receive smb request") at ../source3/lib/smbd_shim.c:131 #19 0x00007ff28ccf2114 in smbd_server_connection_read_handler (xconn=0x7ff28f514d50, fd=36) at ../source3/smbd/process.c:2600 #20 0x00007ff28ac6317c in run_events_poll (ev=0x7ff28f511eb0, pollrtn=<optimized out>, pfds=0x7ff28f5292d0, num_pfds=5) at ../source3/lib/events.c:257 #21 0x00007ff28ac633d0 in s3_event_loop_once (ev=0x7ff28f511eb0, location=<optimized out>) at ../source3/lib/events.c:326 #22 0x00007ff28968a36d in _tevent_loop_once (ev=ev@entry=0x7ff28f511eb0, location=location@entry=0x7ff28ce3cc20 "../source3/smbd/process.c:4117") at ../tevent.c:533 #23 0x00007ff28968a50b in tevent_common_loop_wait (ev=0x7ff28f511eb0, location=0x7ff28ce3cc20 "../source3/smbd/process.c:4117") at ../tevent.c:637 #24 0x00007ff28ccf33a1 in smbd_process (ev_ctx=ev_ctx@entry=0x7ff28f511eb0, msg_ctx=msg_ctx@entry=0x7ff28f511fa0, sock_fd=sock_fd@entry=36, interactive=interactive@entry=false) at ../source3/smbd/process.c:4117 #25 0x00007ff28d7db0b8 in smbd_accept_connection (ev=0x7ff28f511eb0, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:747 #26 0x00007ff28ac6317c in run_events_poll (ev=0x7ff28f511eb0, pollrtn=<optimized out>, pfds=0x7ff28f5292d0, num_pfds=6) at ../source3/lib/events.c:257 #27 0x00007ff28ac633d0 in s3_event_loop_once (ev=0x7ff28f511eb0, location=<optimized out>) at ../source3/lib/events.c:326 #28 0x00007ff28968a36d in _tevent_loop_once (ev=ev@entry=0x7ff28f511eb0, location=location@entry=0x7ff28d7de51a "../source3/smbd/server.c:1112") at ../tevent.c:533 #29 0x00007ff28968a50b in tevent_common_loop_wait (ev=0x7ff28f511eb0, location=0x7ff28d7de51a "../source3/smbd/server.c:1112") at ../tevent.c:637 #30 0x00007ff28d7d6c33 in smbd_parent_loop (parent=<optimized out>, ev_ctx=0x7ff28f511eb0) at ../source3/smbd/server.c:1112 #31 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:1765 --- Additional comment from Poornima G on 2016-04-20 05:23:36 EDT --- Even without AIO there is a hang: [root@dhcp46-40 glusterfs]# pstack 5236 Thread 7 (Thread 0x7f6ee6755700 (LWP 5237)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f6ee7605008 in syncenv_task (proc=proc@entry=0x7f6f018c2320) at syncop.c:607 #2 0x00007f6ee7605d40 in syncenv_processor (thdata=0x7f6f018c2320) at syncop.c:699 #3 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee6755700) at pthread_create.c:308 #4 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 6 (Thread 0x7f6ee5f54700 (LWP 5238)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f6ee7605008 in syncenv_task (proc=proc@entry=0x7f6f018c26e0) at syncop.c:607 #2 0x00007f6ee7605d40 in syncenv_processor (thdata=0x7f6f018c26e0) at syncop.c:699 #3 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee5f54700) at pthread_create.c:308 #4 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 5 (Thread 0x7f6ee4edd700 (LWP 5239)): #0 0x00007f6f00f6596d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f6ee75e09ec in gf_timer_proc (ctx=0x7f6f018a0670) at timer.c:193 #2 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee4edd700) at pthread_create.c:308 #3 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f6ee44d9700 (LWP 5240)): #0 0x00007f6f00f5fef7 in pthread_join (threadid=140114245027584, thread_return=thread_return@entry=0x0) at pthread_join.c:92 #1 0x00007f6ee7622d28 in event_dispatch_epoll (event_pool=0x7f6f018bf130) at event-epoll.c:762 #2 0x00007f6ee7ca1614 in glfs_poller (data=<optimized out>) at glfs.c:587 #3 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee44d9700) at pthread_create.c:308 #4 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f6ee3cd8700 (LWP 5241)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f6ee7cb047b in glfs_lock (fs=0x7f6f018a0510) at glfs-internal.h:312 #2 priv_glfs_subvol_done (fs=fs@entry=0x7f6f018a0510, subvol=subvol@entry=0x7f6ecc0133a0) at glfs-resolve.c:884 #3 0x00007f6ee44dade1 in graph_setup (fs=fs@entry=0x7f6f018a0510, graph=graph@entry=0x7f6ed4091930) at glfs-master.c:75 #4 0x00007f6ee44daf1b in notify (this=<optimized out>, event=<optimized out>, data=0x7f6ed4091930) at glfs-master.c:103 #5 0x00007f6ee75be906 in xlator_notify (xl=0x7f6f018d0770, event=5, data=0x7f6ed4091930) at xlator.c:496 #6 0x00007f6ee75d52fe in default_notify (this=0x7f6ed40a06e0, event=5, data=<optimized out>) at defaults.c:2316 #7 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed40a06e0, event=event@entry=5, data=data@entry=0x7f6ed409f8e0) at xlator.c:496 #8 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409f8e0, event=5, data=<optimized out>) at defaults.c:2320 #9 0x00007f6ee21d0be9 in notify (this=0x7f6ed409f8e0, event=<optimized out>, data=0x7f6ed409ec00) at io-stats.c:3064 #10 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409f8e0, event=event@entry=5, data=data@entry=0x7f6ed409ec00) at xlator.c:496 #11 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409ec00, event=5, data=<optimized out>) at defaults.c:2320 #12 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409ec00, event=event@entry=5, data=data@entry=0x7f6ed409df30) at xlator.c:496 #13 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409df30, event=5, data=<optimized out>) at defaults.c:2320 #14 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409df30, event=event@entry=5, data=data@entry=0x7f6ed409d490) at xlator.c:496 #15 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409d490, event=5, data=<optimized out>) at defaults.c:2320 #16 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409d490, event=event@entry=5, data=data@entry=0x7f6ed409c9f0) at xlator.c:496 #17 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409c9f0, event=5, data=<optimized out>) at defaults.c:2320 #18 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409c9f0, event=event@entry=5, data=data@entry=0x7f6ed409bf50) at xlator.c:496 #19 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409bf50, event=5, data=<optimized out>) at defaults.c:2320 #20 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409bf50, event=event@entry=5, data=data@entry=0x7f6ed409b4b0) at xlator.c:496 #21 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409b4b0, event=5, data=<optimized out>) at defaults.c:2320 #22 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409b4b0, event=event@entry=5, data=data@entry=0x7f6ed409aa10) at xlator.c:496 #23 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409aa10, event=event@entry=5, data=data@entry=0x7f6ed4099f70) at defaults.c:2320 #24 0x00007f6ee3062b3f in dht_notify (this=<optimized out>, event=5, data=0x7f6ed4099f70) at dht-common.c:7844 #25 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409aa10, event=event@entry=5, data=data@entry=0x7f6ed4099f70) at xlator.c:496 #26 0x00007f6ee75d51f3 in default_notify (this=this@entry=0x7f6ed4099f70, event=event@entry=5, data=data@entry=0x0) at defaults.c:2320 #27 0x00007f6ee32a0859 in client_notify_dispatch (this=this@entry=0x7f6ed4099f70, event=event@entry=5, data=data@entry=0x0) at client.c:88 #28 0x00007f6ee32c5406 in client_notify_parents_child_up (this=0x7f6ed4099f70) at client-handshake.c:138 #29 0x00007f6ee32c715e in client_post_handshake (frame=frame@entry=0x7f6ee56a77d0, this=0x7f6ed4099f70) at client-handshake.c:1069 #30 0x00007f6ee32c7894 in client_setvolume_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7f6ee56a77d0) at client-handshake.c:1226 #31 0x00007f6ee7a8fa20 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f6ed40d3b90, pollin=pollin@entry=0x7f6ed4091470) at rpc-clnt.c:764 #32 0x00007f6ee7a8fcdf in rpc_clnt_notify (trans=<optimized out>, mydata=0x7f6ed40d3bc0, event=<optimized out>, data=0x7f6ed4091470) at rpc-clnt.c:905 #33 0x00007f6ee7a8b823 in rpc_transport_notify (this=this@entry=0x7f6ed40e3870, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7f6ed4091470) at rpc-transport.c:546 #34 0x00007f6edbac7666 in socket_event_poll_in (this=this@entry=0x7f6ed40e3870) at socket.c:2237 #35 0x00007f6edbaca2c4 in socket_event_handler (fd=fd@entry=40, idx=idx@entry=4, data=0x7f6ed40e3870, poll_in=1, poll_out=0, poll_err=0) at socket.c:2350 #36 0x00007f6ee76229ca in event_dispatch_epoll_handler (event=0x7f6ee3cd7d80, event_pool=0x7f6f018bf130) at event-epoll.c:575 #37 event_dispatch_epoll_worker (data=0x7f6edc000920) at event-epoll.c:678 #38 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee3cd8700) at pthread_create.c:308 #39 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7f6ee1dfb700 (LWP 5242)): #0 0x00007f6efcf83863 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f6ee7622820 in event_dispatch_epoll_worker (data=0x7f6ed402f6a0) at event-epoll.c:668 #2 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee1dfb700) at pthread_create.c:308 #3 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7f6f01321880 (LWP 5236)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f6ee7606a2b in syncop_lookup (subvol=subvol@entry=0x7f6ed40a06e0, loc=loc@entry=0x7ffd0405f6f0, iatt=iatt@entry=0x7ffd0405f730, parent=parent@entry=0x0, xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:1227 #2 0x00007f6ee7caea23 in glfs_refresh_inode_safe (subvol=subvol@entry=0x7f6ed40a06e0, oldinode=oldinode@entry=0x7f6ed9ba106c, need_lookup=need_lookup@entry=_gf_false) at glfs-resolve.c:120 #3 0x00007f6ee7caebf8 in __glfs_refresh_inode (fs=fs@entry=0x7f6f018a0510, subvol=0x7f6ed40a06e0, inode=inode@entry=0x7f6ed9ba106c, need_lookup=need_lookup@entry=_gf_false) at glfs-resolve.c:154 #4 0x00007f6ee7cb0526 in __glfs_cwd_set (fs=fs@entry=0x7f6f018a0510, inode=inode@entry=0x7f6ed9ba106c) at glfs-resolve.c:933 #5 0x00007f6ee7cb061b in __glfs_active_subvol (fs=fs@entry=0x7f6f018a0510) at glfs-resolve.c:863 #6 0x00007f6ee7cb08a5 in priv_glfs_active_subvol (fs=0x7f6f018a0510) at glfs-resolve.c:908 #7 0x00007f6ee7ca7fc0 in pub_glfs_pwritev (glfd=0x7f6f01903f60, iovec=iovec@entry=0x7ffd0405f9f0, iovcnt=iovcnt@entry=1, offset=52035584, flags=0) at glfs-fops.c:1018 #8 0x00007f6ee7ca82da in pub_glfs_pwrite (glfd=<optimized out>, buf=<optimized out>, count=<optimized out>, offset=<optimized out>, flags=<optimized out>) at glfs-fops.c:1133 #9 0x00007f6f008a607c in vfs_pwrite_data (req=req@entry=0x7f6f01904400, fsp=fsp@entry=0x7f6f01903500, buffer=buffer@entry=0x7f6f01906454 "", N=N@entry=65536, offset=offset@entry=52035584) at ../source3/smbd/vfs.c:502 #10 0x00007f6f008453be in real_write_file (req=req@entry=0x7f6f01904400, fsp=fsp@entry=0x7f6f01903500, data=data@entry=0x7f6f01906454 "", pos=pos@entry=52035584, n=n@entry=65536) at ../source3/smbd/fileio.c:128 #11 0x00007f6f0084601c in write_file (req=req@entry=0x7f6f01904400, fsp=fsp@entry=0x7f6f01903500, data=data@entry=0x7f6f01906454 "", pos=52035584, n=n@entry=65536) at ../source3/smbd/fileio.c:381 #12 0x00007f6f0086e50b in reply_write_and_X (req=req@entry=0x7f6f01904400) at ../source3/smbd/reply.c:5082 #13 0x00007f6f008b6bc0 in switch_message (type=<optimized out>, req=req@entry=0x7f6f01904400) at ../source3/smbd/process.c:1726 #14 0x00007f6f008b8960 in construct_reply (deferred_pcd=0x0, encrypted=false, seqnum=0, unread_bytes=0, size=65604, inbuf=0x0, xconn=0x7f6f0188a640) at ../source3/smbd/process.c:1762 #15 process_smb (xconn=xconn@entry=0x7f6f0188f530, inbuf=<optimized out>, nread=65604, unread_bytes=0, seqnum=0, encrypted=<optimized out>, deferred_pcd=deferred_pcd@entry=0x0) at ../source3/smbd/process.c:2008 #16 0x00007f6f008ba051 in smbd_server_connection_read_handler (xconn=0x7f6f0188f530, fd=36) at ../source3/smbd/process.c:2607 #17 0x00007f6efe82b17c in run_events_poll (ev=0x7f6f01872dd0, pollrtn=<optimized out>, pfds=0x7f6f0188a1f0, num_pfds=4) at ../source3/lib/events.c:257 #18 0x00007f6efe82b3d0 in s3_event_loop_once (ev=0x7f6f01872dd0, location=<optimized out>) at ../source3/lib/events.c:326 #19 0x00007f6efd25236d in _tevent_loop_once (ev=ev@entry=0x7f6f01872dd0, location=location@entry=0x7f6f00a04c20 "../source3/smbd/process.c:4117") at ../tevent.c:533 #20 0x00007f6efd25250b in tevent_common_loop_wait (ev=0x7f6f01872dd0, location=0x7f6f00a04c20 "../source3/smbd/process.c:4117") at ../tevent.c:637 #21 0x00007f6f008bb3a1 in smbd_process (ev_ctx=ev_ctx@entry=0x7f6f01872dd0, msg_ctx=msg_ctx@entry=0x7f6f01872ec0, sock_fd=sock_fd@entry=36, interactive=interactive@entry=false) at ../source3/smbd/process.c:4117 #22 0x00007f6f013a30b8 in smbd_accept_connection (ev=0x7f6f01872dd0, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:747 #23 0x00007f6efe82b17c in run_events_poll (ev=0x7f6f01872dd0, pollrtn=<optimized out>, pfds=0x7f6f0188a1f0, num_pfds=6) at ../source3/lib/events.c:257 #24 0x00007f6efe82b3d0 in s3_event_loop_once (ev=0x7f6f01872dd0, location=<optimized out>) at ../source3/lib/events.c:326 #25 0x00007f6efd25236d in _tevent_loop_once (ev=ev@entry=0x7f6f01872dd0, location=location@entry=0x7f6f013a651a "../source3/smbd/server.c:1112") at ../tevent.c:533 #26 0x00007f6efd25250b in tevent_common_loop_wait (ev=0x7f6f01872dd0, location=0x7f6f013a651a "../source3/smbd/server.c:1112") at ../tevent.c:637 #27 0x00007f6f0139ec33 in smbd_parent_loop (parent=<optimized out>, ev_ctx=0x7f6f01872dd0) at ../source3/smbd/server.c:1112 #28 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:1765 --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-29 06:36:42 EDT --- Since this bug has been approved for the z-stream release of Red Hat Gluster Storage 3, through release flag 'rhgs-3.1.z+', and has been marked for RHGS 3.1 Update 3 release through the Internal Whiteboard entry of '3.1.3', the Target Release is being automatically set to 'RHGS 3.1.3' --- Additional comment from Vijay Bellur on 2016-05-02 07:31:59 EDT --- REVIEW: http://review.gluster.org/14148 (gfapi: Fix the synctask deadlock) posted (#1) for review on master by Poornima G (pgurusid) --- Additional comment from Vijay Bellur on 2016-05-02 07:55:37 EDT --- REVIEW: http://review.gluster.org/14148 (gfapi: Fix the hang caused, when a graph switch is triggered, while async io is going on) posted (#2) for review on master by Poornima G (pgurusid) --- Additional comment from Vijay Bellur on 2016-05-05 03:10:30 EDT --- REVIEW: http://review.gluster.org/14148 (gfapi: Fix a deadlock caused by graph switch while aio in progress) posted (#3) for review on master by Poornima G (pgurusid) --- Additional comment from Vijay Bellur on 2016-05-05 03:16:42 EDT --- REVIEW: http://review.gluster.org/14221 (gfapi: Fix a deadlock caused by graph switch while aio in progress) posted (#1) for review on release-3.8 by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14223 (gfapi: Fix a deadlock caused by graph switch while aio in progress) posted (#1) for review on release-3.7 by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/14223 committed in release-3.7 by Atin Mukherjee (amukherj) ------ commit 3639c84d9a3b7e3e490c0b87964d33422ba922a9 Author: Poornima G <pgurusid> Date: Fri Apr 29 12:24:24 2016 -0400 gfapi: Fix a deadlock caused by graph switch while aio in progress RCA: Currently async nature is achieved by submitting a syncop operation to synctask threads. Consider a scenario where the graph switch is triggered, the next write fop checks for the next available graph and sets fs->migration_in_progess and triggers the migration of fds and other things, which can cause some syncop_lookup operation. While this fop (on synctask thread) is waiting for syncop_lookup to return, lets say there are another 17 write async calls submitted, all these writes are blocked waiting for fs->migration_in_progress to be unset, hence all the 16 synctask threads are blocked waiting for fs->migration_in_progress to be unset. Now the syncop_lookup returns, but there are no synctask threads to process the lookup_cbk. If this syncop_lookup doesn't return, then fs->migration_in_progress can not be unset by the first fop. Thus causing a deadlock. To fix this deadlock, changing all the async APIs to use STACK_WIND, instead of syntask to achieve async nature. glfs_preadv_async is already implemented using STACK_WIND, now changing all the other async APIs also to do the same. This patch as such will not reduce the performance of async IO, the only thing that can affect is that, in case of write, the buf passed by application is copied onto iobuf in the same thread wheras before it was being copied in synctask thread. Since, the syncop + graph switch logic (lock across fops) is not a good candidate for synctask, changing the async APIs to use STACK_WIND Backport of http://review.gluster.org/#/c/14148/ Change-Id: Idf665cae0a8e27697fbfc5ec8d93a6d6bae3a4f1 BUG: 1333268 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/14223 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra Talur <rtalur> Reviewed-by: Rajesh Joseph <rjoseph> Reviewed-by: Atin Mukherjee <amukherj>
This patch causes a problem in several places where glfs_preadv_async_cbk is specifically called with a NULL iovec. 0 glfs-fops.c glfs_preadv_async_cbk 854 glfs_io_async_cbk (op_ret, op_errno, frame, cookie, iovec, count); 1 glfs-fops.c glfs_pwritev_async_cbk 1168 glfs_io_async_cbk (op_ret, op_errno, frame, cookie, NULL, 0); 2 glfs-fops.c glfs_fsync_async_cbk 1367 glfs_io_async_cbk (op_ret, op_errno, frame, cookie, NULL, 0); 3 glfs-fops.c glfs_ftruncate_async_cbk 1573 glfs_io_async_cbk (op_ret, op_errno, frame, cookie, NULL, 0); 4 glfs-fops.c glfs_discard_async_cbk 2429 glfs_io_async_cbk (op_ret, op_errno, frame, cookie, NULL, 0); 5 glfs-fops.c glfs_zerofill_async_cbk 2514 glfs_io_async_cbk (op_ret, op_errno, frame, cookie, NULL, 0);
Meant to continue to say it causes a problem because glfs_preadv_async_cbk does: GF_VALIDATE_OR_GOTO ("gfapi", iovec, inval); Since iovec is NULL, this fails.
Thanks for reporting this, glfs_io_async_cbk() should not check for GF_VALIDATE_OR_GOTO ("gfapi", iovec, inval);, as fsync and other fops will send iovec as NULL, But you mentioned read is returning ioec as NULL, which shouldn't happen? I will send the patch to fix the same
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.12, please open a new bug report. glusterfs-3.7.12 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://www.gluster.org/pipermail/gluster-devel/2016-June/049918.html [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.