Description of problem: *************************** While running I/O's on cifs mount and changing the graph by disabling and enabling stat-prefetch , the cifs mount hangs and there are errors seen in the log. Version-Release number of selected component (if applicable): glusterfs-3.7.9-1.el7rhgs.x86_64 samba-4.4.0-2.el7rhgs.x86_64 How reproducible: tried once Steps to Reproduce: 1.Create a volume and mount it on linux cifs client 2.start dd on the mount point "dd if=/dev/zero of=/mnt/cifs/file.dd bs=1M count=1024" 3.On the server do stat-prefetch off , on and off 4. check the mount point, the i/o's are hung. Actual results: The i/o is hung and there are errors in the logs. Expected results: The i/o should not get hung. Additional info: [2016-04-19 07:37:01.717797] E [dht-helper.c:1592:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.9/xlator/protocol/client.so(client3_3_lookup _cbk+0x707) [0x7ff26f259417] -->/usr/lib64/glusterfs/3.7.9/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7ff26efed269] -->/usr/lib64/glust erfs/3.7.9/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7ff26efca680] ) 76-batman-dht: invalid argument: inode [Invalid argument] [2016-04-19 07:33:22.400959] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 76-batman-client-0: remote operation failed. Path: / (000 00000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
Here is the bt of all threads: Thread 21 (Thread 0x7ff272aec700 (LWP 4938)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f53d4c0, iovcnt=1, offset=25624576, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 20 (Thread 0x7ff2722eb700 (LWP 4939)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fea1f70, iovcnt=1, offset=26083328, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 19 (Thread 0x7ff271274700 (LWP 4940)): #0 0x00007ff28d39d96d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff2739979ec in gf_timer_proc (ctx=0x7ff28f540a90) at timer.c:193 #2 0x00007ff28d396dc5 in start_thread (arg=0x7ff271274700) at pthread_create.c:308 #3 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 18 (Thread 0x7ff270870700 (LWP 4941)): #0 0x00007ff28d397ef7 in pthread_join (threadid=140679236122368, thread_return=thread_return@entry=0x0) at pthread_join.c:92 #1 0x00007ff2739d9d28 in event_dispatch_epoll (event_pool=0x7ff28f55f550) at event-epoll.c:762 #2 0x00007ff274058614 in glfs_poller (data=<optimized out>) at glfs.c:587 #3 0x00007ff28d396dc5 in start_thread (arg=0x7ff270870700) at pthread_create.c:308 #4 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 17 (Thread 0x7ff26fe57700 (LWP 4942)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff27406747b in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_subvol_done (fs=fs@entry=0x7ff28f540930, subvol=subvol@entry=0x7ff2600866d0) at glfs-resolve.c:884 #3 0x00007ff270871de1 in graph_setup (fs=fs@entry=0x7ff28f540930, graph=graph@entry=0x7ff260075630) at glfs-master.c:75 #4 0x00007ff270871f1b in notify (this=<optimized out>, event=<optimized out>, data=0x7ff260075630) at glfs-master.c:103 #5 0x00007ff273975906 in xlator_notify (xl=0x7ff28f570b30, event=5, data=0x7ff260075630) at xlator.c:496 #6 0x00007ff27398c2fe in default_notify (this=0x7ff2602d2620, event=5, data=<optimized out>) at defaults.c:2316 #7 0x00007ff273975906 in xlator_notify (xl=0x7ff2602d2620, event=event@entry=5, data=data@entry=0x7ff2602d1820) at xlator.c:496 #8 0x00007ff27398c1f3 in default_notify (this=0x7ff2602d1820, event=5, data=<optimized out>) at defaults.c:2320 #9 0x00007ff26e34fbe9 in notify (this=0x7ff2602d1820, event=<optimized out>, data=0x7ff2602d0b40) at io-stats.c:3064 #10 0x00007ff273975906 in xlator_notify (xl=0x7ff2602d1820, event=event@entry=5, data=data@entry=0x7ff2602d0b40) at xlator.c:496 #11 0x00007ff27398c1f3 in default_notify (this=0x7ff2602d0b40, event=5, data=<optimized out>) at defaults.c:2320 #12 0x00007ff273975906 in xlator_notify (xl=0x7ff2602d0b40, event=event@entry=5, data=data@entry=0x7ff2602cfe70) at xlator.c:496 #13 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cfe70, event=5, data=<optimized out>) at defaults.c:2320 #14 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cfe70, event=event@entry=5, data=data@entry=0x7ff2602cf3d0) at xlator.c:496 #15 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cf3d0, event=5, data=<optimized out>) at defaults.c:2320 #16 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cf3d0, event=event@entry=5, data=data@entry=0x7ff2602ce930) at xlator.c:496 #17 0x00007ff27398c1f3 in default_notify (this=0x7ff2602ce930, event=5, data=<optimized out>) at defaults.c:2320 #18 0x00007ff273975906 in xlator_notify (xl=0x7ff2602ce930, event=event@entry=5, data=data@entry=0x7ff2602cde90) at xlator.c:496 #19 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cde90, event=5, data=<optimized out>) at defaults.c:2320 #20 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cde90, event=event@entry=5, data=data@entry=0x7ff2602cd3f0) at xlator.c:496 #21 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cd3f0, event=5, data=<optimized out>) at defaults.c:2320 #22 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cd3f0, event=event@entry=5, data=data@entry=0x7ff2602cc950) at xlator.c:496 #23 0x00007ff27398c1f3 in default_notify (this=0x7ff2602cc950, event=event@entry=5, data=data@entry=0x7ff2602cbeb0) at defaults.c:2320 #24 0x00007ff26f1e1b3f in dht_notify (this=<optimized out>, event=5, data=0x7ff2602cbeb0) at dht-common.c:7844 #25 0x00007ff273975906 in xlator_notify (xl=0x7ff2602cc950, event=event@entry=5, data=data@entry=0x7ff2602cbeb0) at xlator.c:496 #26 0x00007ff27398c1f3 in default_notify (this=this@entry=0x7ff2602cbeb0, event=event@entry=5, data=data@entry=0x0) at defaults.c:2320 #27 0x00007ff26f41f859 in client_notify_dispatch (this=this@entry=0x7ff2602cbeb0, event=event@entry=5, data=data@entry=0x0) at client.c:88 #28 0x00007ff26f444406 in client_notify_parents_child_up (this=0x7ff2602cbeb0) at client-handshake.c:138 #29 0x00007ff26f44615e in client_post_handshake (frame=frame@entry=0x7ff271a40560, this=0x7ff2602cbeb0) at client-handshake.c:1069 #30 0x00007ff26f446894 in client_setvolume_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7ff271a40560) at client-handshake.c:1226 #31 0x00007ff273e46a20 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7ff2604b2d00, pollin=pollin@entry=0x7ff2602c33d0) at rpc-clnt.c:764 #32 0x00007ff273e46cdf in rpc_clnt_notify (trans=<optimized out>, mydata=0x7ff2604b2d30, event=<optimized out>, data=0x7ff2602c33d0) at rpc-clnt.c:905 #33 0x00007ff273e42823 in rpc_transport_notify (this=this@entry=0x7ff26050c230, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7ff2602c33d0) at rpc-transport.c:546 #34 0x00007ff26fe5e666 in socket_event_poll_in (this=this@entry=0x7ff26050c230) at socket.c:2237 #35 0x00007ff26fe612c4 in socket_event_handler (fd=fd@entry=44, idx=idx@entry=4, data=0x7ff26050c230, poll_in=1, poll_out=0, poll_err=0) at socket.c:2350 #36 0x00007ff2739d99ca in event_dispatch_epoll_handler (event=0x7ff26fe56d80, event_pool=0x7ff28f55f550) at event-epoll.c:575 #37 event_dispatch_epoll_worker (data=0x7ff268000920) at event-epoll.c:678 #38 0x00007ff28d396dc5 in start_thread (arg=0x7ff26fe57700) at pthread_create.c:308 #39 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 16 (Thread 0x7ff26df7a700 (LWP 4943)): #0 0x00007ff2893bb863 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff2739d9820 in event_dispatch_epoll_worker (data=0x7ff26002f6a0) at event-epoll.c:668 #2 0x00007ff28d396dc5 in start_thread (arg=0x7ff26df7a700) at pthread_create.c:308 #3 0x00007ff2893bb28d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7ff25f57e700 (LWP 4970)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f53c5c0, iovcnt=1, offset=25427968, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 14 (Thread 0x7ff25ed7d700 (LWP 4971)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe44500, iovcnt=1, offset=25886720, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 13 (Thread 0x7ff25e57c700 (LWP 4974)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f59fe70, iovcnt=1, offset=25493504, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 12 (Thread 0x7ff25dd7b700 (LWP 4976)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28feb3990, iovcnt=1, offset=26411008, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 11 (Thread 0x7ff25d57a700 (LWP 4977)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5a1300, iovcnt=1, offset=25690112, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 10 (Thread 0x7ff25cd79700 (LWP 4978)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fea1e80, iovcnt=1, offset=26017792, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 9 (Thread 0x7ff23ffff700 (LWP 4981)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe29de0, iovcnt=1, offset=26279936, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 8 (Thread 0x7ff23f7fe700 (LWP 4985)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5f4e20, iovcnt=1, offset=25952256, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 7 (Thread 0x7ff23effd700 (LWP 4994)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5c92a0, iovcnt=1, offset=25362432, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 6 (Thread 0x7ff23e7fc700 (LWP 4995)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe29cf0, iovcnt=1, offset=26214400, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 5 (Thread 0x7ff23dffb700 (LWP 4996)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28fe29c00, iovcnt=1, offset=26148864, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 4 (Thread 0x7ff23d7fa700 (LWP 4997)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5a40f0, iovcnt=1, offset=25821184, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 3 (Thread 0x7ff23cff9700 (LWP 4998)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f572020, iovcnt=1, offset=25755648, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 2 (Thread 0x7ff22bfff700 (LWP 4999)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405efc0 in pub_glfs_pwritev (glfd=0x7ff28f5a0110, iovec=0x7ff28f5a4590, iovcnt=1, offset=25559040, flags=0) at glfs-fops.c:1018 #4 0x00007ff2740642fb in glfs_io_async_task (data=<optimized out>) at glfs-fops.c:784 #5 0x00007ff2739b9ef2 in synctask_wrap (old_task=<optimized out>) at syncop.c:380 #6 0x00007ff28930c110 in ?? () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () Thread 1 (Thread 0x7ff28d759880 (LWP 4937)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ff274067893 in glfs_lock (fs=0x7ff28f540930) at glfs-internal.h:312 #2 priv_glfs_active_subvol (fs=0x7ff28f540930) at glfs-resolve.c:906 #3 0x00007ff27405d1cd in pub_glfs_close (glfd=0x7ff28f5a0110) at glfs-fops.c:251 #4 0x00007ff28ccce870 in fd_close (fsp=fsp@entry=0x7ff28f5a2b10) at ../source3/smbd/open.c:431 #5 0x00007ff28ccd7b65 in close_normal_file (close_type=SHUTDOWN_CLOSE, fsp=0x7ff28f5a2b10, req=0x0) at ../source3/smbd/close.c:753 #6 close_file (req=req@entry=0x0, fsp=0x7ff28f5a2b10, close_type=close_type@entry=SHUTDOWN_CLOSE) at ../source3/smbd/close.c:1220 #7 0x00007ff28cc75fca in file_close_conn (conn=conn@entry=0x7ff28f532a40) at ../source3/smbd/files.c:165 #8 0x00007ff28ccf5d59 in close_cnum (conn=0x7ff28f532a40, vuid=49446) at ../source3/smbd/service.c:1128 #9 0x00007ff28cd240a4 in smbXsrv_tcon_disconnect (tcon=0x7ff28f533a40, vuid=49446) at ../source3/smbd/smbXsrv_tcon.c:983 #10 0x00007ff28cd24372 in smbXsrv_tcon_disconnect_all_callback (local_rec=local_rec@entry=0x7ffd268981b0, private_data=private_data@entry=0x7ffd26898280) at ../source3/smbd/smbXsrv_tcon.c:1064 #11 0x00007ff28736cc2a in db_rbt_traverse_internal (db=db@entry=0x7ff28f52cc90, f=f@entry=0x7ff28cd24300 <smbXsrv_tcon_disconnect_all_callback>, private_data=private_data@entry=0x7ffd26898280, count=count@entry=0x7ffd2689823c, rw=rw@entry=true) at ../lib/dbwrap/dbwrap_rbt.c:434 #12 0x00007ff28736cd67 in db_rbt_traverse (db=0x7ff28f52cc90, f=0x7ff28cd24300 <smbXsrv_tcon_disconnect_all_callback>, private_data=0x7ffd26898280) at ../lib/dbwrap/dbwrap_rbt.c:492 #13 0x00007ff28736b20a in dbwrap_traverse (db=<optimized out>, f=f@entry=0x7ff28cd24300 <smbXsrv_tcon_disconnect_all_callback>, private_data=private_data@entry=0x7ffd26898280, count=count@entry=0x7ffd2689827c) at ../lib/dbwrap/dbwrap.c:343 #14 0x00007ff28cd230a9 in smbXsrv_tcon_disconnect_all (table=<optimized out>, vuid=vuid@entry=0) at ../source3/smbd/smbXsrv_tcon.c:1013 #15 0x00007ff28cd24463 in smb1srv_tcon_disconnect_all (conn=conn@entry=0x7ff28f514d50) at ../source3/smbd/smbXsrv_tcon.c:1132 #16 0x00007ff28cd27202 in exit_server_common (how=how@entry=SERVER_EXIT_NORMAL, reason=0x7ff28ce3a214 "failed to receive smb request") at ../source3/smbd/server_exit.c:145 #17 0x00007ff28cd2777e in smbd_exit_server_cleanly (explanation=<optimized out>) at ../source3/smbd/server_exit.c:263 #18 0x00007ff28a603ba2 in exit_server_cleanly (reason=reason@entry=0x7ff28ce3a214 "failed to receive smb request") at ../source3/lib/smbd_shim.c:131 #19 0x00007ff28ccf2114 in smbd_server_connection_read_handler (xconn=0x7ff28f514d50, fd=36) at ../source3/smbd/process.c:2600 #20 0x00007ff28ac6317c in run_events_poll (ev=0x7ff28f511eb0, pollrtn=<optimized out>, pfds=0x7ff28f5292d0, num_pfds=5) at ../source3/lib/events.c:257 #21 0x00007ff28ac633d0 in s3_event_loop_once (ev=0x7ff28f511eb0, location=<optimized out>) at ../source3/lib/events.c:326 #22 0x00007ff28968a36d in _tevent_loop_once (ev=ev@entry=0x7ff28f511eb0, location=location@entry=0x7ff28ce3cc20 "../source3/smbd/process.c:4117") at ../tevent.c:533 #23 0x00007ff28968a50b in tevent_common_loop_wait (ev=0x7ff28f511eb0, location=0x7ff28ce3cc20 "../source3/smbd/process.c:4117") at ../tevent.c:637 #24 0x00007ff28ccf33a1 in smbd_process (ev_ctx=ev_ctx@entry=0x7ff28f511eb0, msg_ctx=msg_ctx@entry=0x7ff28f511fa0, sock_fd=sock_fd@entry=36, interactive=interactive@entry=false) at ../source3/smbd/process.c:4117 #25 0x00007ff28d7db0b8 in smbd_accept_connection (ev=0x7ff28f511eb0, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:747 #26 0x00007ff28ac6317c in run_events_poll (ev=0x7ff28f511eb0, pollrtn=<optimized out>, pfds=0x7ff28f5292d0, num_pfds=6) at ../source3/lib/events.c:257 #27 0x00007ff28ac633d0 in s3_event_loop_once (ev=0x7ff28f511eb0, location=<optimized out>) at ../source3/lib/events.c:326 #28 0x00007ff28968a36d in _tevent_loop_once (ev=ev@entry=0x7ff28f511eb0, location=location@entry=0x7ff28d7de51a "../source3/smbd/server.c:1112") at ../tevent.c:533 #29 0x00007ff28968a50b in tevent_common_loop_wait (ev=0x7ff28f511eb0, location=0x7ff28d7de51a "../source3/smbd/server.c:1112") at ../tevent.c:637 #30 0x00007ff28d7d6c33 in smbd_parent_loop (parent=<optimized out>, ev_ctx=0x7ff28f511eb0) at ../source3/smbd/server.c:1112 #31 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:1765
Even without AIO there is a hang: [root@dhcp46-40 glusterfs]# pstack 5236 Thread 7 (Thread 0x7f6ee6755700 (LWP 5237)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f6ee7605008 in syncenv_task (proc=proc@entry=0x7f6f018c2320) at syncop.c:607 #2 0x00007f6ee7605d40 in syncenv_processor (thdata=0x7f6f018c2320) at syncop.c:699 #3 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee6755700) at pthread_create.c:308 #4 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 6 (Thread 0x7f6ee5f54700 (LWP 5238)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f6ee7605008 in syncenv_task (proc=proc@entry=0x7f6f018c26e0) at syncop.c:607 #2 0x00007f6ee7605d40 in syncenv_processor (thdata=0x7f6f018c26e0) at syncop.c:699 #3 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee5f54700) at pthread_create.c:308 #4 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 5 (Thread 0x7f6ee4edd700 (LWP 5239)): #0 0x00007f6f00f6596d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f6ee75e09ec in gf_timer_proc (ctx=0x7f6f018a0670) at timer.c:193 #2 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee4edd700) at pthread_create.c:308 #3 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f6ee44d9700 (LWP 5240)): #0 0x00007f6f00f5fef7 in pthread_join (threadid=140114245027584, thread_return=thread_return@entry=0x0) at pthread_join.c:92 #1 0x00007f6ee7622d28 in event_dispatch_epoll (event_pool=0x7f6f018bf130) at event-epoll.c:762 #2 0x00007f6ee7ca1614 in glfs_poller (data=<optimized out>) at glfs.c:587 #3 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee44d9700) at pthread_create.c:308 #4 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f6ee3cd8700 (LWP 5241)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f6ee7cb047b in glfs_lock (fs=0x7f6f018a0510) at glfs-internal.h:312 #2 priv_glfs_subvol_done (fs=fs@entry=0x7f6f018a0510, subvol=subvol@entry=0x7f6ecc0133a0) at glfs-resolve.c:884 #3 0x00007f6ee44dade1 in graph_setup (fs=fs@entry=0x7f6f018a0510, graph=graph@entry=0x7f6ed4091930) at glfs-master.c:75 #4 0x00007f6ee44daf1b in notify (this=<optimized out>, event=<optimized out>, data=0x7f6ed4091930) at glfs-master.c:103 #5 0x00007f6ee75be906 in xlator_notify (xl=0x7f6f018d0770, event=5, data=0x7f6ed4091930) at xlator.c:496 #6 0x00007f6ee75d52fe in default_notify (this=0x7f6ed40a06e0, event=5, data=<optimized out>) at defaults.c:2316 #7 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed40a06e0, event=event@entry=5, data=data@entry=0x7f6ed409f8e0) at xlator.c:496 #8 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409f8e0, event=5, data=<optimized out>) at defaults.c:2320 #9 0x00007f6ee21d0be9 in notify (this=0x7f6ed409f8e0, event=<optimized out>, data=0x7f6ed409ec00) at io-stats.c:3064 #10 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409f8e0, event=event@entry=5, data=data@entry=0x7f6ed409ec00) at xlator.c:496 #11 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409ec00, event=5, data=<optimized out>) at defaults.c:2320 #12 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409ec00, event=event@entry=5, data=data@entry=0x7f6ed409df30) at xlator.c:496 #13 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409df30, event=5, data=<optimized out>) at defaults.c:2320 #14 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409df30, event=event@entry=5, data=data@entry=0x7f6ed409d490) at xlator.c:496 #15 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409d490, event=5, data=<optimized out>) at defaults.c:2320 #16 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409d490, event=event@entry=5, data=data@entry=0x7f6ed409c9f0) at xlator.c:496 #17 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409c9f0, event=5, data=<optimized out>) at defaults.c:2320 #18 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409c9f0, event=event@entry=5, data=data@entry=0x7f6ed409bf50) at xlator.c:496 #19 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409bf50, event=5, data=<optimized out>) at defaults.c:2320 #20 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409bf50, event=event@entry=5, data=data@entry=0x7f6ed409b4b0) at xlator.c:496 #21 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409b4b0, event=5, data=<optimized out>) at defaults.c:2320 #22 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409b4b0, event=event@entry=5, data=data@entry=0x7f6ed409aa10) at xlator.c:496 #23 0x00007f6ee75d51f3 in default_notify (this=0x7f6ed409aa10, event=event@entry=5, data=data@entry=0x7f6ed4099f70) at defaults.c:2320 #24 0x00007f6ee3062b3f in dht_notify (this=<optimized out>, event=5, data=0x7f6ed4099f70) at dht-common.c:7844 #25 0x00007f6ee75be906 in xlator_notify (xl=0x7f6ed409aa10, event=event@entry=5, data=data@entry=0x7f6ed4099f70) at xlator.c:496 #26 0x00007f6ee75d51f3 in default_notify (this=this@entry=0x7f6ed4099f70, event=event@entry=5, data=data@entry=0x0) at defaults.c:2320 #27 0x00007f6ee32a0859 in client_notify_dispatch (this=this@entry=0x7f6ed4099f70, event=event@entry=5, data=data@entry=0x0) at client.c:88 #28 0x00007f6ee32c5406 in client_notify_parents_child_up (this=0x7f6ed4099f70) at client-handshake.c:138 #29 0x00007f6ee32c715e in client_post_handshake (frame=frame@entry=0x7f6ee56a77d0, this=0x7f6ed4099f70) at client-handshake.c:1069 #30 0x00007f6ee32c7894 in client_setvolume_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7f6ee56a77d0) at client-handshake.c:1226 #31 0x00007f6ee7a8fa20 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f6ed40d3b90, pollin=pollin@entry=0x7f6ed4091470) at rpc-clnt.c:764 #32 0x00007f6ee7a8fcdf in rpc_clnt_notify (trans=<optimized out>, mydata=0x7f6ed40d3bc0, event=<optimized out>, data=0x7f6ed4091470) at rpc-clnt.c:905 #33 0x00007f6ee7a8b823 in rpc_transport_notify (this=this@entry=0x7f6ed40e3870, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7f6ed4091470) at rpc-transport.c:546 #34 0x00007f6edbac7666 in socket_event_poll_in (this=this@entry=0x7f6ed40e3870) at socket.c:2237 #35 0x00007f6edbaca2c4 in socket_event_handler (fd=fd@entry=40, idx=idx@entry=4, data=0x7f6ed40e3870, poll_in=1, poll_out=0, poll_err=0) at socket.c:2350 #36 0x00007f6ee76229ca in event_dispatch_epoll_handler (event=0x7f6ee3cd7d80, event_pool=0x7f6f018bf130) at event-epoll.c:575 #37 event_dispatch_epoll_worker (data=0x7f6edc000920) at event-epoll.c:678 #38 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee3cd8700) at pthread_create.c:308 #39 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7f6ee1dfb700 (LWP 5242)): #0 0x00007f6efcf83863 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f6ee7622820 in event_dispatch_epoll_worker (data=0x7f6ed402f6a0) at event-epoll.c:668 #2 0x00007f6f00f5edc5 in start_thread (arg=0x7f6ee1dfb700) at pthread_create.c:308 #3 0x00007f6efcf8328d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7f6f01321880 (LWP 5236)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f6ee7606a2b in syncop_lookup (subvol=subvol@entry=0x7f6ed40a06e0, loc=loc@entry=0x7ffd0405f6f0, iatt=iatt@entry=0x7ffd0405f730, parent=parent@entry=0x0, xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at syncop.c:1227 #2 0x00007f6ee7caea23 in glfs_refresh_inode_safe (subvol=subvol@entry=0x7f6ed40a06e0, oldinode=oldinode@entry=0x7f6ed9ba106c, need_lookup=need_lookup@entry=_gf_false) at glfs-resolve.c:120 #3 0x00007f6ee7caebf8 in __glfs_refresh_inode (fs=fs@entry=0x7f6f018a0510, subvol=0x7f6ed40a06e0, inode=inode@entry=0x7f6ed9ba106c, need_lookup=need_lookup@entry=_gf_false) at glfs-resolve.c:154 #4 0x00007f6ee7cb0526 in __glfs_cwd_set (fs=fs@entry=0x7f6f018a0510, inode=inode@entry=0x7f6ed9ba106c) at glfs-resolve.c:933 #5 0x00007f6ee7cb061b in __glfs_active_subvol (fs=fs@entry=0x7f6f018a0510) at glfs-resolve.c:863 #6 0x00007f6ee7cb08a5 in priv_glfs_active_subvol (fs=0x7f6f018a0510) at glfs-resolve.c:908 #7 0x00007f6ee7ca7fc0 in pub_glfs_pwritev (glfd=0x7f6f01903f60, iovec=iovec@entry=0x7ffd0405f9f0, iovcnt=iovcnt@entry=1, offset=52035584, flags=0) at glfs-fops.c:1018 #8 0x00007f6ee7ca82da in pub_glfs_pwrite (glfd=<optimized out>, buf=<optimized out>, count=<optimized out>, offset=<optimized out>, flags=<optimized out>) at glfs-fops.c:1133 #9 0x00007f6f008a607c in vfs_pwrite_data (req=req@entry=0x7f6f01904400, fsp=fsp@entry=0x7f6f01903500, buffer=buffer@entry=0x7f6f01906454 "", N=N@entry=65536, offset=offset@entry=52035584) at ../source3/smbd/vfs.c:502 #10 0x00007f6f008453be in real_write_file (req=req@entry=0x7f6f01904400, fsp=fsp@entry=0x7f6f01903500, data=data@entry=0x7f6f01906454 "", pos=pos@entry=52035584, n=n@entry=65536) at ../source3/smbd/fileio.c:128 #11 0x00007f6f0084601c in write_file (req=req@entry=0x7f6f01904400, fsp=fsp@entry=0x7f6f01903500, data=data@entry=0x7f6f01906454 "", pos=52035584, n=n@entry=65536) at ../source3/smbd/fileio.c:381 #12 0x00007f6f0086e50b in reply_write_and_X (req=req@entry=0x7f6f01904400) at ../source3/smbd/reply.c:5082 #13 0x00007f6f008b6bc0 in switch_message (type=<optimized out>, req=req@entry=0x7f6f01904400) at ../source3/smbd/process.c:1726 #14 0x00007f6f008b8960 in construct_reply (deferred_pcd=0x0, encrypted=false, seqnum=0, unread_bytes=0, size=65604, inbuf=0x0, xconn=0x7f6f0188a640) at ../source3/smbd/process.c:1762 #15 process_smb (xconn=xconn@entry=0x7f6f0188f530, inbuf=<optimized out>, nread=65604, unread_bytes=0, seqnum=0, encrypted=<optimized out>, deferred_pcd=deferred_pcd@entry=0x0) at ../source3/smbd/process.c:2008 #16 0x00007f6f008ba051 in smbd_server_connection_read_handler (xconn=0x7f6f0188f530, fd=36) at ../source3/smbd/process.c:2607 #17 0x00007f6efe82b17c in run_events_poll (ev=0x7f6f01872dd0, pollrtn=<optimized out>, pfds=0x7f6f0188a1f0, num_pfds=4) at ../source3/lib/events.c:257 #18 0x00007f6efe82b3d0 in s3_event_loop_once (ev=0x7f6f01872dd0, location=<optimized out>) at ../source3/lib/events.c:326 #19 0x00007f6efd25236d in _tevent_loop_once (ev=ev@entry=0x7f6f01872dd0, location=location@entry=0x7f6f00a04c20 "../source3/smbd/process.c:4117") at ../tevent.c:533 #20 0x00007f6efd25250b in tevent_common_loop_wait (ev=0x7f6f01872dd0, location=0x7f6f00a04c20 "../source3/smbd/process.c:4117") at ../tevent.c:637 #21 0x00007f6f008bb3a1 in smbd_process (ev_ctx=ev_ctx@entry=0x7f6f01872dd0, msg_ctx=msg_ctx@entry=0x7f6f01872ec0, sock_fd=sock_fd@entry=36, interactive=interactive@entry=false) at ../source3/smbd/process.c:4117 #22 0x00007f6f013a30b8 in smbd_accept_connection (ev=0x7f6f01872dd0, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:747 #23 0x00007f6efe82b17c in run_events_poll (ev=0x7f6f01872dd0, pollrtn=<optimized out>, pfds=0x7f6f0188a1f0, num_pfds=6) at ../source3/lib/events.c:257 #24 0x00007f6efe82b3d0 in s3_event_loop_once (ev=0x7f6f01872dd0, location=<optimized out>) at ../source3/lib/events.c:326 #25 0x00007f6efd25236d in _tevent_loop_once (ev=ev@entry=0x7f6f01872dd0, location=location@entry=0x7f6f013a651a "../source3/smbd/server.c:1112") at ../tevent.c:533 #26 0x00007f6efd25250b in tevent_common_loop_wait (ev=0x7f6f01872dd0, location=0x7f6f013a651a "../source3/smbd/server.c:1112") at ../tevent.c:637 #27 0x00007f6f0139ec33 in smbd_parent_loop (parent=<optimized out>, ev_ctx=0x7f6f01872dd0) at ../source3/smbd/server.c:1112 #28 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:1765
Patch posted upstream master: http://review.gluster.org/14148
Started dd on the mount point "dd if=/dev/zero of=/mnt/cifs/file.dd bs=1M count=1024" On the server did stat-prefetch off , on and off The IO's and the mount point did not hung.Hence marking it as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240