Created attachment 827715 [details] strace from brick 1 Description of problem: Our test often fail in racy situation throwing I/O error, while replicating their workload I hit issue with rm failing on Directory not empty, same happends on our tests but on MKDIR. Version-Release number of selected component (if applicable): 3.4.1 How reproducible: Always Steps to Reproduce: 1. prepare distribute gluster mount on /media/test 2. while true; do rm -rf /media/test/* && umount /media/test && mount /media/test && mkdir -p /media/test/a/{a,b,c,d,g} || exit; done 3. wait Actual results: Client logs: [2013-11-22 12:06:23.221816] I [client-handshake.c:1468:client_setvolume_cbk] 0-Staging-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-11-22 12:06:23.221823] D [client-handshake.c:1318:client_post_handshake] 0-Staging-client-1: No fds to open - notifying all parents child up [2013-11-22 12:06:23.221832] D [client-handshake.c:486:client_set_lk_version] 0-Staging-client-1: Sending SET_LK_VERSION [2013-11-22 12:06:23.221938] D [fuse-bridge.c:4787:notify] 0-fuse: got event 5 on graph 0 [2013-11-22 12:06:23.227027] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-11-22 12:06:23.227230] I [client-handshake.c:450:client_set_lk_version_cbk] 0-Staging-client-1: Server lk version = 1 [2013-11-22 12:06:23.227321] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-1': avail_percent is: 87.00 and avail_space is: 15789551616 and avail_inodes is: 99.00 [2013-11-22 12:06:23.227586] D [fuse-bridge.c:4413:fuse_get_mount_status] 0-fuse: mount status is 0 [2013-11-22 12:06:23.227747] I [fuse-bridge.c:3724:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 [2013-11-22 12:06:23.783102] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:06:23.791998] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:06:23.794085] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:06:23.796023] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:06:23.797043] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-0': avail_percent is: 87.00 and avail_space is: 15789465600 and avail_inodes is: 99.00 [2013-11-22 12:06:23.797406] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-1': avail_percent is: 87.00 and avail_space is: 15789465600 and avail_inodes is: 99.00 [2013-11-22 12:06:23.842052] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/15: failed to resolve (No such file or directory) [2013-11-22 12:06:23.843965] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/15: failed to resolve (No such file or directory) [2013-11-22 12:06:23.846208] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/15: failed to resolve (No such file or directory) [2013-11-22 12:06:23.848101] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/15: failed to resolve (No such file or directory) [2013-11-22 12:06:23.849985] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/15: failed to resolve (No such file or directory) [2013-11-22 12:06:23.873940] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/18: failed to resolve (No such file or directory) [2013-11-22 12:06:23.875797] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/18: failed to resolve (No such file or directory) [2013-11-22 12:06:23.877502] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/18: failed to resolve (No such file or directory) [2013-11-22 12:06:23.879291] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/18: failed to resolve (No such file or directory) [2013-11-22 12:06:23.881671] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/18: failed to resolve (No such file or directory) [2013-11-22 12:06:23.904481] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/fb: failed to resolve (No such file or directory) [2013-11-22 12:06:23.906479] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/fb: failed to resolve (No such file or directory) [2013-11-22 12:06:23.908320] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/fb: failed to resolve (No such file or directory) [2013-11-22 12:06:23.910205] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/fb: failed to resolve (No such file or directory) [2013-11-22 12:06:23.912142] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/fb: failed to resolve (No such file or directory) [2013-11-22 12:06:23.938480] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/74: failed to resolve (No such file or directory) [2013-11-22 12:06:23.941339] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/74: failed to resolve (No such file or directory) [2013-11-22 12:06:23.943944] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/74: failed to resolve (No such file or directory) [2013-11-22 12:06:23.947805] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/74: failed to resolve (No such file or directory) [2013-11-22 12:06:23.951836] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/74: failed to resolve (No such file or directory) [2013-11-22 12:06:23.981733] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/6d: failed to resolve (No such file or directory) [2013-11-22 12:06:23.984948] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/6d: failed to resolve (No such file or directory) [2013-11-22 12:06:23.987365] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/6d: failed to resolve (No such file or directory) [2013-11-22 12:06:23.990465] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/6d: failed to resolve (No such file or directory) [2013-11-22 12:06:23.993170] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/6d: failed to resolve (No such file or directory) [2013-11-22 12:06:24.021581] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-1': avail_percent is: 87.00 and avail_space is: 15788797952 and avail_inodes is: 99.00 [2013-11-22 12:06:24.022231] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-0': avail_percent is: 87.00 and avail_space is: 15788793856 and avail_inodes is: 99.00 [2013-11-22 12:06:24.049626] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/ba: failed to resolve (No such file or directory) [2013-11-22 12:06:24.052554] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/ba: failed to resolve (No such file or directory) [2013-11-22 12:06:24.055023] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/ba: failed to resolve (No such file or directory) [2013-11-22 12:06:24.058083] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/ba: failed to resolve (No such file or directory) [2013-11-22 12:06:24.060904] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/ba: failed to resolve (No such file or directory) [2013-11-22 12:06:24.091706] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/59: failed to resolve (No such file or directory) [2013-11-22 12:06:24.094477] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/59: failed to resolve (No such file or directory) [2013-11-22 12:06:24.096942] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/59: failed to resolve (No such file or directory) [2013-11-22 12:06:24.099657] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/59: failed to resolve (No such file or directory) [2013-11-22 12:06:24.102270] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/59: failed to resolve (No such file or directory) [2013-11-22 12:06:24.131683] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/e7: failed to resolve (No such file or directory) [2013-11-22 12:06:24.134477] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/e7: failed to resolve (No such file or directory) [2013-11-22 12:06:24.137001] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/e7: failed to resolve (No such file or directory) [2013-11-22 12:06:24.140026] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/e7: failed to resolve (No such file or directory) [2013-11-22 12:06:24.142514] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/e7: failed to resolve (No such file or directory) [2013-11-22 12:06:24.254011] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.254079] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.270330] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.270376] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.280014] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.280075] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.294814] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.294881] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.295028] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.295068] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.305832] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.305891] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.312115] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.312169] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.318881] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.318918] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.319000] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.319034] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.330876] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.330931] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.336590] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.336636] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.342123] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.342163] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.342252] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.342297] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.355115] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.355165] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.361112] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.361151] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.366852] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.366899] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.366972] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.367000] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.379305] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.379344] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.385449] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.385481] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.391230] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.391264] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.391335] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.391364] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.403102] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.403144] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.408434] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.408481] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.415743] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.415792] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.415803] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.415844] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.431296] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.431348] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.446621] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.446671] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.459980] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.460027] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.460125] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.460164] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.490455] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.490497] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.509885] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.509936] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.525358] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.525403] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.525484] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.525514] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.542389] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.542427] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.557378] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.557425] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.582441] D [fuse-bridge.c:1233:fuse_unlink_cbk] 0-glusterfs-fuse: 201: RMDIR() /fb => -1 (Directory not empty) [2013-11-22 12:06:24.582512] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:06:24.582569] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.583183] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-1: sending releasedir on fd [2013-11-22 12:06:24.583242] D [client-rpc-fops.c:2780:client_fdctx_destroy] 0-Staging-client-0: sending releasedir on fd [2013-11-22 12:07:02.735140] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:07:02.744086] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:07:02.744896] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:07:02.746868] D [fuse-resolve.c:83:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/a6: failed to resolve (No such file or directory) [2013-11-22 12:07:02.747362] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-0': avail_percent is: 87.00 and avail_space is: 15787483136 and avail_inodes is: 99.00 [2013-11-22 12:07:02.747469] D [dht-diskusage.c:80:dht_du_info_cbk] 0-Staging-dht: on subvolume 'Staging-client-1': avail_percent is: 87.00 and avail_space is: 15787483136 and avail_inodes is: 99.00 Look on the bricks: [root@nunik:~] ls -lha /mnt/gluster/Staging*/fb /mnt/gluster/Staging1/fb: total 24K d-wx-wx---+ 3 root root 4.0K Nov 22 13:06 . drwxrwx---+ 5 root root 4.0K Nov 22 13:07 .. drwxrwx---+ 2 root root 4.0K Nov 22 13:06 20fe20e39eb1646b14657439530c22 /mnt/gluster/Staging2/fb: total 16K d-wx-wx---+ 2 root root 4.0K Nov 22 13:06 . drwxrwx---+ 5 root root 4.0K Nov 22 13:07 .. Expected results: It should jut go and go and go without any issues. Additional info: Probably related to bug #966848
Created attachment 827716 [details] strace from brick 2
Patch from bug #966848 d402b07952a7d08f36b2a8602b59032da9339508 seems to fix the issue so far.
No, patch d402b07952a7d08f36b2a8602b59032da9339508, so even if it might be related it isn't it. rm: cannot remove `/var/gdc/staging/users/74': Directory not empty
Created attachment 827766 [details] strace from brick 1
Created attachment 827767 [details] strace from brick 2
I can't reproduce the issue on 3.5qa1
158e51c17c0f569a1c107fa8747cf8d3fdb76b6d is the first bad commit commit 158e51c17c0f569a1c107fa8747cf8d3fdb76b6d Author: Vijaykumar Koppad <vkoppad> Date: Wed Mar 27 14:14:41 2013 +0530 libglusterfs/dict: fix infinite loop in dict_keys_join() - missing "pairs = next" caused infinite loop Change-Id: I3edc4f50473f7498815c73e1066167392718fddf BUG: 905871 Signed-off-by: Vijaykumar Koppad <vkoppad> Reviewed-on: http://review.gluster.org/4728 Reviewed-by: Jeff Darcy <jdarcy> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Anand Avati <avati> :040000 040000 1a49416ec21d75ada1a2d2f324be8bfd94099e09 8f6218b67cee938b62b5f3cb151a9cb9f5712da7 M libglusterfs bisect run success
Note I can't reproduce issue unless I have broken georeplication running :/
Now I bisected master for what commit fixes it: b13c483dca20e4015b958f8959328e665a357f60 is the first bad commit commit b13c483dca20e4015b958f8959328e665a357f60 Author: Avra Sengupta <asengupt> Date: Sat Jun 1 16:17:57 2013 +0530 gsyncd: distribute the crawling load * also consume changelog for change detection. * Status fixes * Use new libgfchangelog done API * process (and sync) one changelog at a time Change-Id: I24891615bb762e0741b1819ddfdef8802326cb16 BUG: 847839 Original Author: Csaba Henk <csaba> Original Author: Aravinda VK <avishwan> Original Author: Venky Shankar <vshankar> Original Author: Amar Tumballi <amarts> Original Author: Avra Sengupta <asengupt> Signed-off-by: Avra Sengupta <asengupt> Reviewed-on: http://review.gluster.org/5131 Reviewed-by: Vijay Bellur <vbellur> Tested-by: Vijay Bellur <vbellur> :100644 100644 cfccf3f9e641f03093292cba2de8dc002a8539ef a1a52eebd8e5973cdeb9058d9ab85520988a2a96 M configure.ac :040000 040000 3f91d84ceec56dc9dbd7b5c3b356a0da822d40fc 93b21f3454bcc1cd3536df27ed74a67059d38686 M extras :040000 040000 bfa5d4f28f506666e6a28d78a44d9df9e9dff312 5b240601222bfdbf727e7a0523ec06501b730e80 M geo-replication :100644 100644 2e94c3315dda42539ccf3f5557fda5e188e488bd e92cd6e0a80708f6ff21cc34873fcc2fe8f63f62 M glusterfs.spec.in :040000 040000 38665c85dde61b4b90a1054a4cbf8306880accab 17c4e022ab4a8d448fde8eb015a43d5c17f45915 M xlators bisect run success I'm even more puzzled but geo-replication is probably something triggering the issue.
Reverting 158e51c17c0f569a1c107fa8747cf8d3fdb76b6d in release-3.4 fixes the issue but from look on commit I think we want it to stay, but we probably want to backport changes that lead to b13c483dca20e4015b958f8959328e665a357f60 from master to release-3.4
Cherry-picking changes from bug: 847839 up to b13c483dca20e4015b958f8959328e665a357f60 does fix the issue but I'm quite worried with commits that follow it, it's quite big change :/
Way to reproduce: create two distribute volumes with 2 bricks each start georeplicating from one to another mount the volume for example /mnt run OINK=$(($(date +%s)+500)) while [ $OINK -ge $(date +%s) ] ; do rm -rf /mnt/* && for j in {1..10}; do for i in {1..10} ; do mkdir -p /mnt/${j}/${i} ; done ; done || exit 1; done
What's your sub filesystem? xfs, extN or others? And system 32bit or 64bit?
ext4 and btrfs both 64bit can reproduce the issue.
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5. This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs". If there is no response by the end of the month, this bug will get automatically closed.
If this bug still exists in a later release please reopen this and change the version or open a new bug.