Description of problem: Create 2 node replicate volume and created dir tree. Now ran rm -rf dir and meanwhile added two more bricks to volume, thus making it distributed-replicated volume. rm -rf failed. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa22 How reproducible: thrice in as many tries Steps to Reproduce: 1. Create ans start 2 node pure replicate volume. 2. Create a dir tree. 3. Now rm -rf dir 4. add two more bricks to volume. Actual results: rm -rf failed fts_read failed message. [root@RHEL6 mnt]# time rm -rf a rm: fts_read failed: No such file or directory real 2m22.360s user 0m0.005s sys 0m0.024s Expected results: rm-rf should succeed. Additional info: Entries in client log. [2012-02-17 01:33:16.057531] I [client-handshake.c:1102:select_server_supported_programs] 1-hosdu-client-0: Using Program GlusterFS 3.3.0qa22, Num (1298437), Version (330) [2012-02-17 01:33:16.057872] I [client-handshake.c:923:client_setvolume_cbk] 1-hosdu-client-0: Connected to 10.1.11.113:24009, attached to remote volume '/data/bricks/hosdu_brick1'. [2012-02-17 01:33:16.057895] I [afr-common.c:3461:afr_notify] 1-hosdu-replicate-0: subvol 0 came up, start crawl [2012-02-17 01:33:16.057905] I [afr-common.c:3556:afr_notify] 1-hosdu-replicate-0: All subvolumes came up, start crawl [2012-02-17 01:33:16.058266] I [client-handshake.c:923:client_setvolume_cbk] 1-hosdu-client-2: Connected to 10.1.11.136:24009, attached to remote volume '/data/bricks/hosdu_brick3'. [2012-02-17 01:33:16.058288] I [afr-common.c:3457:afr_notify] 1-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online. [2012-02-17 01:33:16.063598] I [fuse-bridge.c:3699:fuse_graph_setup] 0-fuse: switched to graph 1 [2012-02-17 01:33:16.137570] I [afr-common.c:1827:afr_set_root_inode_on_first_lookup] 1-hosdu-replicate-0: added root inode [2012-02-17 01:33:16.182019] I [afr-common.c:1827:afr_set_root_inode_on_first_lookup] 1-hosdu-replicate-1: added root inode [2012-02-17 01:33:16.272465] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: c9ebb69a-84da-47da-8411-601031535209: failed to resolve (Invalid argument) [2012-02-17 01:33:16.272509] E [fuse-bridge.c:519:fuse_getattr_resume] 0-glusterfs-fuse: 53145: GETATTR 140579743018596 (c9ebb69a-84da-47da-8411-601031535209) resolution failed [2012-02-17 01:33:16.332797] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: c9ebb69a-84da-47da-8411-601031535209: failed to resolve (Invalid argument) [2012-02-17 01:33:16.332842] E [fuse-bridge.c:519:fuse_getattr_resume] 0-glusterfs-fuse: 53146: GETATTR 140579743018596 (c9ebb69a-84da-47da-8411-601031535209) resolution failed [2012-02-17 01:33:16.403585] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: 9b8da545-c674-4848-b8ba-493a8d271543: failed to resolve (Invalid argument) [2012-02-17 01:33:16.403633] E [fuse-bridge.c:1358:fuse_rmdir_resume] 0-glusterfs-fuse: RMDIR 140579743018448 (00000000-0000-0000-0000-000000000000/a) resolution failed [2012-02-17 01:33:16.465284] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: fd08b15a-281c-409a-9a23-ab59edd855dd: failed to resolve (Invalid argument) [2012-02-17 01:33:16.465330] E [fuse-bridge.c:1358:fuse_rmdir_resume] 0-glusterfs-fuse: RMDIR 140579743018300 (00000000-0000-0000-0000-000000000000/a) resolution failed [2012-02-17 01:33:16.526687] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: 49fe2993-a5a9-4bcf-9273-1f9de8e3369d: failed to resolve (Invalid argument) [2012-02-17 01:33:16.526732] E [fuse-bridge.c:519:fuse_getattr_resume] 0-glusterfs-fuse: 54225: GETATTR 140579743018152 (49fe2993-a5a9-4bcf-9273-1f9de8e3369d) resolution failed [2012-02-17 01:33:16.587728] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: 1d25c2a1-96f4-406a-b957-e91044c3cb5a: failed to resolve (Invalid argument) [2012-02-17 01:33:16.587798] E [fuse-bridge.c:519:fuse_getattr_resume] 0-glusterfs-fuse: 54227: GETATTR 140579743018004 (1d25c2a1-96f4-406a-b957-e91044c3cb5a) resolution failed [2012-02-17 01:33:16.653459] W [fuse-resolve.c:148:fuse_resolve_gfid_cbk] 0-fuse: 1d25c2a1-96f4-406a-b957-e91044c3cb5a: failed to resolve (Invalid argument) [2012-02-17 01:33:16.653507] E [fuse-bridge.c:519:fuse_getattr_resume] 0-glusterfs-fuse: 54228: GETATTR 140579743018004 (1d25c2a1-96f4-406a-b957-e91044c3cb5a) resolution failed [2012-02-17 01:33:19.060785] I [client-handshake.c:1102:select_server_supported_programs] 1-hosdu-client-3: Using Program GlusterFS 3.3.0qa22, Num (1298437), Version (330) [2012-02-17 01:33:19.063650] I [client-handshake.c:923:client_setvolume_cbk] 1-hosdu-client-3: Connected to 10.1.11.137:24009, attached to remote volume '/data/bricks/hosdu_brick4'. [2012-02-17 01:33:19.063679] I [afr-common.c:3461:afr_notify] 1-hosdu-replicate-1: subvol 1 came up, start crawl I was able to hit this issue three times in as many tries.
Created attachment 563903 [details] client log file This is happening consistently. Attaching the client log file.
please check the behavior with 3.3.0qa23
I don't see the issue in glusterfs-3.3.0qa23 release, So marking the bug closed upstream.