Description of problem: ======================= Tried to remove all the files from client using "rm -rf *" while demotions were in-progress. rmdir failed complaing "Directory not empty". The file that is present from the client has all the attributes ???? and at the backend it is the link file in the cold tier: From client: ============ [root@dj level75]# ls -l ls: cannot access 5664bb94%%VV2VS9M5R9: No such file or directory total 0 ?????????? ? ? ? ? ? 5664bb94%%VV2VS9M5R9 [root@dj level75]# [root@dj level75]# pwd /mnt/master/thread4/level05/level15/level25/level35/level45/level55/level65/level75 [root@dj level75]# From Cold tier bricks: ====================== [root@dhcp37-110 level75]# ls -l total 0 ---------T. 2 root root 0 Dec 7 05:17 5664bb94%%VV2VS9M5R9 [root@dhcp37-110 level75]# pwd /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75 [root@dhcp37-110 level75]# Errors at the Client logs: =========================== mnt-master.log-20151207:[2015-12-07 17:32:55.063393] I [MSGID: 109070] [dht-common.c:1840:dht_lookup_linkfile_cbk] 2-master-tier-dht: Lookup of /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 on master-hot-dht (following linkfile) failed ,gfid = 00000000-0000-0000-0000-000000000000 [No such file or directory] mnt-master.log-20151207:[2015-12-07 17:32:55.069227] I [MSGID: 109069] [dht-common.c:1159:dht_lookup_unlink_stale_linkto_cbk] 2-master-tier-dht: Returned with op_ret -1 and op_errno 16 for /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 [root@dj glusterfs]# Errors at the Bricks logs: ========================= rhs-brick1-ct-b5.log:[2015-12-07 18:15:06.611511] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-master-server: 76189: UNLINK /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 (691747be-864d-41b0-90f0-0932362702b0/5664bb94%%VV2VS9M5R9) ==> (Device or resource busy) [Device or resource busy] rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789653] I [MSGID: 113030] [posix.c:1552:posix_unlink] 0-master-posix: open-fd-key-status: 0 for /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789717] I [MSGID: 113031] [posix.c:1583:posix_unlink] 0-master-posix: linkto_xattr status: 1 for /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789784] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-master-server: 76208: UNLINK /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 (691747be-864d-41b0-90f0-0932362702b0/5664bb94%%VV2VS9M5R9) ==> (Device or resource busy) [Device or resource busy] [root@dhcp37-110 bricks]# Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.5-9.el7rhgs.x86_64 How reproducible: ================= 1/1 Steps to Reproduce: =================== 1. Create and start Tier volume both hot and cold tier as DR 2. Mount the volume 3. Set the tier mode to test 4. Start creating lots of data 5. Wait for promotion/demotion to start 6. While demotions are happening, perform rm -rf from client Actual results: =============== Link file remain in the cold tier without the data file being in hot tier Expected results: ================= Link file or stale link file should also get deleted Additional info:
upstream patch : http://review.gluster.org/#/c/12829/
Verified with build: glusterfs-3.7.5-12 While promotion and demotiones were inprogress, performed rm -rf * from client having 6k files at 5 level deep directory. All files got removed from client and backend without any stale file being present in the system. Moving this bug to verified state. From Client: ============ [root@dj fuse]# ls t1 test thread0 thread1 thread2 thread3 thread4 [root@dj fuse]# [root@dj fuse]# [root@dj fuse]# rm -rf * [root@dj fuse]# [root@dj fuse]# ls -lRt .: total 0 [root@dj fuse]# From Backend: ============= Cold Tier bricks: ================= [root@dhcp37-165 tiervolume_ct-disp1]# ls -lRt .: total 0 [root@dhcp37-165 tiervolume_ct-disp1]# [root@dhcp37-133 tiervolume_ct-disp1]# ls -lRt .: total 0 [root@dhcp37-133 tiervolume_ct-disp1]# [root@dhcp37-160 tiervolume_ct-disp1]# ls -lRt .: total 0 [root@dhcp37-160 tiervolume_ct-disp1]# [root@dhcp37-158 tiervolume_ct-disp1]# ls -lRt .: total 0 [root@dhcp37-158 tiervolume_ct-disp1]# [root@dhcp37-110 tiervolume_ct-disp1]# ls -lRt .: total 0 [root@dhcp37-110 tiervolume_ct-disp1]# [root@dhcp37-155 tiervolume_ct-disp1]# ls -lRt .: total 0 [root@dhcp37-155 tiervolume_ct-disp1]# Hot Tier Bricks: ================ [root@dhcp37-99 tiervolume_hot]# ls -lRt .: total 0 [root@dhcp37-99 tiervolume_hot]# [root@dhcp37-88 tiervolume_hot]# ls -lRt .: total 0 [root@dhcp37-88 tiervolume_hot]# [root@dhcp37-112 tiervolume_hot]# ls -lRt .: total 0 [root@dhcp37-112 tiervolume_hot]# [root@dhcp37-199 tiervolume_hot]# ls -lRt .: total 0 [root@dhcp37-199 tiervolume_hot]# [root@dhcp37-162 tiervolume_hot]# ls -lRt .: total 0 [root@dhcp37-162 tiervolume_hot]# [root@dhcp37-87 tiervolume_hot]# ls -lRt .: total 0 [root@dhcp37-87 tiervolume_hot]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html