Hide Forgot
Description of problem: I'm unable to delete a directory. rm -rf on thar directory is giving directory not empty errors. The volume is 2*3 distributed-replicated. When I checked in backend one subvolume of dht had another dir in the dir specified whereas other subvolume didn't. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa30 How reproducible: Inconsitent Steps to Reproduce: 1. Create and start a 2*3 dist-rep volume. 2. Run some i/o like run fs-perf-test from a fuse mount and fileop from fusemount. Run iozone and dd and other tests from nfs mount etc etc. 3. take down two of three replicate subvolume in one dht subvolume and other 1 replicate subvolume in another dht subvolume. 4. Bring them back-up after sometime. 5. run rm -rf * from mountpoint. Actual results: [root@QA-23 glusterfs]# ls fileop_L1_19 [root@QA-23 glusterfs]# rm -rf fileop_L1_19/ rm: cannot remove `fileop_L1_19/fileop_L1_19_L2_15': Directory not empty Expected results: rm -rf * should delete all files from the mountpoint. Additional info: As I mentioned out of two dht subvolumes, one had some files in "fileop_L1_19/fileop_L1_19_L2_15" whereas other dht subvolume didn't have any files in them. Archiving all the logs.
This seems to be an issue in dht_readdirp. Because the first subvol, does not have the complete dir hierarchy, we are not getting the complete entry list (as dir entries from other subvols are ignored). dir self heal is also not triggered as dir entries from non-first subvols are ignored in readdirp.
we need to handle this with a proper script (out side of glusterfs codebase) to fix the situation.
with bug 773187 getting fixed, this is no longer a blocker for the 3.3.0 release.
With all users adhering to best practise (ie, making sure all data access will happen through gluster mount/gluster-nfs), this issue won't happen anymore. Hence we are not working on it actively. If someone hits the issue please re-open this. As bug 773187 got fixed in 3.3.0 release, will be closing it with resolution closed currentrelease.