Created attachment 1272007 [details] Script to create a dir, create and rename files inside the dir Description of problem: rm -rf <dir> on a directory which contains a lot of stale linkto files will fail with ENOTEMPTY Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. Create a 3x1 pure distribute volume and FUSE mount it 2. Run the attached script (test.sh from the mount point). This creates a dir dir1, creates 1000 files inside the directory and renames the files so as to create lots of linkto files on the bricks. 3. Run the remove-brick command to start removing the last brick (vol1-client-2) Do not commit the remove brick 4. From the mount point, run `rm -rf dir1` Actual results: rm -rf fails with ENOTEMPTY Expected results: rm -rf should succeed Additional info:
REVIEW: https://review.gluster.org/17065 (cluster/dht: rm -rf fails if dir has stale linkto files) posted (#1) for review on master by N Balachandran (nbalacha)
REVIEW: https://review.gluster.org/17065 (cluster/dht: rm -rf fails if dir has stale linkto files) posted (#2) for review on master by N Balachandran (nbalacha)
REVIEW: https://review.gluster.org/17065 (cluster/dht: rm -rf fails if dir has stale linkto files) posted (#3) for review on master by N Balachandran (nbalacha)
RCA: The attached script creates a lot of files and renames them so we end up with multiple linkto files on the bricks. Start a remove brick and wait until the rebalance completes (do not commit the remove brick). This ensures that the brick is still part of the volume so rmdir commands will be sent to it as well. rm -rf <dir> will send an unlink for each entry returned followed by an unlink on the directory. dht_rmdir first performs a readdirp on each subvol to make sure there are no entries in the directories on the bricks. Any stale linkto files found will be deleted. This is particularly useful as would not be cleaned up otherwise. The issue is that dht sends the readdirp only once to each subvol. If there are more stale linkto files on the brick than can be returned in a single readdirp call, they will not be cleaned up and the rmdir will fail with ENOTEMPTY. The fix involves sending readdirp to each brick until no more entries are returned.
COMMIT: https://review.gluster.org/17065 committed in master by Raghavendra G (rgowdapp) ------ commit e5f9ba138571bd18226462c49ff6a55f5c3ed3a4 Author: N Balachandran <nbalacha> Date: Mon Apr 17 15:21:20 2017 +0530 cluster/dht: rm -rf fails if dir has stale linkto files rm -rf <dir> fails with ENOENT if dir contains a lot of stale linkto files. This is because a single readdirp is sent as part of the rmdir which would return and delete only as many linkto files on the bricks as would fit in one readdirp buffer. Running rm -rf <dir> multiple times will eventually delete all the files. The fix sends readdirp on each subvol until no more entries are returned. Change-Id: I447f2d193de4bd8ac16e4541c6b919d22250e39e BUG: 1442724 Signed-off-by: N Balachandran <nbalacha> Reviewed-on: https://review.gluster.org/17065 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report. glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html [2] https://www.gluster.org/pipermail/gluster-users/