Description of problem: quarantine folder becomes empty and bitrot status does not display anything. Version-Release number of selected component (if applicable): glusterfs-3.7.5-19.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1. Create a volume dist-rep volume 2. Enable bitrot and quota on the volume 3. Mount the volume using fuse and create 100 1GB files using dd. 4. corrupt some files from backend so that scrubber will mark them as bad. 5. Once scrubber scrubs the files run the command "gluster vol bitrot <vol_name> scrub status" command to see the corrupted files. 4. In the mount point perform linux untar and rm -rf <linuxuntar> in a continuous loop. Actual results: After sometime scrub status does not list the files which were corrupted. Expected results: scrub status should always list the files which were corrupted. Additional info:
Hi Kotresh, Can you please put the RCA for the bug? Thanks kasturi
RCA: The files under quarantine directory are removed during inode forget. inode forget is called not only during unlink of a file but also when inode table's LRU size exceeds 16k. Hence when bad file is not accessed for a long time and new files are being created and removed putting pressure on inode->tables LRU list to exceed 16k will result in removing bad file from quarantine directory because of which bitrot status fails to show the bad file even though it's not deleted from mount.
Doc Text looks good.
Upstream Patches: http://review.gluster.org/#/c/13552/ (release/3.7) http://review.gluster.org/#/c/13472/ (master)
Tested and verified this on the build 3.7.9-1 Reduced the network.inode-lru-limit to 50. Created 1000 files, corrupted over 100 files, waited for the scrubber to mark the files as bad, started linux untar and rm -rf simultaneously. Scrub status continued to show the correct output. Deleted about 50 files from the mountpoint, and the changes were correctly reflected in the quarantine folder and hence the scrub output. Moving this BZ to fixed in 3.1.3. Detailed logs are attached.
Created attachment 1148537 [details] CLI logs
Created attachment 1148538 [details] scrub logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240