Description of problem:

Disable and reenable of bit-rot feature on a gluster volume can lead to a situation where some files are marked as bad (even though they are not corrupted).

Consider a gluster volume with bit-rot feature enabled and consisting of files that have been signed with the checksum. Now, disable the feature. The files still continue to contain the version and signature extended attributes. At this stage if some files are modified and later bit-rot feature is reenabled, then those files are which were modified while the feature was off, will be marked as bad by the scrubber after the feature is reenabled.

This happens because of this reason.

The modification of the file(s), while the feature was off, would not have resulted in calculation of the checksum of the file and that checksum being saved as part of the signature xattr. 

And the bit-rot daemon whenever is spawned (either restart or regular start due to feature enable) does a one shot crawling of the entire volume, where it skips calculating the checksum of the files (and saving that checksum as part of signature) if any file contains those xattrs already (assuming their value should be correct).

So when scrubber does its job, it finds the on disk checksum and the calculated checksum to be different. This makes scrubber mark such a file as bad.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create a gluster volume, start it and mount it
2. Enable bit-rot feature
3. Create a file with some data
4. Wait till the file is properly signed (it takes 2 minutes for proper 
   signature to be saved as an xattr)
5. disable bit-rot
6. Modify the contents of the file.
7. Reenable the bit-rot feature
8. Start on-demand scrubbing.


Actual results:

File is marked as bad even though no corruption has happened due to bit-rot.

Expected results:


Additional info: