Hide Forgot
Description of problem: File which is corrupted and marked as bad by scrubber gets promoted to hot tier. Version-Release number of selected component (if applicable): glusterfs-3.7.5-6.el6rhs.x86_64 How reproducible: Always Steps to Reproduce: 1. Create a tiered volume and enable bit rot 2. Fuse mount the volume and create some data. 3. schedule the scrubber frequency as hourly. 4. Edit the file from backend so that scrubber can identify it as bad file. Actual results: File which is marked as bad gets promoted to hot tier. Expected results: File which is marked as bad should not get promoted to hot tier. Additional info:
sos reports can be found at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1283507/
1) IMHO Bad files should never be promoted. And if they are in the hot tier they should be demoted. Venky your thoughts on this. 2) As far as the bug is concerned the reason a bad file is not getting promoted is that bit rot will not serve a bad file to any clients. Tiering migrator is a client by itself. This should be true for files that are in the hot tier, that the bad files are not getting demoted. Now this should be the bug. Bad files shouldn't get the prime storage.
3) Oh ya .. CTR shouldnt heatup bad files :)
(In reply to Joseph Elwin Fernandes from comment #3) > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > they should be demoted. Venky your thoughts on this. Well, once an object is marked bad, any access is disallowed. So, promoting/demoting should be out of the question. Furthermore, promoting/demoting a bad object might screw up things - once the corrupted object gets migrated, it's signed again, but now with bad data. So, one needs to be extra careful here. > > 2) As far as the bug is concerned the reason a bad file is not getting > promoted is that bit rot will not serve a bad file to any clients. Tiering > migrator is a client by itself. This should be true for files that are in > the hot tier, that the bad files are not getting demoted. Now this should be > the bug. Bad files shouldn't get the prime storage. The bug description says "file get promoted". It shouldn't get migrated at all in either direction. You're correct in saying that corrupted objects shouldn't get prime storage, but you'd need to recover them anyway. So, there's less point in moving such objects to the cold tier and then initiate recovery - they can and _should_ be recovered from where ever it lives now given that there's a replica to recover from. Coming to the bug - did the file get migrated by any chance before it was scanned by scrubber?
(In reply to Venky Shankar from comment #5) > (In reply to Joseph Elwin Fernandes from comment #3) > > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > > they should be demoted. Venky your thoughts on this. > > Well, once an object is marked bad, any access is disallowed. So, > promoting/demoting should be out of the question. Furthermore, > promoting/demoting a bad object might screw up things - once the corrupted > object gets migrated, it's signed again, but now with bad data. So, one > needs to be extra careful here. > > > > > 2) As far as the bug is concerned the reason a bad file is not getting > > promoted is that bit rot will not serve a bad file to any clients. Tiering > > migrator is a client by itself. This should be true for files that are in > > the hot tier, that the bad files are not getting demoted. Now this should be > > the bug. Bad files shouldn't get the prime storage. > > The bug description says "file get promoted". It shouldn't get migrated at > all in either direction. You're correct in saying that corrupted objects > shouldn't get prime storage, but you'd need to recover them anyway. So, > there's less point in moving such objects to the cold tier and then initiate > recovery - they can and _should_ be recovered from where ever it lives now > given that there's a replica to recover from. OOPS! missed that. Valid point on recovering and then moving. > > Coming to the bug - did the file get migrated by any chance before it was > scanned by scrubber?
> Coming to the bug - did the file get migrated by any chance before it was > scanned by scrubber? Is it possible, that a file is picked for migraiton and during the migration the scrubber marks it bad ?
(In reply to Venky Shankar from comment #5) > (In reply to Joseph Elwin Fernandes from comment #3) > > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > > they should be demoted. Venky your thoughts on this. > > Well, once an object is marked bad, any access is disallowed. So, > promoting/demoting should be out of the question. Furthermore, > promoting/demoting a bad object might screw up things - once the corrupted > object gets migrated, it's signed again, but now with bad data. So, one > needs to be extra careful here. > > > > > 2) As far as the bug is concerned the reason a bad file is not getting > > promoted is that bit rot will not serve a bad file to any clients. Tiering > > migrator is a client by itself. This should be true for files that are in > > the hot tier, that the bad files are not getting demoted. Now this should be > > the bug. Bad files shouldn't get the prime storage. > > The bug description says "file get promoted". It shouldn't get migrated at > all in either direction. You're correct in saying that corrupted objects > shouldn't get prime storage, but you'd need to recover them anyway. So, > there's less point in moving such objects to the cold tier and then initiate > recovery - they can and _should_ be recovered from where ever it lives now > given that there's a replica to recover from. > > Coming to the bug - did the file get migrated by any chance before it was > scanned by scrubber? This might be one of the reason why the file is promoted .
(In reply to Joseph Elwin Fernandes from comment #6) > (In reply to Venky Shankar from comment #5) > > (In reply to Joseph Elwin Fernandes from comment #3) > > > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > > > they should be demoted. Venky your thoughts on this. > > > > Well, once an object is marked bad, any access is disallowed. So, > > promoting/demoting should be out of the question. Furthermore, > > promoting/demoting a bad object might screw up things - once the corrupted > > object gets migrated, it's signed again, but now with bad data. So, one > > needs to be extra careful here. > > > > > > > > 2) As far as the bug is concerned the reason a bad file is not getting > > > promoted is that bit rot will not serve a bad file to any clients. Tiering > > > migrator is a client by itself. This should be true for files that are in > > > the hot tier, that the bad files are not getting demoted. Now this should be > > > the bug. Bad files shouldn't get the prime storage. > > > > The bug description says "file get promoted". It shouldn't get migrated at > > all in either direction. You're correct in saying that corrupted objects > > shouldn't get prime storage, but you'd need to recover them anyway. So, > > there's less point in moving such objects to the cold tier and then initiate > > recovery - they can and _should_ be recovered from where ever it lives now > > given that there's a replica to recover from. > > OOPS! missed that. Valid point on recovering and then moving. > > > > Coming to the bug - did the file get migrated by any chance before it was > > scanned by scrubber? No, there is no chance because there was no I/O going on the mount point nor the file was accessed.
(In reply to RamaKasturi from comment #9) > (In reply to Joseph Elwin Fernandes from comment #6) > > (In reply to Venky Shankar from comment #5) > > > (In reply to Joseph Elwin Fernandes from comment #3) > > > > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > > > > they should be demoted. Venky your thoughts on this. > > > > > > Well, once an object is marked bad, any access is disallowed. So, > > > promoting/demoting should be out of the question. Furthermore, > > > promoting/demoting a bad object might screw up things - once the corrupted > > > object gets migrated, it's signed again, but now with bad data. So, one > > > needs to be extra careful here. > > > > > > > > > > > 2) As far as the bug is concerned the reason a bad file is not getting > > > > promoted is that bit rot will not serve a bad file to any clients. Tiering > > > > migrator is a client by itself. This should be true for files that are in > > > > the hot tier, that the bad files are not getting demoted. Now this should be > > > > the bug. Bad files shouldn't get the prime storage. > > > > > > The bug description says "file get promoted". It shouldn't get migrated at > > > all in either direction. You're correct in saying that corrupted objects > > > shouldn't get prime storage, but you'd need to recover them anyway. So, > > > there's less point in moving such objects to the cold tier and then initiate > > > recovery - they can and _should_ be recovered from where ever it lives now > > > given that there's a replica to recover from. > > > > OOPS! missed that. Valid point on recovering and then moving. > > > > > > Coming to the bug - did the file get migrated by any chance before it was > > > scanned by scrubber? > > No, there is no chance because there was no I/O going on the mount point nor > the file was accessed. Then how did it become a candidate for migration? If the file got migrated, then it needs to be accessed or modified -- which is disallowed when a file is marked corrupted. So, either the file was not marked bad at all or it got migrated before scrubber could perform integrity checks.
(In reply to Venky Shankar from comment #10) > (In reply to RamaKasturi from comment #9) > > (In reply to Joseph Elwin Fernandes from comment #6) > > > (In reply to Venky Shankar from comment #5) > > > > (In reply to Joseph Elwin Fernandes from comment #3) > > > > > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > > > > > they should be demoted. Venky your thoughts on this. > > > > > > > > Well, once an object is marked bad, any access is disallowed. So, > > > > promoting/demoting should be out of the question. Furthermore, > > > > promoting/demoting a bad object might screw up things - once the corrupted > > > > object gets migrated, it's signed again, but now with bad data. So, one > > > > needs to be extra careful here. > > > > > > > > > > > > > > 2) As far as the bug is concerned the reason a bad file is not getting > > > > > promoted is that bit rot will not serve a bad file to any clients. Tiering > > > > > migrator is a client by itself. This should be true for files that are in > > > > > the hot tier, that the bad files are not getting demoted. Now this should be > > > > > the bug. Bad files shouldn't get the prime storage. > > > > > > > > The bug description says "file get promoted". It shouldn't get migrated at > > > > all in either direction. You're correct in saying that corrupted objects > > > > shouldn't get prime storage, but you'd need to recover them anyway. So, > > > > there's less point in moving such objects to the cold tier and then initiate > > > > recovery - they can and _should_ be recovered from where ever it lives now > > > > given that there's a replica to recover from. > > > > > > OOPS! missed that. Valid point on recovering and then moving. > > > > > > > > Coming to the bug - did the file get migrated by any chance before it was > > > > scanned by scrubber? > > > > No, there is no chance because there was no I/O going on the mount point nor > > the file was accessed. > > Then how did it become a candidate for migration? If the file got migrated, > then it needs to be accessed or modified -- which is disallowed when a file > is marked corrupted. > > So, either the file was not marked bad at all or it got migrated before > scrubber could perform integrity checks. As per my understanding, When scrubber scanned the files (BZ 1283505), files got heated up and this is how the file was moved to hot tier.
(In reply to RamaKasturi from comment #11) > (In reply to Venky Shankar from comment #10) > > (In reply to RamaKasturi from comment #9) > > > (In reply to Joseph Elwin Fernandes from comment #6) > > > > (In reply to Venky Shankar from comment #5) > > > > > (In reply to Joseph Elwin Fernandes from comment #3) > > > > > > 1) IMHO Bad files should never be promoted. And if they are in the hot tier > > > > > > they should be demoted. Venky your thoughts on this. > > > > > > > > > > Well, once an object is marked bad, any access is disallowed. So, > > > > > promoting/demoting should be out of the question. Furthermore, > > > > > promoting/demoting a bad object might screw up things - once the corrupted > > > > > object gets migrated, it's signed again, but now with bad data. So, one > > > > > needs to be extra careful here. > > > > > > > > > > > > > > > > > 2) As far as the bug is concerned the reason a bad file is not getting > > > > > > promoted is that bit rot will not serve a bad file to any clients. Tiering > > > > > > migrator is a client by itself. This should be true for files that are in > > > > > > the hot tier, that the bad files are not getting demoted. Now this should be > > > > > > the bug. Bad files shouldn't get the prime storage. > > > > > > > > > > The bug description says "file get promoted". It shouldn't get migrated at > > > > > all in either direction. You're correct in saying that corrupted objects > > > > > shouldn't get prime storage, but you'd need to recover them anyway. So, > > > > > there's less point in moving such objects to the cold tier and then initiate > > > > > recovery - they can and _should_ be recovered from where ever it lives now > > > > > given that there's a replica to recover from. > > > > > > > > OOPS! missed that. Valid point on recovering and then moving. > > > > > > > > > > Coming to the bug - did the file get migrated by any chance before it was > > > > > scanned by scrubber? > > > > > > No, there is no chance because there was no I/O going on the mount point nor > > > the file was accessed. > > > > Then how did it become a candidate for migration? If the file got migrated, > > then it needs to be accessed or modified -- which is disallowed when a file > > is marked corrupted. > > > > So, either the file was not marked bad at all or it got migrated before > > scrubber could perform integrity checks. > > As per my understanding, When scrubber scanned the files (BZ 1283505), files > got heated up and this is how the file was moved to hot tier. Which is this build? There was a patch to avoid migrating files when scrubber acts on them.
(In reply to Joseph Elwin Fernandes from comment #7) > > Coming to the bug - did the file get migrated by any chance before it was > > scanned by scrubber? > > Is it possible, that a file is picked for migraiton and during the migration > the scrubber marks it bad ? If this happens, then the next I/O on the corrupted objects would result in EIO. In case the object is marked corrupted just after the migration is done (with the source file just about to be purged), then a corrupted object is migrated. This is due to the asynchronous nature of signing and scrubbing.
Apart from the scrubber heating the file which is fixed in the next build. The migration of Bad files is a known issue due to the asynchronous nature of signing and scrubbing, as Venky has pointed out. We need to document this in tiering interop section.
(In reply to Joseph Elwin Fernandes from comment #14) > Apart from the scrubber heating the file which is fixed in the next build. > The migration of Bad files is a known issue due to the asynchronous nature > of signing and scrubbing, as Venky has pointed out. We need to document this > in tiering intro section. Just to be more specific, there are couple of cases here: (jotting down here for ease of documentation) 1. Object is corrupted before it could be signed: In this case, the corrupted object is signed and get migrated upon I/O. There's no way to identify corruption for this set of objects. 2. Object is signed (but not scrubbed) and corruption happens thereafter: In this case, as of now, integrity checking is not done on the fly and the object would get migrated (and signed again in the hot tier). This can be fixed and is being tracked in bz #1227672. But, #1 is the gap where corruption can sneak in.
The doc text doesn't seem to be correct. Pls check with venky and make the needed changes.
A file will not be marked healthy. This bug speaks about how corruption can sneak in and go undetected. As Venky pointed out, these are the scenarios. 1. Object is corrupted before it could be signed: In this case, the corrupted object is signed and get migrated upon I/O. There's no way to identify corruption for this set of objects. 2. Object is signed (but not scrubbed) and corruption happens thereafter: In this case, as of now, integrity checking is not done on the fly and the object would get migrated (and signed again in the hot tier). This can be fixed and is being tracked in bz #1227672. ============================================================================== Just for understanding : Bitrot works like this (explained in a very simple way), STEP 1 :A file is signed, asynchronously using a md5 checksum, by the signer whenever there is some data written to file. STEP 2 : Then the scrubber wakes up in a scheduled manner and regenerates the md5 checksum using the file data and compares it with previously store checksum (from STEP 1 i.e signer). If the checksum dont match then the file is marked as BAD/CORRUPTED. So the above scenarios explains the cases where corruption can go undetected both from signer and scrubber. ==============================================================================
Laura, Doc text and known issue documentation need to be updated with the https://bugzilla.redhat.com/show_bug.cgi?id=1283507#c21
Posted Upstream: http://review.gluster.org/#/c/13969/
As tier is not being actively developed, I'm closing this bug. Feel free to open it if necessary.