Hide Forgot
(In reply to comment #0) > In a three replica setup, if one replica goes down, IO continues as normal to > the two remaining replicas, and self-heals when this replica comes back up. > > When two replicas go down, IO continues to the one remaining replica. When one > replica comes back up, self-heal allows IO on the source replica to continue in > parallel to self-heal on the destination replica. Correct behaviour is to pause IO on all replicas, and allow only self heal to continue on the replica that came back up. Right now, parallel self-heal and IO on the source replica is resulting in data corruption on the recovering replica.
In a three replica setup, if one replica goes down, IO continues as normal to the two remaining replicas, and self-heals when this replica comes back up. When two replicas go down, IO continues to the one remaining replica. When one replica comes back up, self-heal allows IO on the source replica to continue in parallel to self-heal on the destination replica.
I did a similar kind of test. i.e. created a 3 replica volume. Started it and mounted it as nfs mount. Started dd of a 2gb file and brought down 2 bricks. After around 500 MB is copied brought up the 2 bricks. After the file is entirely written, checked the md5sum of the file on all the bricks. md5sum was same on all the bricks and on the mount point. Seems that it is working fine.
The problem is when only one downed node is brought up, not both. Leave the third replica down and let the IO complete. Then check md5 sum. Make sure you're using a file other than /dev/zero for dd input.
Please update the status of this bug as its been more than 6months since its filed (bug id < 2000) Please resolve it with proper resolution if its not valid anymore. If its still valid and not critical, move it to 'enhancement' severity.
Still valid.
I created 3 replica volume, started it and mounted it as nfs mount point. started dd of 2gb with if=/dev/urandom at the mount point. I brought down 2 bricks. I was monitoring du of file in all the bricks and at the mount. when i brought down two bricks, file size was increasing in one brick which was up, and it was constant in other two. When i brought up one of the bricks which were down , IO stopped on the mount point till self healing was complete. Later the file size in both bricks and mount point started increasing simultaneously. I have performed this test for some 7 to 8 times, it was consistently working fine.