Description of problem: In replica 3 volumes there is a possibilities of ending up in split brain, when multiple clients writing data on the same file at non overlapping regions in parallel. Version-Release number of selected component (if applicable): How reproducible: It is very rare to hit hit this case, but we need to imitate the following scenario using gdb to test this. Steps to Reproduce: - Client C0 performs write W1 which fails on brick B0 and succeeds on other two bricks. - C1 performs write W2 which fails on B1 and succeeds on other two bricks. - C2 performs write W3 which fails on B2 and succeeds on other two bricks. - All the 3 writes above happen in parallel and fall on different ranges so afr takes granular locks and all the writes are performed in parallel. Since each client had data-readables as good initially, it does not see file going into split-brain in the in_flight_split_brain check, hence performs the post-op marking the pending xattrs. Now all the bricks are being blamed by each other, ending up in split-brain. Actual results: We will end up in split brain in replica 3 volumes. Expected results: We should not end up in split brain in replica 3 volumes. Additional info:
Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/131966/
Upstream patch: https://review.gluster.org/#/c/19218/
Update: ======= Build Used: glusterfs-3.12.2-15.el7rhgs.x86_64 Discussed with Karthik and below are the scenarios covered Scenario 1: 1) create 1 * 3 volume and start 2) Disable eager lock 3) check full-lock is on or not ( should be on ) 4) write 1GB of file from mount point 5) take state dump for the volume while write is in progress 6) verify the state dump - there should be ONE active lock which contains start=0, len=0 - others should be blocked scenario 2: 1) create 1 * 3 volume and start 2) Disable eager lock 3) disable full-lock 4) write 1GB of file from mount point 5) take state dump for the volume while write is in progress 6) verify the state dump - there should be range locks - locks are not blocked scenario 3: 1) create 1 * 3 volume and start 2) Enable eager lock 3) enable full-lock 4) write 1GB of same file from from 2 clients 5) take state dump for the volume while write is in progress 6) verify the state dump - there should be ONE active lock which contains start=0, len=0 - others should be blocked scenario 4: 1) create 1 * 3 volume and start 2) Enable eager lock 3) disable full-lock 4) overwrite 1GB of same file ( which was written in previous scenario) from from 2 clients 5) take state dump for the volume while write is in progress 6) verify the state dump - there should be range locks which are active Moving status to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607