Bug 1376757 - Data corruption in write ordering of rebalance and application writes
Summary: Data corruption in write ordering of rebalance and application writes
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: mainline
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
Assignee: Karthik U S
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-16 10:41 UTC by Karthik U S
Modified: 2019-08-30 10:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-30 10:21:07 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Karthik U S 2016-09-16 10:41:41 UTC
Description of problem:

Lack of atomicity b/w read-src and write-dst of rebalance process.

If a rebalance process has read the data from a region but, before writing to the destination, an application writes/modifies data on the same region. Then the rebalance process completes but it will not have the modified data on the destination even though the application write was successful.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

1. Attach gdb to rebalance process.
2. Break after reading from src but before you write to dst.
3. While break point is hit in gdb, initiate a write from an application on mount to the same region on which read is done.
4. Wait till the write is complete (better still do an fsync so that write would be reached the disk).
5. Resume rebalance process from gdb and let the file migration complete.
6. Once the migration is complete you would see the write done by application in step 3 is not seen in destination file. Since write/fsync from application was complete, this is a data corruption.


Actual results:

Write done by application is not seen in the destination file.


Expected results:

Write done by application should be reflected in the destination file.


Additional info:

Comment 1 Raghavendra G 2016-09-23 10:50:39 UTC
Currently rebalance process does:

1. read (src)
2. write (dst)

To make sure that src and dst are identical, we need to make combined transaction of 1 and 2 atomic. Otherwise with parallel writes happening to same region during rebalance, writes on dst can go out of order (relative to src) and dst can be different from src which is basically a corruption.

Consider Following sequence of events happening on overlapping/same region of a file:

1. rebalance process read a region (lets say which was written by an earlier write w1).
2. application does a write (w2) to same region. w2 completes on src and dst.
3. rebalance process proceeds to write on dst the data it read in 1. So, w1 is sent to dst.

After the above steps, the order of w1 and w2 in src is (w1, w2) but on dst, it is (w2, w1). Hence w2 is lost on dst, resulting in corruption (as w2 was reported success to application).

To make atomic, we need to:
* lock (src) the region of file being read before 1
* unlock (src) the region of file being read after 2

and make sure that this lock blocks new writes from application (till an unlock is issued). Combine this with the approach that application writes are serially written to src first and then to dst and we have a solution.

Comment 2 Worker Ant 2016-10-21 10:51:57 UTC
REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#1) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 3 Worker Ant 2016-10-21 11:58:12 UTC
REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#2) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 4 Worker Ant 2016-10-21 12:22:39 UTC
REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#3) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 5 Worker Ant 2016-10-24 11:39:04 UTC
REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#4) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 6 Worker Ant 2016-11-06 06:13:16 UTC
REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#5) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 7 Worker Ant 2016-11-07 11:57:37 UTC
REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#6) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 8 Worker Ant 2017-06-28 05:29:59 UTC
REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#7) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 9 Worker Ant 2017-10-27 09:36:59 UTC
REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#8) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 10 Worker Ant 2017-10-27 10:17:11 UTC
REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#9) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 11 Worker Ant 2017-10-30 07:33:16 UTC
REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#10) for review on master by Karthik U S (ksubrahm@redhat.com)

Comment 12 Karthik U S 2019-08-30 10:21:07 UTC
This issue is being tracked by https://github.com/gluster/glusterfs/issues/308.
Since there is no active work going on this closing this for now.
Feel free to reopen this or open a new bug when needed.


Note You need to log in before you can comment on or make changes to this bug.