Bug 1376757

Summary:	Data corruption in write ordering of rebalance and application writes
Product:	[Community] GlusterFS	Reporter:	Karthik U S <ksubrahm>
Component:	distribute	Assignee:	Karthik U S <ksubrahm>
Status:	CLOSED WONTFIX	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	mainline	CC:	atumball, bugs, nbalacha, rgowdapp
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-30 10:21:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Karthik U S 2016-09-16 10:41:41 UTC

Description of problem:

Lack of atomicity b/w read-src and write-dst of rebalance process.

If a rebalance process has read the data from a region but, before writing to the destination, an application writes/modifies data on the same region. Then the rebalance process completes but it will not have the modified data on the destination even though the application write was successful.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

1. Attach gdb to rebalance process.
2. Break after reading from src but before you write to dst.
3. While break point is hit in gdb, initiate a write from an application on mount to the same region on which read is done.
4. Wait till the write is complete (better still do an fsync so that write would be reached the disk).
5. Resume rebalance process from gdb and let the file migration complete.
6. Once the migration is complete you would see the write done by application in step 3 is not seen in destination file. Since write/fsync from application was complete, this is a data corruption.


Actual results:

Write done by application is not seen in the destination file.


Expected results:

Write done by application should be reflected in the destination file.


Additional info:

Comment 1 Raghavendra G 2016-09-23 10:50:39 UTC

Currently rebalance process does:

1. read (src)
2. write (dst)

To make sure that src and dst are identical, we need to make combined transaction of 1 and 2 atomic. Otherwise with parallel writes happening to same region during rebalance, writes on dst can go out of order (relative to src) and dst can be different from src which is basically a corruption.

Consider Following sequence of events happening on overlapping/same region of a file:

1. rebalance process read a region (lets say which was written by an earlier write w1).
2. application does a write (w2) to same region. w2 completes on src and dst.
3. rebalance process proceeds to write on dst the data it read in 1. So, w1 is sent to dst.

After the above steps, the order of w1 and w2 in src is (w1, w2) but on dst, it is (w2, w1). Hence w2 is lost on dst, resulting in corruption (as w2 was reported success to application).

To make atomic, we need to:
* lock (src) the region of file being read before 1
* unlock (src) the region of file being read after 2

and make sure that this lock blocks new writes from application (till an unlock is issued). Combine this with the approach that application writes are serially written to src first and then to dst and we have a solution.

Comment 2 Worker Ant 2016-10-21 10:51:57 UTC

REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#1) for review on master by Karthik U S (ksubrahm)

Comment 3 Worker Ant 2016-10-21 11:58:12 UTC

REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#2) for review on master by Karthik U S (ksubrahm)

Comment 4 Worker Ant 2016-10-21 12:22:39 UTC

REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#3) for review on master by Karthik U S (ksubrahm)

Comment 5 Worker Ant 2016-10-24 11:39:04 UTC

REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#4) for review on master by Karthik U S (ksubrahm)

Comment 6 Worker Ant 2016-11-06 06:13:16 UTC

REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#5) for review on master by Karthik U S (ksubrahm)

Comment 7 Worker Ant 2016-11-07 11:57:37 UTC

REVIEW: http://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#6) for review on master by Karthik U S (ksubrahm)

Comment 8 Worker Ant 2017-06-28 05:29:59 UTC

REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#7) for review on master by Karthik U S (ksubrahm)

Comment 9 Worker Ant 2017-10-27 09:36:59 UTC

REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#8) for review on master by Karthik U S (ksubrahm)

Comment 10 Worker Ant 2017-10-27 10:17:11 UTC

REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#9) for review on master by Karthik U S (ksubrahm)

Comment 11 Worker Ant 2017-10-30 07:33:16 UTC

REVIEW: https://review.gluster.org/15698 (cluster/dht: Lack of atomicity b/w read-src and write-dst of rebalance process) posted (#10) for review on master by Karthik U S (ksubrahm)

Comment 12 Karthik U S 2019-08-30 10:21:07 UTC

This issue is being tracked by https://github.com/gluster/glusterfs/issues/308.
Since there is no active work going on this closing this for now.
Feel free to reopen this or open a new bug when needed.