Bug 1141172 - DATA LOSS - rename from multiple mount ends in data loss(files are deleted), rename from one mount + rebalance + lookup also ends in data loss
Summary: DATA LOSS - rename from multiple mount ends in data loss(files are deleted), ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Nithya Balachandran
QA Contact: Matt Zywusko
URL:
Whiteboard: dht-data-loss, dht-pm-query
Depends On: 1141368 1166570
Blocks: 1087818
TreeView+ depends on / blocked
 
Reported: 2014-09-12 11:34 UTC by Rachana Patel
Modified: 2016-09-16 06:13 UTC (History)
9 users (show)

Fixed In Version: 3.7.9-10
Doc Type: Known Issue
Doc Text:
When a file is renamed from multiple mount points, there is a possibility to lose the file. This is noticed when the mv command internally processes it as an unlink operation after a rename, when the source and destination paths are hardlinks of each other. This issue occurs in the coreutil packages in various Linux distributions. For example, if there are parallel rename operation (mv a b) and (mv b a) where a and b are hardlinks to the same file, because of the above mentioned behaviour of mv, unlink (a) and unlink (b) would be issued from both instances of mv. This results in losing both the links a and b and hence the file.
Clone Of:
Environment:
Last Closed: 2016-09-16 06:13:50 UTC
Embargoed:


Attachments (Terms of Use)

Description Rachana Patel 2014-09-12 11:34:26 UTC
Description of problem:
=======================
DATA LOSS - rename from multiple mount ends in data loss(files are deleted), rename from one mount + rebalance + lookup also ends in data loss

Version-Release number of selected component (if applicable):
=============================================================
3.6.0.28-1.el6rhs.x86_64

How reproducible:
=================
always

Steps to Reproduce:
===================
1. created Ditributed volume having 4 bricks.
2. created 100 files on mount - mb{1..100}-1
case 1:- rename from multiple mount :-
3. create link file for each file. eg. for file 'a', create link 'b' and then from one mount execute'mv a b' and from another mount execute 'mv b a' at same time. keep sending lookup from another mount.
4.  many files are missing from mount  as well as backend
*{3,5,17,18,20,15,31,36,41,52,58,62,65,74,76,77,85,87,95}-1 are missing

case 2: - rename from one mount + rebalance + lookup
5. now run the rename from one mount as below
 for file 'a', create link 'b' and then from one mount execute'mv a b'
at same time add-brick and start rebalance and keep sending lookup from another mount

6. many files are missing from mount as well as bricks
*{46,64,97}-1 are missing

Actual results:
===============
data loss - files are missing from mount and brick




Document URL: 

Section Number and Name: 

Describe the issue: 

Suggestions for improvement: 

Additional information: 


Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 Shyamsundar 2014-09-12 19:46:37 UTC
Added a depends on to the RHEL bug on mv (please read root causes more on that bug). This is for case 1 issues mentioned in the Description.

Comment 5 Sayan Saha 2014-09-12 20:26:06 UTC
If RHEL's "mv" has an issue we can't have this as a blocker for Denali.

Comment 6 Shalaka 2014-09-22 10:33:19 UTC
Please review and sign-off edited doc text.

Comment 7 Pavithra 2014-11-05 07:39:46 UTC
Hi Du, 

Can you please sign off the edited known issue doc text?

Comment 10 Raghavendra G 2016-07-01 06:13:08 UTC
Bz 1166570, on which this bug depends on, has been resolved in RHEL-7.2. Since we shipped rhgs-3.1.3 on RHEL-7.2, this bug should not be present on rhel-3.1.3

Status: ASSIGNED → MODIFIED
Fixed In Version: coreutils-8.22-13.el7

Comment 11 Prasad Desala 2016-09-16 06:06:22 UTC
Verified this bug on glusterfs version 3.7.9-12.el7rhgs.x86_64.

Here are the steps that were performed,

1. Created a ditributed replica volume and started it.
2. FUSE and nfs mounted the volume on two different clients.
3. Created files on the mount point.
case 1:- rename from multiple mounts:
4. Created link file for each file. eg. for file 'a', create link 'b' and then from one mount executed 'mv a b' and from another mount executed 'mv b a' at same time. Also, sending continuous lookups from one mount.
5. Checked the files count after some time. File count remains same and No data loss was seen.
case 2: - rename from one mount + rebalance + lookup
6. Renamed files from one mount as below
 for file 'a', create link 'b' and then from one mount execute'mv a b'
at same time add-brick and started rebalance and continuous lookups from another mount
7. Checked the file count after rebalance. File count remains same and No data loss was seen.

Hence, moving this bug to Verified.

Comment 12 Nithya Balachandran 2016-09-16 06:13:50 UTC
Thanks Prasad.

I am closing this BZ based on comment#11


Note You need to log in before you can comment on or make changes to this bug.