Bug 1553677 - [Remove-brick] Many files were not migrated from the decommissioned bricks; commit results in data loss
Summary: [Remove-brick] Many files were not migrated from the decommissioned bricks; c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: RHGS 3.4.0
Assignee: Nithya Balachandran
QA Contact: Prasad Desala
URL:
Whiteboard: dht-data-loss
Depends On:
Blocks: 1503137 1564198 1566820 1566822
TreeView+ depends on / blocked
 
Reported: 2018-03-09 10:02 UTC by Prasad Desala
Modified: 2018-09-17 09:05 UTC (History)
4 users (show)

Fixed In Version: glusterfs-3.12.2-8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1564198 (view as bug list)
Environment:
Last Closed: 2018-09-04 06:44:14 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 0 None None None 2018-09-04 06:45:42 UTC

Description Prasad Desala 2018-03-09 10:02:04 UTC
Description of problem:
=======================
Many files were not migrated from the decommissioned bricks; commit results in data loss.

Version-Release number of selected component (if applicable):
3.12.2-5.el7rhgs.x86_64

How reproducible:
Reporting at first occurrence

Steps to Reproduce:
===================
1) Create a x3 volume with brick-mux enabled and start it.
2) FUSE mount it on multiple clients.
3) From Client-1 : run script to create folders and files continuously 
 From client-2 : start linux kernel untar
 From client-3 : while true;do find;done
 From client-4 : while true;do ls -lRt;done
4) While step-3 is in-progress, killed server-1 brick process using kill -9 <pid>. 
As brick mux is enabled killing single brick on the server using kill -9 would take down all the bricks on the node.
5) Now, add 3 bricks to the volume and after few secs immediately start removing old bricks.
6) Wait for remove-brick to complete.

Actual results:
===============
Many files were not migrated from the decommissioned bricks; commit results in data loss.

Expected results:
=================
Remove-brick operation should migrate all the files from the decommissioned brick.

Comment 27 Nithya Balachandran 2018-04-06 10:47:39 UTC
Upstream patches:
https://review.gluster.org/#/c/19827/
https://review.gluster.org/#/c/19831

Comment 33 Prasad Desala 2018-05-08 05:53:51 UTC
Verified this BZ on glusterfs version 3.12.2-8.el7rhgs.

Followed the same steps as in the description, there are no left over files on the decommissioned bricks. 

Moving this BZ to Verified.

Comment 35 errata-xmlrpc 2018-09-04 06:44:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607


Note You need to log in before you can comment on or make changes to this bug.