Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1553677 - [Remove-brick] Many files were not migrated from the decommissioned bricks; commit results in data loss
[Remove-brick] Many files were not migrated from the decommissioned bricks; c...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
3.4
Unspecified Unspecified
high Severity high
: ---
: RHGS 3.4.0
Assigned To: Nithya Balachandran
Prasad Desala
dht-data-loss
:
Depends On:
Blocks: 1503137 1564198 1566820 1566822
  Show dependency treegraph
 
Reported: 2018-03-09 05:02 EST by Prasad Desala
Modified: 2018-09-17 05:05 EDT (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.12.2-8
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1564198 (view as bug list)
Environment:
Last Closed: 2018-09-04 02:44:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 02:45 EDT

  None (edit)
Description Prasad Desala 2018-03-09 05:02:04 EST
Description of problem:
=======================
Many files were not migrated from the decommissioned bricks; commit results in data loss.

Version-Release number of selected component (if applicable):
3.12.2-5.el7rhgs.x86_64

How reproducible:
Reporting at first occurrence

Steps to Reproduce:
===================
1) Create a x3 volume with brick-mux enabled and start it.
2) FUSE mount it on multiple clients.
3) From Client-1 : run script to create folders and files continuously 
 From client-2 : start linux kernel untar
 From client-3 : while true;do find;done
 From client-4 : while true;do ls -lRt;done
4) While step-3 is in-progress, killed server-1 brick process using kill -9 <pid>. 
As brick mux is enabled killing single brick on the server using kill -9 would take down all the bricks on the node.
5) Now, add 3 bricks to the volume and after few secs immediately start removing old bricks.
6) Wait for remove-brick to complete.

Actual results:
===============
Many files were not migrated from the decommissioned bricks; commit results in data loss.

Expected results:
=================
Remove-brick operation should migrate all the files from the decommissioned brick.
Comment 27 Nithya Balachandran 2018-04-06 06:47:39 EDT
Upstream patches:
https://review.gluster.org/#/c/19827/
https://review.gluster.org/#/c/19831
Comment 33 Prasad Desala 2018-05-08 01:53:51 EDT
Verified this BZ on glusterfs version 3.12.2-8.el7rhgs.

Followed the same steps as in the description, there are no left over files on the decommissioned bricks. 

Moving this BZ to Verified.
Comment 35 errata-xmlrpc 2018-09-04 02:44:14 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607

Note You need to log in before you can comment on or make changes to this bug.