Red Hat Bugzilla – Bug 1022510
GlusterFS client crashes during add-brick and rebalance
Last modified: 2015-10-07 09:15:42 EDT
Created attachment 815391 [details]
GlusterFS client log during rebalance
Description of problem:
GlusterFS client crashes during rebalance after add-brick.
GlusterFS setup before add-brick
Volume Name: dev-el6-sata1
Volume ID: 840eccd5-b3fb-4dc8-b67d-966bd22e8557
Number of Bricks: 1 x 2 = 2
Version-Release number of selected component (if applicable):
RHEL 6.5 (beta)
Steps to Reproduce:
Backend filesystem is on logical volume mounted as:
/dev/mapper/sata--brick1-export on /gluster/sata/brick1 type xfs (rw,noatime,inode64,nobarrier,nobarrier)
For testing purposes new bricks are on same logical volume as older ones
1. gluster vol add-brick dev-el6-sata1 replica 2 boar1:/gluster/sata/brick1/dev-el6-sata2 boar2:/gluster/sata/brick1/dev-el6-sata2
2. gluster vol rebalance dev-el6-sata1 fix-layout start
3. gluster vol rebalance dev-el6-sata1 start
GlusterFS client crashes during rebalance and mount point goes unaccessible (Transpoint endpoint is not connected). After rebalance is finished it's required to use umount -fl to unmount the volume.
Gluster client doesn't crash and mount point is usable during rebalance.
Created attachment 815392 [details]
Backtrace of coredump
*** Bug 1104940 has been marked as a duplicate of this bug. ***
afaict, this bug occurs as the file is migrated to a different server and a fuse cache invalidation is triggered.
I'm not sure if this is relevant.
On the source server, some of the files that were migrated to the destination are still showing as open in lsof, despite their having been deleted.
Hey Joe, a patch(http://review.gluster.org/#/c/8029/) is sent addressing the same crash as part of the bug: https://bugzilla.redhat.com/show_bug.cgi?id=961615
*** Bug 1019874 has been marked as a duplicate of this bug. ***
In bug 961615 (above) I tested the backport against 3.4.4. Prior to applying the patch I could crash the clients every time. After the patch I could not. (Yes, I reviewed it verified)
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5.
This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "email@example.com".
If there is no response by the end of the month, this bug will get automatically closed.
GlusterFS 3.4.x has reached end-of-life.
If this bug still exists in a later release please reopen this and change the version or open a new bug.