Description of problem: ====================== Rebalance failed when a plain disperse 1x(8+4) volume is converted to distributed disperse volume 2x(8+4). [root@vertigo ~]# gluster v rebalance testvol status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 224 11.9MB 29485 0 14532 in progress 48509.00 ninja 0 0Bytes 49629 0 0 in progress 48509.00 transformers 0 0Bytes 60960 0 0 in progress 48511.00 interstellar 0 0Bytes 0 0 0 failed 1759.00 volume rebalance: testvol: success: [root@vertigo ~]# Version-Release number of selected component (if applicable): ============================================================= [root@vertigo ~]# gluster --version glusterfs 3.7dev built on Apr 18 2015 01:08:09 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@vertigo ~]# How reproducible: ================= 100% Steps to Reproduce: 1. Create a disperse volume 1x(8+4) 2. Create data of around 100GB. 3. Add bricks to convert the disperse volume to distributed disperse volume 4. Trigger rebalance on the volume Actual results: ============== Failed Expected results: Additional info: ================ sosreports will be copied to rhsqe-repo/sosreports/<budid> folder.
Rebalance is going on from 12 hrs and only 12MB of data is moved.
Volume status and info : ======================== [root@ninja ~]# gluster v status Status of volume: testvol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick vertigo:/rhs/brick1/b1 49156 0 Y 21054 Brick ninja:/rhs/brick1/b2 49152 0 Y 11259 Brick transformers:/rhs/brick1/b3 49152 0 Y 2432 Brick interstellar:/rhs/brick1/b4 49152 0 Y 2603 Brick vertigo:/rhs/brick2/b5 49157 0 Y 21071 Brick ninja:/rhs/brick2/b6 49153 0 Y 11276 Brick transformers:/rhs/brick2/b7 49153 0 Y 2473 Brick interstellar:/rhs/brick2/b8 49153 0 Y 2620 Brick vertigo:/rhs/brick3/b9 49158 0 Y 21088 Brick ninja:/rhs/brick3/b10 49154 0 Y 11293 Brick transformers:/rhs/brick3/b11 49154 0 Y 2490 Brick interstellar:/rhs/brick3/b12 49154 0 Y 2637 Brick vertigo:/rhs/brick4/b13 49160 0 Y 21928 Brick ninja:/rhs/brick4/b14 49156 0 Y 12130 Brick transformers:/rhs/brick4/b15 49156 0 Y 3264 Brick interstellar:/rhs/brick4/b16 49156 0 Y 3416 Brick vertigo:/rhs/brick1/b17 49161 0 Y 21947 Brick ninja:/rhs/brick1/b18 49157 0 Y 12149 Brick transformers:/rhs/brick5/b19 49157 0 Y 3283 Brick interstellar:/rhs/brick5/b20 49157 0 Y 3435 Brick vertigo:/rhs/brick2/b21 49162 0 Y 21966 Brick ninja:/rhs/brick2/b22 49158 0 Y 12168 Brick transformers:/rhs/brick6/b23 49158 0 Y 3302 Brick interstellar:/rhs/brick6/b24 49158 0 Y 3454 Snapshot Daemon on localhost 49155 0 Y 11363 NFS Server on localhost 2049 0 Y 12188 Quota Daemon on localhost N/A N/A Y 12207 Snapshot Daemon on 10.70.34.56 49159 0 Y 21167 NFS Server on 10.70.34.56 2049 0 Y 21987 Quota Daemon on 10.70.34.56 N/A N/A Y 22004 Snapshot Daemon on interstellar 49155 0 Y 2707 NFS Server on interstellar 2049 0 Y 3477 Quota Daemon on interstellar N/A N/A Y 3495 Snapshot Daemon on transformers 49155 0 Y 2562 NFS Server on transformers 2049 0 Y 3325 Quota Daemon on transformers N/A N/A Y 3344 Task Status of Volume testvol ------------------------------------------------------------------------------ Task : Rebalance ID : 5fe22604-3b35-4557-8bd6-8022d7c18905 Status : in progress [root@ninja ~]# gluster v i Volume Name: testvol Type: Distributed-Disperse Volume ID: 811f6c7d-6b70-4795-8585-d1121b32a673 Status: Started Number of Bricks: 2 x (8 + 4) = 24 Transport-type: tcp Bricks: Brick1: vertigo:/rhs/brick1/b1 Brick2: ninja:/rhs/brick1/b2 Brick3: transformers:/rhs/brick1/b3 Brick4: interstellar:/rhs/brick1/b4 Brick5: vertigo:/rhs/brick2/b5 Brick6: ninja:/rhs/brick2/b6 Brick7: transformers:/rhs/brick2/b7 Brick8: interstellar:/rhs/brick2/b8 Brick9: vertigo:/rhs/brick3/b9 Brick10: ninja:/rhs/brick3/b10 Brick11: transformers:/rhs/brick3/b11 Brick12: interstellar:/rhs/brick3/b12 Brick13: vertigo:/rhs/brick4/b13 Brick14: ninja:/rhs/brick4/b14 Brick15: transformers:/rhs/brick4/b15 Brick16: interstellar:/rhs/brick4/b16 Brick17: vertigo:/rhs/brick1/b17 Brick18: ninja:/rhs/brick1/b18 Brick19: transformers:/rhs/brick5/b19 Brick20: interstellar:/rhs/brick5/b20 Brick21: vertigo:/rhs/brick2/b21 Brick22: ninja:/rhs/brick2/b22 Brick23: transformers:/rhs/brick6/b23 Brick24: interstellar:/rhs/brick6/b24 Options Reconfigured: client.event-threads: 4 server.event-threads: 4 features.quota: on features.uss: on [root@ninja ~]#
Bhaskar confirmed that the failure was because of a crash which is being tracked in BZ#1215117. Hence marking this bug a duplicate of 1215117. *** This bug has been marked as a duplicate of bug 1215117 ***
Verified this on 3.7.1-1 build and rebalance is successful. Moving this to verified.
as per comment #9, this is fixed.