Bug 1465290 - [Scale] : Rebalancing a volume with no files takes a lot more time compared to RHGS 3.2 [NEEDINFO]
[Scale] : Rebalancing a volume with no files takes a lot more time compared t...
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
3.3
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Nithya Balachandran
Ambarish
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-27 02:42 EDT by Ambarish
Modified: 2017-07-06 04:54 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
nbalacha: needinfo? (asoman)


Attachments (Terms of Use)

  None (edit)
Description Ambarish 2017-06-27 02:42:45 EDT
Description of problem:
------------------------

I attempted to create an EC volume with a lot of bricks 12*(4+2).

Since there are 72 bricks,I generally create a 1*(4+2) and add bricks 11 times to make it a 11*(4+2),mostly to avoid mistakes,and its easier this way with linux CLI hacks.

This was my use case in 3.2 and 3.3.


************
On 3.2 gold
************


[root@gqas013 ~]# gluster v rebalance butcher status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0            completed        0:0:1
      gqas005.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas006.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas008.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas007.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas014.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
volume rebalance: butcher: success
[root@gqas013 ~]# 

***********
On 3.8.4-31
***********

This was the worst case I hit :

[root@gqas013 ~]# gluster v rebalance butcher status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0            completed        0:00:26
      gqas005.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas006.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas008.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas007.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas014.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
volume rebalance: butcher: success
[root@gqas013 ~]# 


Version-Release number of selected component (if applicable):
--------------------------------------------------------------

3.8.4-31

How reproducible:
-----------------

Every which way I try on EC.


Actual results:
---------------

Slower rebalance completion.

Expected results:
-----------------

Rebalance completion time should  be comparable to 3.2 


Additional info:
-----------------


[root@gqas013 ~]# gluster v info
 
Volume Name: butcher
Type: Distributed-Disperse
Volume ID: 0f76daa9-4069-464e-a242-d5c6f86fd9dc
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x (4 + 2) = 72
Transport-type: tcp
Bricks:
Brick1: gqas005.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick2: gqas006.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick3: gqas013.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick5: gqas007.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick6: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick7: gqas005.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick8: gqas006.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick9: gqas013.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick10: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick11: gqas007.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick12: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick13: gqas005.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick14: gqas006.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick15: gqas013.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick16: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick17: gqas007.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick18: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick19: gqas005.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick20: gqas006.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick21: gqas013.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick22: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick23: gqas007.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick24: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick25: gqas005.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick26: gqas006.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick27: gqas013.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick28: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick29: gqas007.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick30: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick31: gqas005.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick32: gqas006.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick33: gqas013.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick34: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick35: gqas007.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick36: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick37: gqas005.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick38: gqas006.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick39: gqas013.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick40: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick41: gqas007.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick42: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick43: gqas005.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick44: gqas006.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick45: gqas013.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick46: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick47: gqas007.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick48: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick49: gqas005.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick50: gqas006.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick51: gqas013.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick52: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick53: gqas007.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick54: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick55: gqas005.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick56: gqas006.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick57: gqas013.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick58: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick59: gqas007.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick60: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick61: gqas005.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick62: gqas006.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick63: gqas013.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick64: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick65: gqas007.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick66: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick67: gqas005.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick68: gqas006.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick69: gqas013.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick70: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick71: gqas007.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick72: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Options Reconfigured:
client.event-threads: 4
server.event-threads: 4
network.inode-lru-limit: 50000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: off
[root@gqas013 ~]#

Note You need to log in before you can comment on or make changes to this bug.