1465290 – [Scale] : Rebalancing a volume with no files takes a lot more time compared to RHGS 3.2

Bug 1465290 - [Scale] : Rebalancing a volume with no files takes a lot more time compared to RHGS 3.2

Summary: [Scale] : Rebalancing a volume with no files takes a lot more time compared t...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	distribute
Sub Component:
Version:	rhgs-3.3
Hardware:	x86_64
OS:	Linux
Priority:	low
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Nithya Balachandran
QA Contact:	Ambarish
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-27 06:42 UTC by Ambarish
Modified:	2023-09-14 03:59 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-08-29 06:05:17 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Ambarish 2017-06-27 06:42:45 UTC

Description of problem:
------------------------

I attempted to create an EC volume with a lot of bricks 12*(4+2).

Since there are 72 bricks,I generally create a 1*(4+2) and add bricks 11 times to make it a 11*(4+2),mostly to avoid mistakes,and its easier this way with linux CLI hacks.

This was my use case in 3.2 and 3.3.


************
On 3.2 gold
************


[root@gqas013 ~]# gluster v rebalance butcher status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0            completed        0:0:1
      gqas005.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas006.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas008.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas007.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
      gqas014.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:0:3
volume rebalance: butcher: success
[root@gqas013 ~]# 

***********
On 3.8.4-31
***********

This was the worst case I hit :

[root@gqas013 ~]# gluster v rebalance butcher status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0            completed        0:00:26
      gqas005.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas006.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas008.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas007.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
      gqas014.sbu.lab.eng.bos.redhat.com                0        0Bytes             0             0             0            completed        0:00:26
volume rebalance: butcher: success
[root@gqas013 ~]# 


Version-Release number of selected component (if applicable):
--------------------------------------------------------------

3.8.4-31

How reproducible:
-----------------

Every which way I try on EC.


Actual results:
---------------

Slower rebalance completion.

Expected results:
-----------------

Rebalance completion time should  be comparable to 3.2 


Additional info:
-----------------


[root@gqas013 ~]# gluster v info
 
Volume Name: butcher
Type: Distributed-Disperse
Volume ID: 0f76daa9-4069-464e-a242-d5c6f86fd9dc
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x (4 + 2) = 72
Transport-type: tcp
Bricks:
Brick1: gqas005.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick2: gqas006.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick3: gqas013.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick5: gqas007.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick6: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/A1
Brick7: gqas005.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick8: gqas006.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick9: gqas013.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick10: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick11: gqas007.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick12: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/A1
Brick13: gqas005.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick14: gqas006.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick15: gqas013.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick16: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick17: gqas007.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick18: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/A1
Brick19: gqas005.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick20: gqas006.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick21: gqas013.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick22: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick23: gqas007.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick24: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/A1
Brick25: gqas005.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick26: gqas006.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick27: gqas013.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick28: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick29: gqas007.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick30: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/A1
Brick31: gqas005.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick32: gqas006.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick33: gqas013.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick34: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick35: gqas007.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick36: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/A1
Brick37: gqas005.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick38: gqas006.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick39: gqas013.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick40: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick41: gqas007.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick42: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/A1
Brick43: gqas005.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick44: gqas006.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick45: gqas013.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick46: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick47: gqas007.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick48: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/A1
Brick49: gqas005.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick50: gqas006.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick51: gqas013.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick52: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick53: gqas007.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick54: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/A1
Brick55: gqas005.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick56: gqas006.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick57: gqas013.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick58: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick59: gqas007.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick60: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/A1
Brick61: gqas005.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick62: gqas006.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick63: gqas013.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick64: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick65: gqas007.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick66: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/A1
Brick67: gqas005.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick68: gqas006.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick69: gqas013.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick70: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick71: gqas007.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Brick72: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/A1
Options Reconfigured:
client.event-threads: 4
server.event-threads: 4
network.inode-lru-limit: 50000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: off
[root@gqas013 ~]#

Comment 10 Nithya Balachandran 2017-08-29 06:05:17 UTC

There have been several changes made to rebalance for the estimates which could increase the time it takes to start up.

After discussing this with Ambarish, I cam closing this with resolution WontFix.

Comment 11 Red Hat Bugzilla 2023-09-14 03:59:55 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days

Note You need to log in before you can comment on or make changes to this bug.