Bug 1044685

Summary: [Scale] Creating Volume With 8-Hosts and 8-Bricks each Host tasks >1.5 min
Product: Red Hat Gluster Storage Reporter: Matt Mahoney <mmahoney>
Component: rhscAssignee: Sahina Bose <sabose>
Status: CLOSED EOL QA Contact: storage-qa-internal <storage-qa-internal>
Severity: low Docs Contact:
Priority: low    
Version: 2.1CC: bengland, knarra, mmahoney, mmccune, rhs-bugs, sabose, sasundar, ssampat
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-03 17:23:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Matt Mahoney 2013-12-18 19:48:32 UTC 
Description of problem:
Creating a 64 brick volume that consists of 8-hosts and 8-bricks each host takes 90+ seconds, and has been measured to take as long as 3 minutes.

Note: creating the same volume via glusterfs takes ~30 seconds.

Version-Release number of selected component (if applicable):
cb12

How reproducible:
Always

Steps to Reproduce:
1. Create a cluster
2. Add 8 servers to the cluster
3. Create a 64 brick volume (8-hosts with 8-bricks per host). Note the time for the volume to be added to the Volume list and for the New Volume dialog to close.

Actual results:


Expected results:


Additional info:
Comment 2 Ben England 2014-01-28 14:12:12 UTC 
Unfortunately I agree that it's low priority for RHS 2.1 because how many customers are using 64 bricks, and how often do they have to do this.  But if we ever want to support JBOD configurations or higher-density servers (e.g. 60 disks/server), this has to be fixed. 

I'm assuming that none of your 64 nodes are sick.  Did you try it for 32 bricks?  16 bricks?  How long did it take in those cases?

addition of bricks should be parallelized and should not take significantly more than the max time required for any brick.   Is glusterfs doing this?

  for each brick
   initialize

If so instead do this:

  for each brick
   start initialization
  for each brick
   wait for initialization to complete

Or use multiple threads.
Comment 3 Ben England 2014-05-30 10:59:53 UTC 
Matt, see comment below, this may be a solution for you.

https://bugzilla.redhat.com/show_bug.cgi?id=1044693#c8
Comment 4 Vivek Agarwal 2015-12-03 17:23:09 UTC 
Thank you for submitting this issue for consideration in Red Hat Gluster Storage. The release for which you requested us to review, is now End of Life. Please See https://access.redhat.com/support/policy/updates/rhs/

If you can reproduce this bug against a currently maintained version of Red Hat Gluster Storage, please feel free to file a new report against the current release.