Bug 1044685 - [Scale] Creating Volume With 8-Hosts and 8-Bricks each Host tasks >1.5 min
Summary: [Scale] Creating Volume With 8-Hosts and 8-Bricks each Host tasks >1.5 min
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhsc
Version: 2.1
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Sahina Bose
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-18 19:48 UTC by Matt Mahoney
Modified: 2015-12-03 17:23 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-03 17:23:09 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Matt Mahoney 2013-12-18 19:48:32 UTC
Description of problem:
Creating a 64 brick volume that consists of 8-hosts and 8-bricks each host takes 90+ seconds, and has been measured to take as long as 3 minutes.

Note: creating the same volume via glusterfs takes ~30 seconds.

Version-Release number of selected component (if applicable):
cb12

How reproducible:
Always

Steps to Reproduce:
1. Create a cluster
2. Add 8 servers to the cluster
3. Create a 64 brick volume (8-hosts with 8-bricks per host). Note the time for the volume to be added to the Volume list and for the New Volume dialog to close.

Actual results:


Expected results:


Additional info:

Comment 2 Ben England 2014-01-28 14:12:12 UTC
Unfortunately I agree that it's low priority for RHS 2.1 because how many customers are using 64 bricks, and how often do they have to do this.  But if we ever want to support JBOD configurations or higher-density servers (e.g. 60 disks/server), this has to be fixed. 

I'm assuming that none of your 64 nodes are sick.  Did you try it for 32 bricks?  16 bricks?  How long did it take in those cases?

addition of bricks should be parallelized and should not take significantly more than the max time required for any brick.   Is glusterfs doing this?

  for each brick
   initialize

If so instead do this:

  for each brick
   start initialization
  for each brick
   wait for initialization to complete

Or use multiple threads.

Comment 3 Ben England 2014-05-30 10:59:53 UTC
Matt, see comment below, this may be a solution for you.

https://bugzilla.redhat.com/show_bug.cgi?id=1044693#c8

Comment 4 Vivek Agarwal 2015-12-03 17:23:09 UTC
Thank you for submitting this issue for consideration in Red Hat Gluster Storage. The release for which you requested us to review, is now End of Life. Please See https://access.redhat.com/support/policy/updates/rhs/

If you can reproduce this bug against a currently maintained version of Red Hat Gluster Storage, please feel free to file a new report against the current release.


Note You need to log in before you can comment on or make changes to this bug.