Bug 1746027

Summary: systemctl start glusterd is getting timed out on the scaled setup with 2000 volumes
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Bala Konda Reddy M <bmekala>
Component: glusterdAssignee: Mohit Agrawal <moagrawa>
Status: CLOSED ERRATA QA Contact: Bala Konda Reddy M <bmekala>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.5CC: amukherj, rhs-bugs, sheggodu, srakonde, storage-qa-internal, vbellur, vdas
Target Milestone: ---   
Target Release: RHGS 3.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-6.0-14 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1746228 (view as bug list) Environment:
Last Closed: 2019-10-30 12:23:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696809, 1746228    

Description Bala Konda Reddy M 2019-08-27 13:50:29 UTC 
Description of problem:
On a three node cluster with 2000 replica volumes on systemctl glusterd start seeing the command getting timed out as below.

[root@dhcp46-222 ~]# systemctl start glusterd
Job for glusterd.service failed because a timeout was exceeded. See "systemctl status glusterd.service" and "journalctl -xe" for details.



Version-Release number of selected component (if applicable):
glusterfs-6.0-13.el7rhgs.x86_64

How reproducible:
2/2

Steps to Reproduce:
1. On three node cluster, enable brick-multiplexing and create 2000 replica volumes
2. Stop glusterd -> "systemctl stop glusterd", pkill glusterfsd, pkill glusterfs
3. Start glusterd -> "systemctl start glusterd"

Actual results:
[root@dhcp46-222 ~]# systemctl start glusterd
Job for glusterd.service failed because a timeout was exceeded. See "systemctl status glusterd.service" and "journalctl -xe" for details.

Expected results:
systemctl start glusterd, should start the daemon without any issue

Additional info:

Journalctl -xe output
Aug 27 15:35:08 dhcp46-222.lab.eng.blr.redhat.com systemd[1]: glusterd.service start operation timed out. Terminating.
Aug 27 15:35:08 dhcp46-222.lab.eng.blr.redhat.com systemd[1]: Failed to start GlusterFS, a clustered file-system server.
-- Subject: Unit glusterd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit glusterd.service has failed.
Comment 9 errata-xmlrpc 2019-10-30 12:23:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249