Bug 1517987

Summary:	[GSS] high mem/cpu usage, brick processes not starting and ssl encryption issues while testing CRS scaling with multiplexing (500-800 vols)
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Pan Ousley <pousley>
Component:	core	Assignee:	Mohit Agrawal <moagrawa>
Status:	CLOSED ERRATA	QA Contact:	Bala Konda Reddy M <bmekala>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, bmekala, kaushal, moagrawa, nchilaka, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	RHGS 3.4.0
Hardware:	All
OS:	Linux
Whiteboard:	brick-multiplexing
Fixed In Version:	glusterfs-3.12.2-4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1520245 (view as bug list)		Environment:
Last Closed:	2018-09-04 06:39:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1520245
Bug Blocks:	1503138

Description Pan Ousley 2017-11-27 20:27:00 UTC

Description of problem:

A customer is testing CRS scaling with brick multiplexing, with and without SSL encryption. They are encountering unexpected behavior, high cpu/mem usage and are unable to provision volumes with SSL encryption enabled.

These are the most recent tests, after disabling monitoring to avoid locking issues:

* rhgs-az[b,c,d]-1 - 3 node cluster with SSL encryption enabled - 500 volumes
* rhgs-az[b,c,d]-2 - 3 node cluster without encryption - 500 volumes
* rhs-c01-n0[1-6] - 6 node cluster without encryption - 800 volumes

Results:

- glusterd process eats a huge amount of memory (all available) during volume provisioning. No matter if SSL enabled or disabled. Memory consumption is lower (50%) after bouncing - restarting all gluster processes. 

- provisioning succeeds only on non-encrypted clusters 

- volumes are not started properly ~30 bricks processes are not started after provisioning. Needs gluster volume start force, or restarting glusterd on all nodes

- SSL encrypted cluster fails to provision volumes after some time, ends in non-consistent state.

- SSL encrypted cluster consumes more resources (RAM,CPU), but that's expected, due to encryption layer

- SSL errors like these:

/var/log/glusterfs/glusterd.log:[2017-11-17 17:49:11.859095] E [socket.c:2510:socket_poller] 0-socket.management: server setup failed
/var/log/glusterfs/glusterd.log:[2017-11-17 17:49:11.859779] E [socket.c:358:ssl_setup_connection] 0-socket.management: SSL connect error (client: 10.3.107.49:1013) (server: 10.3.104.154:24007)
/var/log/glusterfs/glusterd.log:[2017-11-17 17:49:11.859984] E [socket.c:2510:socket_poller] 0-socket.management: server setup failed
/var/log/glusterfs/glusterd.log:[2017-11-17 17:49:11.954674] E [socket.c:358:ssl_setup_connection] 0-socket.management: SSL connect error (client: 10.3.104.154:974) (server: 10.3.104.154:24007)
/var/log/glusterfs/glusterd.log:[2017-11-17 17:49:11.955464] E [socket.c:2510:socket_poller] 0-socket.management: server setup failed
/var/log/glusterfs/glusterd.log:[2017-11-17 17:49:11.955829] E [socket.c:358:ssl_setup_connection] 0-socket.management: SSL connect error (client: 10.3.104.154:996) (server: 10.3.104.154:24007)


Are there any recommended settings for cluster.max-bricks-per-process? The customer tried to use a single brick process for all volumes but encountered high mem/cpu usage. It looks like 15-25 is working fine for non-encrypted clusters.


Version-Release number of selected component (if applicable): RHGS 3.3, CRS 3.6


Please let me know what information would be useful for investigating these issues. The customer has uploaded many log files to the case which can be accessed in collab-shell.

Comment 35 Bala Konda Reddy M 2018-05-23 10:06:24 UTC

From comment 29:

Build: 3.12.2-9

On a three node cluster brick-mux enabled, with and without SSL enabled.
Created 500+ volumes of type replicate(2X3). Performed node reboot scenarios and volume set operation when a node is down. After the node reboot, all bricks are online.
Haven't seen much hike in glusterd memory from 11MB, it increased to around 300MB which is acceptable, in all the nodes in the cluster.

Hence marking it as verified

Comment 37 errata-xmlrpc 2018-09-04 06:39:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607