Bug 1655420

Summary:	Profiling job should be moved to beginning of the import flow.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Anmol Sachan <asachan>
Component:	web-admin-tendrl-commons	Assignee:	Anmol Sachan <asachan>
Status:	CLOSED ERRATA	QA Contact:	Filip Balák <fbalak>
Severity:	low	Docs Contact:
Priority:	low
Version:	rhgs-3.4	CC:	asachan, dahorak, fbalak, mbukatov, nthomas, rhs-bugs, sankarshan
Target Milestone:	---	Keywords:	Improvement, ZStream
Target Release:	RHGS 3.4.z Batch Update 4
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tendrl-commons-1.6.3-16.el7rhgs	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-03-27 03:49:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Anmol Sachan 2018-12-03 06:34:15 UTC

Description of problem: Profiling job should be moved to beginning of the import flow. This change is an enhancement and is required for improving the import performance. With this change coming in, it will make sure that there is no race condition between gluster-integration sds-sync and gluster profiling commands.


Version-Release number of selected component (if applicable):


How reproducible: 100 %


Steps to Reproduce: N/A


Actual results: Currently profiling job is executed during the import job as a part of the gluster-integration sds syn.


Expected results: Profiling job will be one of the initials jobs of the import flow and gluster-integration sds-sync will be invoked later on in the import flow.


Additional info:

Comment 2 Martin Bukatovic 2018-12-10 19:05:52 UTC

I have 2 questions:

1) The import should achieve the same end result, but do it in bit different order. Right?
2) What is the performance improvement you are expecting to achieve?

I'm asking 2nd question as I'm not willing to implement this change without expected performance improvement is reached.

Comment 3 Anmol Sachan 2018-12-10 19:39:02 UTC

(In reply to Martin Bukatovic from comment #2)
> I have 2 questions:
> 
> 1) The import should achieve the same end result, but do it in bit different
> order. Right?

This is correct.

> 2) What is the performance improvement you are expecting to achieve?
> 
> I'm asking 2nd question as I'm not willing to implement this change without
> expected performance improvement is reached.

Earlier profiling was being done during the gluster_integration sds_sync along with the sync. Before this point, high number of volumes were never tested for performance and scale. During the testing we observed that gluster profiling commands were competing with the gluster commands required for the sds sync. This scenario was always present there but never was really observed. When we shift the profiling to the beginning of the import, gluster integration is not started yet, which makes gluster completely free to execute the profiling commands. To give a fare comparison for improvement, for 400 volumes, the orignal code used to take range from 30 mins to more than an hour, but with this shift the profiling gets completed in around 10 mins or even less for the same number of volumes.

Comment 4 Nishanth Thomas 2018-12-13 10:09:05 UTC

This bug is taken out from BU3

Comment 6 Filip Balák 2019-02-27 12:52:35 UTC

I have tested Import task with the same fresh setup with the same 3 volumes but with different versions of tendrl:

New version:
tendrl-ansible-1.6.3-11.el7rhgs.noarch
tendrl-api-1.6.3-12.el7rhgs.noarch
tendrl-api-httpd-1.6.3-12.el7rhgs.noarch
tendrl-commons-1.6.3-17.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-20.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-20.el7rhgs.noarch
tendrl-node-agent-1.6.3-18.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-3.el7rhgs.noarch
tendrl-ui-1.6.3-15.el7rhgs.noarch


Old version:
tendrl-ansible-1.6.3-11.el7rhgs.noarch
tendrl-api-1.6.3-10.el7rhgs.noarch
tendrl-api-httpd-1.6.3-10.el7rhgs.noarch
tendrl-commons-1.6.3-15.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-20.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-20.el7rhgs.noarch
tendrl-node-agent-1.6.3-15.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-3.el7rhgs.noarch
tendrl-ui-1.6.3-14.el7rhgs.noarch

According to test results is cluster import time reduced in newer version. Median of 3 test run times with new version is 63 seconds and median of 3 test run times with older version is 85 seconds. All cluster imports with new version were finished before any import with older version.

In task messages is now message `Starting profiling for volumes` before importing hosts.

--> VERIFIED

Comment 8 errata-xmlrpc 2019-03-27 03:49:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0660