Bug 1516417

Summary: Expanding an existing RHGS cluster managed by RHGS WA by adding nodes and monitoring
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Daniel Horák <dahorak>
Component: web-admin-tendrl-monitoring-integrationAssignee: gowtham <gshanmug>
Status: CLOSED ERRATA QA Contact: Filip Balák <fbalak>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.3CC: fbalak, jefbrown, mbukatov, nthomas, rcyriac, rghatvis, rhinduja, rhs-bugs, sankarshan, ssaha
Target Milestone: ---Keywords: FutureFeature
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tendrl-commons-1.6.1-2.el7rhgs tendrl-node-agent-1.6.1-2.el7rhgs tendrl-api-1.6.1-2.el7rhgs tendrl-gluster-integration-1.6.1-2.el7rhgs tendrl-ui-1.6.1-2.el7rhgs tendrl-monitoring-integration-1.6.1-2.el7rhgs Doc Type: Enhancement
Doc Text:
Previously, Web Administration was unable to detect or expand new storage nodes added to gluster trusted storage pool. As a result, Web Administration could not manage and provide metrics to newly added nodes to a cluster after initial Import Cluster. With this fix, Web Administration can now detect and expand new nodes to an already managed cluster once the new nodes are added to the gluster trusted storage pool.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 06:58:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1503132, 1517065    

Description Daniel Horák 2017-11-22 15:11:59 UTC
Description of problem:
   of Gluster cluster previously imported into Tendrl is not properly reflected in Grafana. 
The new nodes are visible in Tendrl UI, but there are no statistics in Grafana for this new nodes.

Version-Release number of selected component (if applicable):
  Tendrl Server
  tendrl-ansible-1.5.4-1.el7rhgs.noarch
  tendrl-api-1.5.4-2.el7rhgs.noarch
  tendrl-api-httpd-1.5.4-2.el7rhgs.noarch
  tendrl-commons-1.5.4-3.el7rhgs.noarch
  tendrl-grafana-plugins-1.5.4-4.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.3-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.5.4-4.el7rhgs.noarch
  tendrl-node-agent-1.5.4-3.el7rhgs.noarch
  tendrl-notifier-1.5.4-2.el7rhgs.noarch
  tendrl-selinux-1.5.3-2.el7rhgs.noarch
  tendrl-ui-1.5.4-3.el7rhgs.noarch

  Gluster Storage Server
  tendrl-collectd-selinux-1.5.3-2.el7rhgs.noarch
  tendrl-commons-1.5.4-3.el7rhgs.noarch
  tendrl-node-agent-1.5.4-3.el7rhgs.noarch
  tendrl-selinux-1.5.3-2.el7rhgs.noarch

How reproducible:
  100%

Steps to Reproduce:
1. Prepare, install and configure Gluster Cluster.
2. Prepare, install and configure Tendrl server.
3. Configure Tendrl nodes on Gluster Storage nodes.
4. Import Gluster Cluster into Tendrl.

5. Prepare additional Gluster Storage nodes.
6. Extend the existing Gluster cluster by the additional Storage nodes.
  (peer probe via glsuter command or via gdeploy)
7. Install and configure Tendrl nodes on the newly added Storage nodes.

Actual results:
  Newly added Storage nodes are visible in Tendrl UI, but they are not visible in Grafana.

Expected results:
  Gluster expansion should be properly reflected in both Tendrl and Grafana.

Additional info:
  The main problem seems to be, that tendrl-gluster-integration and other packages installed automatically by Tendrl during importing the cluster, are not installed on the newly added nodes and there is no way how to install and configure them.

Comment 4 Daniel Horák 2017-11-23 09:48:53 UTC
just small update: There seems to be issues also in Tendrl. When I created volume on the newly added hosts (or extend existing volume with bricks from the new hosts), the new volume is visible in Tendrl Cluster -> Volumes tab, but there are no details related to Bricks on the new hosts.

Comment 11 Martin Bukatovic 2018-04-04 08:27:50 UTC
QE note: this should be tested together with BZ 1559368.

Comment 12 Filip Balák 2018-04-09 12:55:08 UTC
I have expanded cluster and new hosts are shown in grafana correctly. I have also added new bricks on new hosts to the volume that was part of cluster before expand. New brick dashboards are shown correctly. There is an issue with creating empty dashboards before user triggers cluster expand (BZ 1564510) but otherwise the grafana dashboards are created correctly. --> VERIFIED

Tested with:
tendrl-ansible-1.6.1-3.el7rhgs.noarch
tendrl-api-1.6.1-3.el7rhgs.noarch
tendrl-api-httpd-1.6.1-3.el7rhgs.noarch
tendrl-commons-1.6.1-3.el7rhgs.noarch
tendrl-grafana-plugins-1.6.1-3.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-monitoring-integration-1.6.1-3.el7rhgs.noarch
tendrl-node-agent-1.6.1-3.el7rhgs.noarch
tendrl-notifier-1.6.0-1.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-ui-1.6.1-3.el7rhgs.noarch
glusterfs-3.12.2-7.el7rhgs.x86_64

Comment 17 errata-xmlrpc 2018-09-04 06:58:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616