Bug 1599634

Summary: Expand cluster imports only one node
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Filip Balák <fbalak>
Component: web-admin-tendrl-node-agentAssignee: Shubhendu Tripathi <shtripat>
Status: CLOSED ERRATA QA Contact: Filip Balák <fbalak>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: dahorak, mbukatov, nthomas, rhs-bugs, sankarshan, shtripat
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tendrl-node-agent-1.6.3-9.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 07:08:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1503137    
Attachments:
Description Flags
Expand task
none
1 - Node is expanded by task
none
2 - Node remains unmanaged none

Description Filip Balák 2018-07-10 09:04:26 UTC 
Created attachment 1457720 [details]
Expand task

Description of problem:
When expand cluster job is triggered it imports only one node. After that it is marked as finished but the other node remains unmanaged and option expand is no longer possible. The node that remains unmangaged is the one that have created Host dashboard before Cluster Expand is triggered as described in BZ 1599630.


Version-Release number of selected component (if applicable):
tendrl-ansible-1.6.3-5.el7rhgs.noarch
tendrl-api-1.6.3-4.el7rhgs.noarch
tendrl-api-httpd-1.6.3-4.el7rhgs.noarch
tendrl-commons-1.6.3-8.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-6.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-6.el7rhgs.noarch
tendrl-node-agent-1.6.3-8.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-ui-1.6.3-6.el7rhgs.noarch


How reproducible:
100%

Steps to Reproduce:
1. Import cluster with 4 nodes into WA.
2. Add 2 more nodes to cluster and create distributed replicated volume.
3. Restart all nodes.
4. Expand cluster in WA.

Actual results:
Only one node is imported.

Expected results:
All nodes should be imported.

Additional info:
For configuration with created distributed-replicated volume it happens every time but when tried expand only with peer probe without any volume it imported all nodes.
Comment 1 Filip Balák 2018-07-10 09:05:21 UTC 
Created attachment 1457721 [details]
1 - Node is expanded by task
Comment 2 Filip Balák 2018-07-10 09:05:57 UTC 
Created attachment 1457722 [details]
2 - Node remains unmanaged
Comment 3 Shubhendu Tripathi 2018-07-13 07:34:27 UTC 
This is actually not an issue with the expand flow. Rather what has happened in this scenario is that the one of nodes pending for expansion got elected as the provisioner of the cluster and so got removed from the list of expansion nodes.

It would need change in provisioner election logic and have an additional check to make sure nodes pending for expansion should not participate in the election.
Comment 4 Rohan Kanade 2018-07-13 08:07:33 UTC 
Added upstream issue and PR: https://github.com/Tendrl/node-agent/pull/841
Comment 7 Shubhendu Tripathi 2018-07-17 03:03:44 UTC 
Now with the fix the nodes pending for expansion cannot claim the provisioner tag, and so such issue where one node gets left out while expansion of cluster.
Comment 9 Martin Bukatovic 2018-07-17 17:58:17 UTC 
This problem breaks expand RFE BZ 1516417.
Comment 10 Filip Balák 2018-07-23 13:35:05 UTC 
Tested multiple times. Seems ok. --> VERIFIED

Tested with:
tendrl-ansible-1.6.3-5.el7rhgs.noarch
tendrl-api-1.6.3-4.el7rhgs.noarch
tendrl-api-httpd-1.6.3-4.el7rhgs.noarch
tendrl-commons-1.6.3-9.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-7.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-7.el7rhgs.noarch
tendrl-node-agent-1.6.3-9.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-ui-1.6.3-8.el7rhgs.noarch
Comment 12 errata-xmlrpc 2018-09-04 07:08:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616