Bug 1660681

Summary: Heketi does not list storage endpoint of newly added node
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: vinutha <vinug>
Component: heketiAssignee: Raghavendra Talur <rtalur>
Status: CLOSED ERRATA QA Contact: Aditya Ramteke <aramteke>
Severity: high Docs Contact:
Priority: unspecified    
Version: ocs-3.11CC: aramteke, asriram, ccalhoun, hchiramm, jmulligan, knarra, kramdoss, madam, ndevos, nigoyal, pasik, pprakash, puebele, rhs-bugs, rtalur, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: OCS 3.11.z Batch Update 4   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: heketi-9.0.0-4.el7rhgs Doc Type: Bug Fix
Doc Text:
When a node is removed or added to a gluster trusted storage pool using heketi, the existing endpoints do not get updated automatically. With this update, users can execute `heketi-cli volume endpoint patch <volume-id>`command to get a patch file to perform kubectl/oc patch on the endpoints.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-30 12:34:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1573420, 1622458, 1707226    

Description vinutha 2018-12-19 01:11:19 UTC
Description of problem:
On adding a new node by running scaleup.yml playbook newly added node lists in
1. oc get nodes 
2. The node has a gluster pod hosted on it which is in 1/1 Running state 
3. heketi node list and topology info 
4. gluster peers 

BUT the newly added node does not list in end-points and heketi blockhosting volume info 

Version-Release number of selected component (if applicable):
]# rpm  -qa| grep openshift
openshift-ansible-docs-3.11.43-1.git.0.fa69a02.el7.noarch
atomic-openshift-excluder-3.11.43-1.git.0.647ac05.el7.noarch
atomic-openshift-hyperkube-3.11.43-1.git.0.647ac05.el7.x86_64
atomic-openshift-node-3.11.43-1.git.0.647ac05.el7.x86_64
openshift-ansible-playbooks-3.11.43-1.git.0.fa69a02.el7.noarch
openshift-ansible-3.11.43-1.git.0.fa69a02.el7.noarch
atomic-openshift-clients-3.11.43-1.git.0.647ac05.el7.x86_64
atomic-openshift-3.11.43-1.git.0.647ac05.el7.x86_64
openshift-ansible-roles-3.11.43-1.git.0.fa69a02.el7.noarch
atomic-openshift-docker-excluder-3.11.43-1.git.0.647ac05.el7.noarch

# oc rsh glusterfs-storage-xg8q2 rpm -qa| grep gluster 
glusterfs-server-3.12.2-32.el7rhgs.x86_64
gluster-block-0.2.1-30.el7rhgs.x86_64
glusterfs-api-3.12.2-32.el7rhgs.x86_64
glusterfs-cli-3.12.2-32.el7rhgs.x86_64
python2-gluster-3.12.2-32.el7rhgs.x86_64
glusterfs-fuse-3.12.2-32.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-32.el7rhgs.x86_64
glusterfs-libs-3.12.2-32.el7rhgs.x86_64
glusterfs-3.12.2-32.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-32.el7rhgs.x86_64

# oc rsh heketi-storage-1-lgqlm rpm -qa| grep heketi 
heketi-8.0.0-3.el7rhgs.x86_64
heketi-client-8.0.0-3.el7rhgs.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Created a OCS setup with TSP of 4 nodes and having 25 file and 25 block pvcs.

2. To add a new node to the OCP cluster edited the inventory file and ran the scaleup.yml playbook. 

3. The newly added node lists in oc nodes and from heketi. There is also a gluster pod running on the new host 

4. Observed that the ep and heketi block hosting volume info do not list the newly added node . 

Actual results:
Heketi does not list the newly added nodes in storage emdpoint and block hosting volume info. 

Expected results:
Heketi should list the newly added node in storage endpoints and block hosting volume info. 

Additional info:

Comment 5 Niels de Vos 2019-02-06 19:07:58 UTC
Hi Vinutha,

What is the reason the newly added nodes should get listed in the existing endpoints for a PVC/volume? Because there are no bricks on the newly added node for the existing PVC/volume, there is no benefit to have them listed.

Possibly on PVC expansion the endpoints could get updated in case the newly added nodes hosts a brick that was added to the PVC/volume.

Comment 7 Yaniv Kaul 2019-05-02 09:43:57 UTC
Ping?

Comment 13 John Mulligan 2019-06-27 15:41:29 UTC
*** Bug 1567236 has been marked as a duplicate of this bug. ***

Comment 18 Raghavendra Talur 2019-07-24 12:06:36 UTC
Verification steps:

1. create a volume using provisioner in a 3 node trusted storage pool
2. add a node to the trusted storage pool using heketi
3. remove a node(from original 3 nodes) from the trusted storage pool using heketi
4. use the `heketi-cli volume endpoint patch VOLUMEID` command to generate the patch file for endpoint
5. use the patch file to oc patch the endpoint.
6. verify that the endpoint now show new IPs.

Comment 21 John Mulligan 2019-07-25 14:15:53 UTC
*** Bug 1667031 has been marked as a duplicate of this bug. ***

Comment 25 errata-xmlrpc 2019-10-30 12:34:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3255