Description of problem:
Customer's Observation:
"With heketi it is really easy and dynamic to replace a node (due to a failure or decommission), but we have seen that in Openshift the endpoints of heketi do not change and it can become a problem:
[root@ocp]# oc get endpoints --all-namespaces
NAMESPACE NAME ENDPOINTS AGE
...
storage-project heketi-storage-endpoints 10.10.10.10:1,10.10.10.20:1,10.10.10.30:1 54m
testproject glusterfs-dynamic-claim1 10.10.10.10:1,10.10.10.20:1,10.10.10.30:1 49m
Now, with heketi-cli you can remove node 10.10.10.10 and add a new node 10.10.10.40. The endpoints will not change automatically in Openshift and you have to manually edit all the endpoints for every PV to reflect this change in heketi topology.
Thinking in an edge case such as replacing all three gluster nodes, even though everything will work because the glusterfs daemon detect correctly the new volume graph, if the pods (or the entire host) restart, they will not know where the new endpoints are and will never connect to the gluster nodes.
Is there an easy way to replace the endpoints for all pods in case a node replacement has been performed by heketi-cli?
Is it possible to add functionality to update the endpoints of the pods in case the heketi topology has changed?"
Please let me know if I can supply any additional information to assist in this effort.