Bug 1428316

Summary:	cns-deploy: gluster cluster cannot be created in two different namespaces using different sets of nodes
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Shekhar Berry <shberry>
Component:	CNS-deployment	Assignee:	Humble Chirammal <hchiramm>
Status:	CLOSED DUPLICATE	QA Contact:	Prasanth <pprakash>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.2	CC:	akhakhar, annair, ekuric, hchiramm, jarrpa, jeder, madam, mliyazud, mzywusko, pprakash, rreddy, rtalur, shberry
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:	aos-scalability-35
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-27 09:59:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Shekhar Berry 2017-03-02 09:59:17 UTC

Description of problem:

IF a gluster cluster deployed using cns-deploy is running in one namespace using 3 storage nodes, it's not possible to create another gluster cluster in another namespace using different set of 3 storage nodes.

The reasoning behind this is that, when first gluster cluster is created using cns-deploy, it labels those 3 nodes as storagenode=glusterfs and entry is made to daemonset. This labeling is global and in 2nd instance when we give 3 different set of nodes in topology file, it doesn't reads from it and picks up OCP nodes based on node label (storagenode=glusterfs ) and not based on what nodes are in topology.json

Version-Release number of selected component (if applicable):

cns-deploy-3.1.0-14.el7rhgs.x86_64
heketi-3.1.0-14.el7rhgs.x86_64
heketi-client-3.1.0-14.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create topology file and deploy cns using the same in a namespace.
2. Create a new namespace
3. Create a new topology file and try to create another gluster cluster in a new namespace

Actual results:

Waiting for GlusterFS pods to start ... Timed out waiting for pods matching 'glusterfs-node=pod'.
No resources found
Error from server (NotFound): deploymentconfig "heketi" not found
Error from server (NotFound): services "heketi" not found
Error from server (NotFound): routes "heketi" not found
Error from server (NotFound): services "heketi-storage-endpoints" not found
serviceaccount "heketi-service-account" deleted
template "deploy-heketi" deleted
template "heketi" deleted
node "ip-172-31-41-103.us-west-2.compute.internal" labeled
node "ip-172-31-43-103.us-west-2.compute.internal" labeled
node "ip-172-31-56-4.us-west-2.compute.internal" labeled
daemonset "glusterfs" deleted
template "glusterfs" deleted

If you see node labels after executing this, it still shows only storagenode=glusterfs on those nodes which was used to create CNS in first instance. See oc get nodes --show-labels output for that.

Expected results:
CNS-deploy should progress smoothly and we have two different gluster cluster on two different name spaces.

Comment 2 Humble Chirammal 2017-03-02 10:17:27 UTC

If you are planning for different cluster in different namespace, you can try different storage node labelling. To accommodate/achieve this we have to edit a new label in the template and trycns-deploy with second namespace.

Comment 3 Mohamed Ashiq 2017-03-02 13:12:28 UTC

Yeah As for longer plan to support this,

As the node label are global, DaemonSet being Namespace specific which deploys based on node labels set. There is no validation to stop you from starting the same application on different namespace with same label. 

From cns-deploy point, We can add an option to do configurable label. So that the deployment of cns on another namespace do not clash on the node labels used by the first Deployment. 

We can have a flag in cns-deploy like --node-label with just like --namespace. If not specified we can fall to default value.

Humble and I discussed about this, Please share your thoughts Jose.

Comment 7 Jose A. Rivera 2017-03-14 14:05:13 UTC

The proposed solution seems all right. Given that heketi limits its pod search by namespace, I don't see a conflict.

I do have a question about the initial request, however: Why do we need to run cns-deploy more than once? Couldn't the second cluster just be managed by the same heketi instance?

Comment 8 Shekhar Berry 2017-03-15 08:18:14 UTC

I am trying to create CNS cluster in a new namespace and not in the existing namespace. Hence I was trying it by running cns-deploy again.

What are the steps to create a new CNS cluster in a new namespace using the existing heketi instance?

Comment 9 Humble Chirammal 2017-04-27 09:59:00 UTC

Thi is fixed in CNS 3.5 release. Precisely with an async release.

Comment 10 Humble Chirammal 2017-04-27 09:59:37 UTC


*** This bug has been marked as a duplicate of bug 1444460 ***

Comment 11 Red Hat Bugzilla 2023-09-14 03:54:35 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days