Bug 1400965

Summary:	Prevent the RHGS docker image in a production setup from getting replaced with any new image with "latest" TAG pushed to the public Registry.
Product:	Red Hat Gluster Storage	Reporter:	Prasanth <pprakash>
Component:	doc-Container_Native_Storage_with_OpenShift	Assignee:	Divya <divya>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Prasanth <pprakash>
Severity:	high	Docs Contact:
Priority:	high
Version:	cns-3.4	CC:	akhakhar, annair, asriram, divya, hchiramm, jrivera, madam, mliyazud, mzywusko, pprakash, rcyriac, rhs-bugs, rreddy, rtalur, storage-doc
Target Milestone:	---
Target Release:	CNS 3.4
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1411227 (view as bug list)		Environment:
Last Closed:	2017-01-23 07:21:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1385254

Description Prasanth 2016-12-02 12:49:48 UTC

Description of problem:

We should prevent the RHGS/Volmanager docker image in a production setup from getting replaced with any new image with "latest" TAG pushed at a later stage to the public registry: registry.access.redhat.com.

If there is a new image tagged as "latest" is pushed to the Registry, there is a high chance that the RHGS pods running in ANY existing customer deployment setups will try to pull the "latest" image on a node reboot or a docker restart, etc. Since pulling the "latest" image will automatically happen even though there is no specific requirement at that moment, there is a high chance that this might even break the production setup if:
* the gluster pods comes back online with the "latest" image which has some major code changes/bug fixes, RHEL base image updates or whatever that may lead to the breaking of the existing setup. 
* the gluster pods fails to come back online due to any reason
* the gluster pods comes back online with the "latest" image but it ends up having different "IMAGE ID" in different nodes as the rebooted nodes will now end up having the "latest" image and the end-result is UNKNOWN


Ideally, the image upgrade should have happened ONLY when the user was actually doing a proper major/minor version upgrade by following the official documented steps. The point to note is that, here it happened even otherwise outside of a normal M-window. 

So I believe, this should somehow be restricted from happening to avoid any post deployment issues. Either via hard coding the specific "TAG" in the respective template files or by some other better ways.


Additional info:

Comment 1 Humble Chirammal 2016-12-02 12:57:40 UTC

Need to revisit the template image policy for our templates. Lets discuss and have a consensus on this.

Comment 2 Humble Chirammal 2016-12-02 13:02:40 UTC

(In reply to Humble Chirammal from comment #1)
> Need to revisit the template image policy for our templates. Lets discuss
> and have a consensus on this.

Just to be clear, `Tag + image policy` has to be adjusted based on our plan to support upgrades and such.

Comment 3 Mohamed Ashiq 2016-12-02 13:23:15 UTC

(In reply to Humble Chirammal from comment #2)
> (In reply to Humble Chirammal from comment #1)
> > Need to revisit the template image policy for our templates. Lets discuss
> > and have a consensus on this.
> 
> Just to be clear, `Tag + image policy` has to be adjusted based on our plan
> to support upgrades and such.

Yes. Tags are really important to maintain in templates based releases of container and cns itself.

Comment 4 Michael Adam 2016-12-12 17:36:37 UTC

Decision was made to not put the tags into the templates in the heketi RPMs, because this creates a circular dependency.

Instead keep the un-tagged images in the RPM and document to nail it down with the tags manually.

Comment 5 Humble Chirammal 2016-12-13 12:07:40 UTC

The container images ( vol manager and glusterfs) are tagged ( for ex: 3-1-3-19) from now on. If admin want to deploy a particular version of the heketi and glusterfs template, they need to edit the template files as shown below:


from

 "image": "<IMAGE NAME>:latest",

to

 "image": "<IMAGE NAME>:3-1-3-19",

where `latest` and `3-1-3-19` are tags of the container image.

Comment 6 Mohamed Ashiq 2016-12-13 12:13:03 UTC

(In reply to Humble Chirammal from comment #5)
> The container images ( vol manager and glusterfs) are tagged ( for ex:
> 3-1-3-19) from now on. If admin want to deploy a particular version of the
> heketi and glusterfs template, they need to edit the template files as shown
> below:
> 
> 
> from
> 
>  "image": "<IMAGE NAME>:latest",
> 
> to
> 
>  "image": "<IMAGE NAME>:3-1-3-19",
> 
> where `latest` and `3-1-3-19` are tags of the container image.

from

   "image": "<IMAGE_NAME>",

to

   "image": "<IMAGE_NAME>:3.1.3-19"

image will be without any tag which internally means latest,just add the tag specified in above.

Comment 9 Divya 2016-12-15 09:19:09 UTC

Ashiq,

According to my discussion with Anjana, a smalll note needs to be added in the deployment section to address this bug.

Could you please share the information that needs to be added?

Thanks,
Divya

Comment 10 Mohamed Ashiq 2016-12-20 07:22:18 UTC

Divya,

https://bugzilla.redhat.com/show_bug.cgi?id=1400965#c6

We have to add the above comment more cleaner in the place where we have  
# oc create -f /usr/share/heketi/templates

Thanks,
Ashiq

Comment 11 Divya 2016-12-23 10:33:35 UTC

Adding a note in the deployment section: http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Manual_Deployment

Comment 12 Prasanth 2017-01-09 13:57:20 UTC

(In reply to Divya from comment #11)
> Adding a note in the deployment section:
> http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-
> Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-
> branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.
> html#chap-Documentation-
> Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-
> Manual_Deployment


I think the above changes are no longer required as we are fixing it in code itself [1]. Please confirm the same once again with devel and make the required changes.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1411227

Comment 14 Humble Chirammal 2017-01-10 09:29:31 UTC

please revert the changes in c#11 and close this bug.

Comment 15 Divya 2017-01-10 11:44:05 UTC

Deleted the note in the deployment section.

Link to the latest doc: http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Manual_Deployment

Comment 16 Prasanth 2017-01-10 12:10:49 UTC

(In reply to Mohamed Ashiq from comment #10)
> Divya,
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1400965#c6
> 
> We have to add the above comment more cleaner in the place where we have  
> # oc create -f /usr/share/heketi/templates
> 
> Thanks,

Ashiq,

I don't think the above step is no longer valid as we have added two extra sample files as well to the same path. In that case, shouldn't we:

* separate it out like the following:

# oc create -f /usr/share/heketi/templates/deploy-heketi-template.yaml
# oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
# oc create -f /usr/share/heketi/templates/heketi-service-account.yaml
# oc create -f /usr/share/heketi/templates/heketi-template.yaml


* or consolidate it out in one single line without taking the sample files.

Divya, once you get a confirmation on the above, please make the required changes accordingly.

Comment 17 Mohamed Ashiq 2017-01-10 12:34:00 UTC

(In reply to Prasanth from comment #16)
> (In reply to Mohamed Ashiq from comment #10)
> > Divya,
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1400965#c6
> > 
> > We have to add the above comment more cleaner in the place where we have  
> > # oc create -f /usr/share/heketi/templates
> > 
> > Thanks,
> 
> Ashiq,
> 
> I don't think the above step is no longer valid as we have added two extra
> sample files as well to the same path. In that case, shouldn't we:
> 
> * separate it out like the following:
> 
> # oc create -f /usr/share/heketi/templates/deploy-heketi-template.yaml
> # oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
> # oc create -f /usr/share/heketi/templates/heketi-service-account.yaml
> # oc create -f /usr/share/heketi/templates/heketi-template.yaml
> 
> 
> * or consolidate it out in one single line without taking the sample files.
> 
> Divya, once you get a confirmation on the above, please make the required
> changes accordingly.

Prasanth,

Agreed.

Divya,

Please add the separate commands instead of "# oc create -f /usr/share/heketi/templates" Output is available consolidated for the above command please take from it with respect to the commands below.

# oc create -f /usr/share/heketi/templates/deploy-heketi-template.yaml
# oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
# oc create -f /usr/share/heketi/templates/heketi-service-account.yaml
# oc create -f /usr/share/heketi/templates/heketi-template.yaml

--Ashiq

Comment 18 Divya 2017-01-10 12:40:28 UTC

Thanks Prasanth, Ashiq.

I have updated the doc based on Comments 16 and 17.

http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#idm139833904759088

Comment 19 Prasanth 2017-01-10 13:58:15 UTC

(In reply to Divya from comment #18)
> Thanks Prasanth, Ashiq.
> 
> I have updated the doc based on Comments 16 and 17.
> 
> http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-
> Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-
> branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.
> html#idm139833904759088


Divya, did you missed adding the following for the verification of serviceaccount created as part of the above step?

#####
# oc get serviceaccount heketi-service-account
NAME                     SECRETS   AGE
heketi-service-account   2         7d
#####

If so, you might have to include that as well.

Comment 20 Divya 2017-01-11 06:13:32 UTC

(In reply to Prasanth from comment #19)
> (In reply to Divya from comment #18)
> > Thanks Prasanth, Ashiq.
> > 
> > I have updated the doc based on Comments 16 and 17.
> > 
> > http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-
> > Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-
> > branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.
> > html#idm139833904759088
> 
> 
> Divya, did you missed adding the following for the verification of
> serviceaccount created as part of the above step?
> 
> #####
> # oc get serviceaccount heketi-service-account
> NAME                     SECRETS   AGE
> heketi-service-account   2         7d
> #####
> 
> If so, you might have to include that as well.

Prasanth,

According to your suggestion in Comment 16 or Ashiq's Comment 17, verification of serviceaccount was not listed. Hence, did not include it in the doc.

Ashiq,

Could you please share your thoughts?

Thanks,
Divya

Comment 21 Mohamed Ashiq 2017-01-11 07:07:09 UTC

(In reply to Divya from comment #20)
> (In reply to Prasanth from comment #19)
> > (In reply to Divya from comment #18)
> > > Thanks Prasanth, Ashiq.
> > > 
> > > I have updated the doc based on Comments 16 and 17.
> > > 
> > > http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-
> > > Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-
> > > branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.
> > > html#idm139833904759088
> > 
> > 
> > Divya, did you missed adding the following for the verification of
> > serviceaccount created as part of the above step?
> > 
> > #####
> > # oc get serviceaccount heketi-service-account
> > NAME                     SECRETS   AGE
> > heketi-service-account   2         7d
> > #####
> > 
> > If so, you might have to include that as well.
> 
> Prasanth,
> 
> According to your suggestion in Comment 16 or Ashiq's Comment 17,
> verification of serviceaccount was not listed. Hence, did not include it in
> the doc.
> 
> Ashiq,
> 
> Could you please share your thoughts?
> 
> Thanks,
> Divya

Divya,

I agree with prasanth's comment #19, please add heketi-service-account verification step in the doc.

--Ashiq

Comment 22 Divya 2017-01-11 07:23:15 UTC

I have added serviceaccount verification step in the doc.


Link to the doc: http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-Manual_Deployment

Comment 23 Prasanth 2017-01-11 07:58:30 UTC

(In reply to Divya from comment #20)

> Prasanth,
> 
> According to your suggestion in Comment 16 or Ashiq's Comment 17,
> verification of serviceaccount was not listed. Hence, did not include it in
> the doc.

Divya, if you are giving the validation for the other 3 in the doc (# oc get templates in Step3), it is expected to give it for the remaining one as all to maintain consistency in the doc.

Comment 24 Prasanth 2017-01-11 08:07:41 UTC

(In reply to Divya from comment #22)
> I have added serviceaccount verification step in the doc.
> 
> 
> Link to the doc:
> http://ccs-jenkins.gsslab.brq.redhat.com:8080/job/doc-
> Red_Hat_Gluster_Storage-3.4-Container_Native_Storage_with_OpenShift_Platform-
> branch-master/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.
> html#chap-Documentation-
> Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-
> Manual_Deployment


###############
4. Execute the following command to verify that the serviceaccount is created:

# oc get serviceaccount heketi-service-account

For example:

# oc get serviceaccount heketi-service-account
NAME                     SECRETS   AGE
heketi-service-account   2         7d
###############

Verified