Bug 1679050

Document URL: 

In 

https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html/deployment_guide/chap-documentation-red_hat_gluster_storage_container_native_with_openshift_platform-upgrade-gluster_pods

we say "If the setup also has registry deployed, the templates have to be modified accordingly to upgrade the pods in the registry namespace. The same steps for gluster pod upgrade can be followed for glusterfs registry pod upgrade by making necessary changes to the parameters accordingly. 

as well as 

"Note
If the setup has registry configured using glusterfs, then the glusterfs registry pods should also be upgraded before setting cluster.op-version"

But that is by no means sufficient to be considered an appropriate guide to what is really needed and should be much better explained from my point of view 

Section Number and Name: 
- https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html/deployment_guide/chap-documentation-red_hat_gluster_storage_container_native_with_openshift_platform-upgrade-gluster_pods

-- 6.4. Upgrading the Red Hat Gluster Storage Pods
--- 2. Execute the following steps to enable server.tcp-user-timeout on all volumes.
--- 7.  If CNS 3.9 is deployed via Ansible, then execute the following command to edit the old glusterfs template. 
--- 9. Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:
--- 15. Execute the following command to verify if you have upgraded the pod to the latest version:




Describe the issue: 
- We just refer that with registry storage the same steps should be repeated and point to the labels and both is either not sufficient or incorrect.


looking at the node labels we have (mind difference in storage-/registry-): 

# oc get nodes --show-labels |grep glust
inf155.example.com   Ready     infra     1d        v1.9.1+a0ce1bc657   [...],glusterfs=registry-host,[...]
inf156.example.com   Ready     infra     1d        v1.9.1+a0ce1bc657   [...],glusterfs=registry-host,[...]
inf157.example.com   Ready     infra     1d        v1.9.1+a0ce1bc657   [...],glusterfs=registry-host,[...]
inf158.example.com   Ready     compute   1d        v1.9.1+a0ce1bc657   [...],glusterfs=storage-host,[...]
inf159.example.com   Ready     compute   1d        v1.9.1+a0ce1bc657   [...],glusterfs=storage-host,[...]
inf160.example.com   Ready     compute   1d        v1.9.1+a0ce1bc657   [...],glusterfs=storage-host,[...]

# oc get po --show-labels -n infra-storage
NAME                                           READY     STATUS    RESTARTS   AGE       LABELS
glusterblock-registry-provisioner-dc-1-jwwqk   1/1       Running   0          4m        deployment=glusterblock-registry-provisioner-dc-1,deploymentconfig=glusterblock-registry-provisioner-dc,glusterfs=block-registry-provisioner-pod
glusterfs-registry-k4zbn                       1/1       Running   0          37m       [...],glusterfs=registry-pod,[...]
glusterfs-registry-zfv6l                       1/1       Running   0          52m       [...],glusterfs=registry-pod,[...]
glusterfs-registry-zs8h2                       1/1       Running   0          1h        [...],glusterfs=registry-pod,[...]
heketi-registry-1-22mb2                        1/1       Running   0          3m        deployment=heketi-registry-1,deploymentconfig=heketi-registry,glusterfs=heketi-registry-pod,heketi=registry-pod
# 
# oc get po --show-labels -n app-storage
NAME                                          READY     STATUS    RESTARTS   AGE       LABELS
glusterblock-storage-provisioner-dc-1-q5pxf   1/1       Running   0          15h       deployment=glusterblock-storage-provisioner-dc-1,deploymentconfig=glusterblock-storage-provisioner-dc,glusterfs=block-storage-provisioner-pod
glusterfs-storage-9x9nt                       1/1       Running   0          15h       [...],glusterfs=storage-pod,[...]
glusterfs-storage-czjrt                       1/1       Running   0          15h       [...],glusterfs=storage-pod,[...]
glusterfs-storage-kf95h                       1/1       Running   0          15h       [...],glusterfs=storage-pod,[...]
heketi-storage-1-pj44k                        1/1       Running   0          1h        deployment=heketi-storage-1,deploymentconfig=heketi-storage,glusterfs=heketi-storage-pod,heketi=storage-pod
# 

this has been set up with latest 3.9 installer (openshift-ansible-3.9.65-1.git.0.a14009a.el7.noarch) and the following gluster settings in the inventory:

openshift_hosted_registry_storage_kind=glusterfs
openshift_metrics_cassanda_pvc_storage_class_name="glusterfs-registry-block" 
openshift_logging_es_pvc_storage_class_name="glusterfs-registry-block" 
openshift_storage_glusterfs_registry_namespace=infra-storage  
openshift_storage_glusterfs_namespace=app-storage
openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7
openshift_storage_glusterfs_heketi_version=v3.9
openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs-server-rhel7
openshift_storage_glusterfs_version=v3.9
openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7
openshift_storage_glusterfs_block_version=v3.9
openshift_storage_glusterfs_s3_image=registry.access.redhat.com/rhgs3/rhgs-gluster-s3-server-rhel7
openshift_storage_glusterfs_s3_version=v3.9
openshift_storage_glusterfs_registry_block_deploy=true
openshift_storage_glusterfs_registry_block_host_vol_size=100
openshift_storage_glusterfs_registry_block_storageclass=true
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_block_storageclass=true
openshift_storage_glusterfs_storageclass_default=true
openshift_prometheus_storage_class=glusterfs-registry-block
#gluster internal
[glusterfs_registry]
inf155.example.com glusterfs_devices='[ "/dev/vdc" ]'
inf156.example.com glusterfs_devices='[ "/dev/vdc" ]'
inf157.example.com glusterfs_devices='[ "/dev/vdc" ]'
#gluster apps
[glusterfs]
inf158.example.com glusterfs_devices='[ "/dev/vdc" ]'
inf159.example.com glusterfs_devices='[ "/dev/vdc" ]'
inf160.example.com glusterfs_devices='[ "/dev/vdc" ]'


so having app storage in project app-storage
and internal/registry storage in infra-storage 


looking at the templates for app-storage we see the following 

# oc -n app-storage get templates -o yaml |grep -B1 storage
[...]
--
    name: CLUSTER_NAME
    value: storage
[..]
--
    name: NODE_LABELS
    value: '{ "glusterfs": "storage-host" }'
--
    name: CLUSTER_NAME
    value: storage
--
    name: CLUSTER_NAME
    value: storage

which is pretty fine for app-storage, HOWEVER registry storage has the very same in it which leads to duplicate pods or even all pods down if CLUSTER_NAME and NODE_LABELS are not changed accordingly :

# oc get template heketi -o yaml |grep -B1 registry 
  name: CLUSTER_NAME
  value: registry <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

# oc get template glusterfs -o yaml |grep -B1 registry
  name: NODE_LABELS
  value: '{ "glusterfs": "registry-host" }'  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< see node label above ! (glusterfs=registry-host)
--
  name: CLUSTER_NAME
  value: registry  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

# oc get template glusterblock-provisioner -o yaml |grep -B1 registry
  name: CLUSTER_NAME
  value: registry  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<




Suggestions for improvement: 

I think we need to much better explain the link between pod-labels, node-labels and what needs to be based on those changed in the templates and how, also a bit better higlighting when chancing namespaces in the templates would be great 


Additional information: