Bug 1662298

Description of problem:
++++++++ This bug was hit while verifying bug https://bugzilla.redhat.com/show_bug.cgi?id=1639253 ++++++++++++++++++

Version-Release number of selected component (if applicable):


How reproducible:
2X2 on this setup 

Steps to Reproduce:
1.On a fresh setup enabled management encryption following the steps in doc 
https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/index#idm140486996110288

Section 8.3.2

2. As per doc, Section 8.3.2  step 6. edited glusterfs template to include the image version and created a new daemonset 

3. All the gluster pods spinned up are stuck in 0/1 Running state with Liveliness probe failed and Readiness probe failed error message 


4. Heketi pod is in CrashLoopBackOff state with error message 

Error: failed to start container "heketi": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:364: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/origin/openshift.local.volumes/pods/797e4c3e-09bd-11e9-b461-005056a58f75/volumes/kubernetes.io~glusterfs/db\\\" to rootfs \\\"/var/lib/docker/devicemapper/mnt/b25b3753cd677e992e3922e464f8bf69751053dd84b23a9c6468369a3f5f9625/rootfs\\\" at \\\"/var/lib/heketi\\\" caused \\\"stat /var/lib/origin/openshift.local.volumes/pods/797e4c3e-09bd-11e9-b461-005056a58f75/volumes/kubernetes.io~glusterfs/db: transport endpoint is not connected\\\"\""


 

Actual results:
1. Gluster pods failed to come up due to liveliness probe and readiness probe failure 
2. Heketi pod goes to crash loop state with error message 'Transport Endpoint is not connected' 

Expected results:
Gluster pods should be spawned from the new daemonset and be in 1/1 Running state
Heketi pod should not be affected and remain in 1/1 state 


Additional info:
logs will be attached