Description of problem:
++++++++++++++++++++++++
We had an OCP 3.10 + OCS 3.10 setup with gluster-bits=3.12.2-15. Gluser-block version = gluster-block-0.2.1-24.el7rhgs.x86_64. The setup has logging pods configured and the metrics pods couldn't come up.

Created around 50 block pvcs in two loops and then attached them to app pods 

Loop #1 :  101..130 and created app pods bk-101 to bk-130. 
Result : All pods were in running state.


Loop #2:   131..150 and created app pods bk-101 to bk-130. 

Result: 
++++++++++++

1. None of the new pods came up and the oc describe pod printed following error message. iscsiadm logins were successful on the initator nodes though.
============================================================

Events:
  Type     Reason                  Age              From                                       Message
  ----     ------                  ----             ----                                       -------
  Normal   Scheduled               7m               default-scheduler                          Successfully assigned bk142-1-fz5gx to dhcp46-65.lab.eng.blr.redhat.com
  Normal   SuccessfulAttachVolume  7m               attachdetach-controller                    AttachVolume.Attach succeeded for volume "pvc-ae117351-a45f-11e8-92b7-005056a52fd4"
  Warning  FailedMount             6m (x3 over 6m)  kubelet, dhcp46-65.lab.eng.blr.redhat.com  MountVolume.MountDevice failed for volume "pvc-ae117351-a45f-11e8-92b7-005056a52fd4" : exit status 1
  Warning  FailedMount             1m (x3 over 5m)  kubelet, dhcp46-65.lab.eng.blr.redhat.com  Unable to mount volumes for pod "bk142-1-fz5gx_glusterfs(cd399d0f-a460-11e8-92b7-005056a52fd4)": timeout expired waiting for volumes to attach or mount for pod "glusterfs"/"bk142-1-fz5gx". list of unmounted volumes=[foo-vol]. list of unattached volumes=[foo-vol default-token-8fbpq]

2. Two existing RUNNING pods started going into CrashLoppBackState with following error message:
========================================================

oc describe pod bk124-1-zmq99
+++++++++++++++++++++++++++++++

Events:
  Type     Reason                  Age   From                                        Message
  ----     ------                  ----  ----                                        -------
  Normal   Scheduled               1h    default-scheduler                           Successfully assigned bk124-1-zmq99 to dhcp46-181.lab.eng.blr.redhat.com
  Normal   SuccessfulAttachVolume  1h    attachdetach-controller                     AttachVolume.Attach succeeded for volume "pvc-11a2ccad-a439-11e8-b3b0-005056a52fd4"
  Normal   Started                 1h    kubelet, dhcp46-181.lab.eng.blr.redhat.com  Started container
  Warning  Unhealthy               16m   kubelet, dhcp46-181.lab.eng.blr.redhat.com  Liveness probe failed: /dev/sdr on /mnt type xfs (rw,seclabel,relatime,attr2,inode64,noquota)
sh: can't create /mnt/random-data.log: Input/output error
  Normal   Pulled   14m (x5 over 1h)   kubelet, dhcp46-181.lab.eng.blr.redhat.com  Container image "cirros" already present on machine
  Normal   Created  14m (x5 over 1h)   kubelet, dhcp46-181.lab.eng.blr.redhat.com  Created container
  Warning  Failed   14m (x4 over 16m)  kubelet, dhcp46-181.lab.eng.blr.redhat.com  Error: failed to start container "foo": Error response from daemon: error setting label on mount source '/var/lib/origin/openshift.local.volumes/pods/dde3d3d9-a458-11e8-92b7-005056a52fd4/volumes/kubernetes.io~iscsi/pvc-11a2ccad-a439-11e8-b3b0-005056a52fd4': SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/dde3d3d9-a458-11e8-92b7-005056a52fd4/volumes/kubernetes.io~iscsi/pvc-11a2ccad-a439-11e8-b3b0-005056a52fd4 is not allowed: "input/output error"
  Warning  BackOff  1m (x57 over 16m)  kubelet, dhcp46-181.lab.eng.blr.redhat.com  Back-off restarting failed container

oc describe pod bk129-1-qzcs5
++++++++++++++++++++++++++++++++++
     Message:      error setting label on mount source '/var/lib/origin/openshift.local.volumes/pods/f0f77021-a458-11e8-92b7-005056a52fd4/volumes/kubernetes.io~iscsi/pvc-213178f1-a439-11e8-b3b0-005056a52fd4': SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/f0f77021-a458-11e8-92b7-005056a52fd4/volumes/kubernetes.io~iscsi/pvc-213178f1-a439-11e8-b3b0-005056a52fd4 is not allowed: "input/output error"
      Exit Code:    128



Some info about the setup: 
============================

1. it was seen that the brick for heketidbstorage was NOT ONLINE for the gluster pod -10.70.46.150. 
2. Also,  on 10.70.46.150, the 2 block-hosting vols had 2 separate PIDS and the brick for vol_9f93ae4c845f3910f5d1558cc5ae9f0a was NOT ONLINE.
(We shall be raising a separate bug for the above two issues)




Version-Release number of selected component (if applicable):
++++++++++++++++++++++++

[root@dhcp46-137 ~]# oc version
oc v3.10.14
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dhcp46-137.lab.eng.blr.redhat.com:8443
openshift v3.10.14
kubernetes v1.10.0+b81c8f8
[root@dhcp46-137 ~]# 


Gluster 3.4.0
==============

[root@dhcp46-137 ~]# oc rsh glusterfs-storage-q22cl rpm -qa|grep gluster
glusterfs-client-xlators-3.12.2-15.el7rhgs.x86_64
glusterfs-cli-3.12.2-15.el7rhgs.x86_64
python2-gluster-3.12.2-15.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-15.el7rhgs.x86_64
glusterfs-libs-3.12.2-15.el7rhgs.x86_64
glusterfs-3.12.2-15.el7rhgs.x86_64
glusterfs-api-3.12.2-15.el7rhgs.x86_64
glusterfs-fuse-3.12.2-15.el7rhgs.x86_64
glusterfs-server-3.12.2-15.el7rhgs.x86_64
gluster-block-0.2.1-24.el7rhgs.x86_64
[root@dhcp46-137 ~]# 


[root@dhcp46-137 ~]# oc rsh heketi-storage-1-px7jd rpm -qa|grep heketi
python-heketi-7.0.0-6.el7rhgs.x86_64
heketi-7.0.0-6.el7rhgs.x86_64
heketi-client-7.0.0-6.el7rhgs.x86_64
[root@dhcp46-137 ~]# 


gluster client version
=========================
[root@dhcp46-65 ~]# rpm -qa|grep gluster
glusterfs-libs-3.12.2-15.el7.x86_64
glusterfs-3.12.2-15.el7.x86_64
glusterfs-fuse-3.12.2-15.el7.x86_64
glusterfs-client-xlators-3.12.2-15.el7.x86_64
[root@dhcp46-65 ~]# 




How reproducible:
++++++++++++++++++++++++
The issue was seen on one setup. The setup is kept in the same condition.

Steps to Reproduce:
++++++++++++++++++++++++
1. Create an OCP +OCS 3.10 setup.
2. Upgrade the docker version to 1.13.1.74 and also update the gluster client packages. The pods will be restarted as docker is upgraded.
3. Once setup is up, create block pvcs and then bound them to app pods.
4. Check the pod status and the gluster v status.


Actual results:
++++++++++++++++++++++++
The pods should 

Expected results:
++++++++++++++++++++++++
The new pods should have got created successfully and old ones should keep running.