Bug 1622645

Block app pods unable to re-mount block pvcs once BHVs have less space - I/O error

Description of problem:
++++++++++++++++++++++++
We had a CNS 3.9 setup with 2 block-hosting volumes[BHV's](created for validating the fix of BZ#1612049). The BHV's were consumed completely and no space was left on the volumes. Creation of 1 new pvc(new3) was in pending  state.
The setup was then upgraded to CNS 3.10(all glusterfs pods -3.12.2-17, block-prov and heketi-7.0.0.6). After upgrade, some of the app pods were -respinned and it was seen that none of them were able to re-mount the block pvcs. 


Steps Performed:

1. Created a CNS 3.9 setup.
2. Created block-devices bk1-bk9 to consume all the space of block-hosting vol_ed54bbe8770a4597f8c350f044177bca
3. Created some more block devices to fill up the 2nd BHV - vol_868297e82df2a131b228a20a746813da . Also created app pods(bk2-1, bk3-1, bk4-1) with bind-mounted pvcs bk2, bk3 and bk4
4. Heketi reported free size=100 on vol_ed54bbe8770a4597f8c350f044177bca because of BZ#1584123 even though its space was completely consumed.
5. Hence, tried creating a new pvc(new3) of 5GB but it still tried to use the already full BHV-vol_ed54bbe8770a4597f8c350f044177bca and pvc statyed in pending state.
6. Upgraded the setup to CNS 3.10 and once heketi was upgraded, the pvc request for new3 was automatically completed(a new BHV vol_4805e6aa548e1db5713a46c55ca51f95 was created).
7. Even though reserved size is 2GB for a BHV of 100GB, the two older BHV's had resevred size=0( as they were fully consumed).
8. Created one or two more pvc(after-upfdate) etc and then re-spinned some of the app pods.
9. Multiple Re-spins of the pod bk-2 failed to mount the block device with following error message:

"""""""""""""""""""
  Warning  Failed                 11s (x3 over 28s)  kubelet, dhcp42-223.lab.eng.blr.redhat.com  Error: failed to start container "foo": Error response from daemon: error setting label on mount source '/var/lib/origin/openshift.local.volumes/pods/3b044cde-a8fd-11e8-adda-005056a52b66/volumes/kubernetes.io~iscsi/pvc-6d5d4be1-a7ad-11e8-adda-005056a52b66': SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/3b044cde-a8fd-11e8-adda-005056a52b66/volumes/kubernetes.io~iscsi/pvc-6d5d4be1-a7ad-11e8-adda-005056a52b66 is not allowed: "input/output error

"""""""""""""""""""

10. All the block-devices(bk1-bk8 and new1-new2) carved out of BHV's (with free size in MBs) are NOT getting mounted on any app pod and are reporing one or the other issue.

bknew2-2
+++++++++

"""""""""""""""""""""""""""""""""""""""
Events:
  Type     Reason                 Age   From                                       Message
  ----     ------                 ----  ----                                       -------
  Normal   Scheduled              5m    default-scheduler                          Successfully assigned bknew2-2-kktx5 to dhcp42-84.lab.eng.blr.redhat.com
  Normal   SuccessfulMountVolume  5m    kubelet, dhcp42-84.lab.eng.blr.redhat.com  MountVolume.SetUp succeeded for volume "default-token-d8wh2"
  Warning  FailedMount            4m    kubelet, dhcp42-84.lab.eng.blr.redhat.com  MountVolume.WaitForAttach failed for volume "pvc-d0bde07f-a8ee-11e8-adda-005056a52b66" : failed to mount the volume as "xfs", it already contains mpath_member. Mount error: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 --scope -- mount -t xfs -o defaults /dev/disk/by-path/ip-10.70.41.217:3260-iscsi-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0
Output: Running scope as unit run-101409.scope.
mount: /dev/sdg is already mounted or /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 busy
  Warning  FailedMount  1m (x2 over 3m)  kubelet, dhcp42-84.lab.eng.blr.redhat.com  Unable to mount volumes for pod "bknew2-2-kktx5_glusterfs(73a215ab-a9ed-11e8-adda-005056a52b66)": timeout expired waiting for volumes to attach/mount for pod "glusterfs"/"bknew2-2-kktx5". list of unattached/unmounted volumes=[foo-vol]

""""""""""""""""""""""""""""""""""""""""""""""""""""


"""""""""""""""""""""""""""

bk5-2-z89ph
Events:
  Type     Reason                 Age   From                                        Message
  ----     ------                 ----  ----                                        -------
  Normal   Scheduled              2m    default-scheduler                           Successfully assigned bk5-2-z89ph to dhcp41-217.lab.eng.blr.redhat.com
  Normal   SuccessfulMountVolume  2m    kubelet, dhcp41-217.lab.eng.blr.redhat.com  MountVolume.SetUp succeeded for volume "default-token-d8wh2"
  Warning  FailedMount            10s   kubelet, dhcp41-217.lab.eng.blr.redhat.com  Unable to mount volumes for pod "bk5-2-z89ph_glusterfs(307c2364-a9d8-11e8-adda-005056a52b66)": timeout expired waiting for volumes to attach/mount for pod "glusterfs"/"bk5-2-z89ph". list of unattached/unmounted volumes=[foo-vol]

""""""""""""""""""""""""""""""

11. The block devices from the BHV-vol_4805e6aa548e1db5713a46c55ca51f95 which has enough free space are successfully mounted on app pods.

bkafter-up-1-bp5nq                            1/1       Running             0          5h        10.128.2.43    dhcp41-217.lab.eng.blr.redhat.com
bknew3-1-wkks5                                1/1       Running             0          5h        10.128.2.47    dhcp41-217.lab.eng.blr.redhat.com


I

ssue: Since the block devices are pre-allocated with the size requested during creation, can less space on respective BHV's affect proper mounting of those devices?
==========================================


Space utilization of the bricks
-----------------------------

[root@dhcp42-137 ~]# for i in `oc get pods -o wide| grep glusterfs|cut -d " " -f1` ; do echo $i; echo +++++++++++++++++++++++; oc exec $i -- df -kh|grep heketi; done
glusterfs-storage-59jt5
+++++++++++++++++++++++
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_42d291a6d1f16dfecc8f9c321ec89c31  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_42d291a6d1f16dfecc8f9c321ec89c31
/dev/mapper/vg_49a8f6746a305fd3696f9aa9c1f32d6f-brick_9011b8f81ccc4c9cebf2a74e88ae70d6  100G  100G  180K 100% /var/lib/heketi/mounts/vg_49a8f6746a305fd3696f9aa9c1f32d6f/brick_9011b8f81ccc4c9cebf2a74e88ae70d6
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_a52072845f7ae8361e359a6f907f20cf  120G  120G  931M 100% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_a52072845f7ae8361e359a6f907f20cf
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_c5be1ff405a61f424ed2205d45c5d360  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_c5be1ff405a61f424ed2205d45c5d360
glusterfs-storage-7s5h6
+++++++++++++++++++++++
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_f21f7bd09d1c078c6f3191e811158bb2  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_f21f7bd09d1c078c6f3191e811158bb2
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_4584491f8c346580734d7b1e710b08de  100G  100G  204K 100% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_4584491f8c346580734d7b1e710b08de
/dev/mapper/vg_556c81099602cc71dca84fbe4926c6b0-brick_dad0c9b47e85ac26c7ca09ea1832bedc  120G  120G  931M 100% /var/lib/heketi/mounts/vg_556c81099602cc71dca84fbe4926c6b0/brick_dad0c9b47e85ac26c7ca09ea1832bedc
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_a3f1f0d91f9ab6735b35d605e5ec7d98  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_a3f1f0d91f9ab6735b35d605e5ec7d98
glusterfs-storage-r9b56
+++++++++++++++++++++++
/dev/mapper/vg_4264459401ed48d0939685f2bd5472fb-brick_8724d3d50a9fc4c74f022e9e54b86874  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_4264459401ed48d0939685f2bd5472fb/brick_8724d3d50a9fc4c74f022e9e54b86874
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_7728805e80a10df134c6991134dee9f0  100G  100G  212K 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_7728805e80a10df134c6991134dee9f0
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_2fc66410598a5f3819d686d28fe3882d  120G  120G  931M 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_2fc66410598a5f3819d686d28fe3882d
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_c5a42e8b391e4594c274e9255b65a6a8  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_c5a42e8b391e4594c274e9255b65a6a8
[root@dhcp42-137 ~]# 





Version-Release number of selected component (if applicable):
++++++++++++++++++++++++

Before upgrade
============
Gluster Version before upgrade = -3.8.4-54.8
heketi verison before upgrade - 6.0.0-7.4
gluster-block-0.2.1-14.1.el7rhgs.x86_64

After Upgrade
================

Current gluster verison = 3.12.2-17
gluster-block-0.2.1-25.el7rhgs.x86_64
heketi = heketi-7.0.0-6


How reproducible:
++++++++++++++++++++++++
This testing is performed once


Actual results:
++++++++++++++++++++++++
None of the block dveices which belong to BHV's which are full are now getting mounted on app pods.

Expected results:
++++++++++++++++++++++++
Since the block devices are pre-allocated with space, shouldn't they always get mounted if the block devices itself have enough space.