Bug 1622645 - Block app pods unable to re-mount block pvcs once BHVs have less space - I/O error
Summary: Block app pods unable to re-mount block pvcs once BHVs have less space - I/O ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: cns-3.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: CNS 3.10
Assignee: John Mulligan
QA Contact: Neha Berry
URL: https://github.com/heketi/heketi/pull...
Whiteboard:
Depends On:
Blocks: 1568862
TreeView+ depends on / blocked
 
Reported: 2018-08-27 16:26 UTC by Neha Berry
Modified: 2019-02-11 10:30 UTC (History)
17 users (show)

Fixed In Version: heketi-7.0.0-8.el7rhgs
Doc Type: Bug Fix
Doc Text:
Previously, heketi would allow a block-hosting volume space to be filled completely. Hence, restarting the application pods using block volumes of a completely filled block-hosting volume would fail remounting the block-volume with an I/O error. With this update, heketi reserves 2% of the block hosting volume's raw space and thereby preventing the space from being filled completely and thus remounts are now successful.
Clone Of:
Environment:
Last Closed: 2018-09-12 09:23:51 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2686 0 None None None 2018-09-12 09:24:40 UTC

Description Neha Berry 2018-08-27 16:26:25 UTC
Block app pods unable to re-mount block pvcs once BHVs have less space - I/O error

Description of problem:
++++++++++++++++++++++++
We had a CNS 3.9 setup with 2 block-hosting volumes[BHV's](created for validating the fix of BZ#1612049). The BHV's were consumed completely and no space was left on the volumes. Creation of 1 new pvc(new3) was in pending  state.
The setup was then upgraded to CNS 3.10(all glusterfs pods -3.12.2-17, block-prov and heketi-7.0.0.6). After upgrade, some of the app pods were -respinned and it was seen that none of them were able to re-mount the block pvcs. 


Steps Performed:

1. Created a CNS 3.9 setup.
2. Created block-devices bk1-bk9 to consume all the space of block-hosting vol_ed54bbe8770a4597f8c350f044177bca
3. Created some more block devices to fill up the 2nd BHV - vol_868297e82df2a131b228a20a746813da . Also created app pods(bk2-1, bk3-1, bk4-1) with bind-mounted pvcs bk2, bk3 and bk4
4. Heketi reported free size=100 on vol_ed54bbe8770a4597f8c350f044177bca because of BZ#1584123 even though its space was completely consumed.
5. Hence, tried creating a new pvc(new3) of 5GB but it still tried to use the already full BHV-vol_ed54bbe8770a4597f8c350f044177bca and pvc statyed in pending state.
6. Upgraded the setup to CNS 3.10 and once heketi was upgraded, the pvc request for new3 was automatically completed(a new BHV vol_4805e6aa548e1db5713a46c55ca51f95 was created).
7. Even though reserved size is 2GB for a BHV of 100GB, the two older BHV's had resevred size=0( as they were fully consumed).
8. Created one or two more pvc(after-upfdate) etc and then re-spinned some of the app pods.
9. Multiple Re-spins of the pod bk-2 failed to mount the block device with following error message:

"""""""""""""""""""
  Warning  Failed                 11s (x3 over 28s)  kubelet, dhcp42-223.lab.eng.blr.redhat.com  Error: failed to start container "foo": Error response from daemon: error setting label on mount source '/var/lib/origin/openshift.local.volumes/pods/3b044cde-a8fd-11e8-adda-005056a52b66/volumes/kubernetes.io~iscsi/pvc-6d5d4be1-a7ad-11e8-adda-005056a52b66': SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/3b044cde-a8fd-11e8-adda-005056a52b66/volumes/kubernetes.io~iscsi/pvc-6d5d4be1-a7ad-11e8-adda-005056a52b66 is not allowed: "input/output error

"""""""""""""""""""

10. All the block-devices(bk1-bk8 and new1-new2) carved out of BHV's (with free size in MBs) are NOT getting mounted on any app pod and are reporing one or the other issue.

bknew2-2
+++++++++

"""""""""""""""""""""""""""""""""""""""
Events:
  Type     Reason                 Age   From                                       Message
  ----     ------                 ----  ----                                       -------
  Normal   Scheduled              5m    default-scheduler                          Successfully assigned bknew2-2-kktx5 to dhcp42-84.lab.eng.blr.redhat.com
  Normal   SuccessfulMountVolume  5m    kubelet, dhcp42-84.lab.eng.blr.redhat.com  MountVolume.SetUp succeeded for volume "default-token-d8wh2"
  Warning  FailedMount            4m    kubelet, dhcp42-84.lab.eng.blr.redhat.com  MountVolume.WaitForAttach failed for volume "pvc-d0bde07f-a8ee-11e8-adda-005056a52b66" : failed to mount the volume as "xfs", it already contains mpath_member. Mount error: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 --scope -- mount -t xfs -o defaults /dev/disk/by-path/ip-10.70.41.217:3260-iscsi-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0
Output: Running scope as unit run-101409.scope.
mount: /dev/sdg is already mounted or /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 busy
  Warning  FailedMount  1m (x2 over 3m)  kubelet, dhcp42-84.lab.eng.blr.redhat.com  Unable to mount volumes for pod "bknew2-2-kktx5_glusterfs(73a215ab-a9ed-11e8-adda-005056a52b66)": timeout expired waiting for volumes to attach/mount for pod "glusterfs"/"bknew2-2-kktx5". list of unattached/unmounted volumes=[foo-vol]

""""""""""""""""""""""""""""""""""""""""""""""""""""


"""""""""""""""""""""""""""

bk5-2-z89ph
Events:
  Type     Reason                 Age   From                                        Message
  ----     ------                 ----  ----                                        -------
  Normal   Scheduled              2m    default-scheduler                           Successfully assigned bk5-2-z89ph to dhcp41-217.lab.eng.blr.redhat.com
  Normal   SuccessfulMountVolume  2m    kubelet, dhcp41-217.lab.eng.blr.redhat.com  MountVolume.SetUp succeeded for volume "default-token-d8wh2"
  Warning  FailedMount            10s   kubelet, dhcp41-217.lab.eng.blr.redhat.com  Unable to mount volumes for pod "bk5-2-z89ph_glusterfs(307c2364-a9d8-11e8-adda-005056a52b66)": timeout expired waiting for volumes to attach/mount for pod "glusterfs"/"bk5-2-z89ph". list of unattached/unmounted volumes=[foo-vol]

""""""""""""""""""""""""""""""

11. The block devices from the BHV-vol_4805e6aa548e1db5713a46c55ca51f95 which has enough free space are successfully mounted on app pods.

bkafter-up-1-bp5nq                            1/1       Running             0          5h        10.128.2.43    dhcp41-217.lab.eng.blr.redhat.com
bknew3-1-wkks5                                1/1       Running             0          5h        10.128.2.47    dhcp41-217.lab.eng.blr.redhat.com


I

ssue: Since the block devices are pre-allocated with the size requested during creation, can less space on respective BHV's affect proper mounting of those devices?
==========================================


Space utilization of the bricks
-----------------------------

[root@dhcp42-137 ~]# for i in `oc get pods -o wide| grep glusterfs|cut -d " " -f1` ; do echo $i; echo +++++++++++++++++++++++; oc exec $i -- df -kh|grep heketi; done
glusterfs-storage-59jt5
+++++++++++++++++++++++
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_42d291a6d1f16dfecc8f9c321ec89c31  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_42d291a6d1f16dfecc8f9c321ec89c31
/dev/mapper/vg_49a8f6746a305fd3696f9aa9c1f32d6f-brick_9011b8f81ccc4c9cebf2a74e88ae70d6  100G  100G  180K 100% /var/lib/heketi/mounts/vg_49a8f6746a305fd3696f9aa9c1f32d6f/brick_9011b8f81ccc4c9cebf2a74e88ae70d6
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_a52072845f7ae8361e359a6f907f20cf  120G  120G  931M 100% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_a52072845f7ae8361e359a6f907f20cf
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_c5be1ff405a61f424ed2205d45c5d360  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_c5be1ff405a61f424ed2205d45c5d360
glusterfs-storage-7s5h6
+++++++++++++++++++++++
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_f21f7bd09d1c078c6f3191e811158bb2  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_f21f7bd09d1c078c6f3191e811158bb2
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_4584491f8c346580734d7b1e710b08de  100G  100G  204K 100% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_4584491f8c346580734d7b1e710b08de
/dev/mapper/vg_556c81099602cc71dca84fbe4926c6b0-brick_dad0c9b47e85ac26c7ca09ea1832bedc  120G  120G  931M 100% /var/lib/heketi/mounts/vg_556c81099602cc71dca84fbe4926c6b0/brick_dad0c9b47e85ac26c7ca09ea1832bedc
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_a3f1f0d91f9ab6735b35d605e5ec7d98  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_a3f1f0d91f9ab6735b35d605e5ec7d98
glusterfs-storage-r9b56
+++++++++++++++++++++++
/dev/mapper/vg_4264459401ed48d0939685f2bd5472fb-brick_8724d3d50a9fc4c74f022e9e54b86874  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_4264459401ed48d0939685f2bd5472fb/brick_8724d3d50a9fc4c74f022e9e54b86874
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_7728805e80a10df134c6991134dee9f0  100G  100G  212K 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_7728805e80a10df134c6991134dee9f0
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_2fc66410598a5f3819d686d28fe3882d  120G  120G  931M 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_2fc66410598a5f3819d686d28fe3882d
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_c5a42e8b391e4594c274e9255b65a6a8  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_c5a42e8b391e4594c274e9255b65a6a8
[root@dhcp42-137 ~]# 





Version-Release number of selected component (if applicable):
++++++++++++++++++++++++

Before upgrade
============
Gluster Version before upgrade = -3.8.4-54.8
heketi verison before upgrade - 6.0.0-7.4
gluster-block-0.2.1-14.1.el7rhgs.x86_64

After Upgrade
================

Current gluster verison = 3.12.2-17
gluster-block-0.2.1-25.el7rhgs.x86_64
heketi = heketi-7.0.0-6


How reproducible:
++++++++++++++++++++++++
This testing is performed once


Actual results:
++++++++++++++++++++++++
None of the block dveices which belong to BHV's which are full are now getting mounted on app pods.

Expected results:
++++++++++++++++++++++++
Since the block devices are pre-allocated with space, shouldn't they always get mounted if the block devices itself have enough space.

Comment 24 Anjana KD 2018-09-07 17:38:23 UTC
updated the doc text field. kindly review.

Comment 26 Anjana KD 2018-09-07 18:49:30 UTC
changed it to raw space.

Comment 27 John Mulligan 2018-09-07 19:06:58 UTC
Doc Text looks OK

Comment 29 errata-xmlrpc 2018-09-12 09:23:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686


Note You need to log in before you can comment on or make changes to this bug.