Block app pods unable to re-mount block pvcs once BHVs have less space - I/O error Description of problem: ++++++++++++++++++++++++ We had a CNS 3.9 setup with 2 block-hosting volumes[BHV's](created for validating the fix of BZ#1612049). The BHV's were consumed completely and no space was left on the volumes. Creation of 1 new pvc(new3) was in pending state. The setup was then upgraded to CNS 3.10(all glusterfs pods -3.12.2-17, block-prov and heketi-7.0.0.6). After upgrade, some of the app pods were -respinned and it was seen that none of them were able to re-mount the block pvcs. Steps Performed: 1. Created a CNS 3.9 setup. 2. Created block-devices bk1-bk9 to consume all the space of block-hosting vol_ed54bbe8770a4597f8c350f044177bca 3. Created some more block devices to fill up the 2nd BHV - vol_868297e82df2a131b228a20a746813da . Also created app pods(bk2-1, bk3-1, bk4-1) with bind-mounted pvcs bk2, bk3 and bk4 4. Heketi reported free size=100 on vol_ed54bbe8770a4597f8c350f044177bca because of BZ#1584123 even though its space was completely consumed. 5. Hence, tried creating a new pvc(new3) of 5GB but it still tried to use the already full BHV-vol_ed54bbe8770a4597f8c350f044177bca and pvc statyed in pending state. 6. Upgraded the setup to CNS 3.10 and once heketi was upgraded, the pvc request for new3 was automatically completed(a new BHV vol_4805e6aa548e1db5713a46c55ca51f95 was created). 7. Even though reserved size is 2GB for a BHV of 100GB, the two older BHV's had resevred size=0( as they were fully consumed). 8. Created one or two more pvc(after-upfdate) etc and then re-spinned some of the app pods. 9. Multiple Re-spins of the pod bk-2 failed to mount the block device with following error message: """"""""""""""""""" Warning Failed 11s (x3 over 28s) kubelet, dhcp42-223.lab.eng.blr.redhat.com Error: failed to start container "foo": Error response from daemon: error setting label on mount source '/var/lib/origin/openshift.local.volumes/pods/3b044cde-a8fd-11e8-adda-005056a52b66/volumes/kubernetes.io~iscsi/pvc-6d5d4be1-a7ad-11e8-adda-005056a52b66': SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/3b044cde-a8fd-11e8-adda-005056a52b66/volumes/kubernetes.io~iscsi/pvc-6d5d4be1-a7ad-11e8-adda-005056a52b66 is not allowed: "input/output error """"""""""""""""""" 10. All the block-devices(bk1-bk8 and new1-new2) carved out of BHV's (with free size in MBs) are NOT getting mounted on any app pod and are reporing one or the other issue. bknew2-2 +++++++++ """"""""""""""""""""""""""""""""""""""" Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m default-scheduler Successfully assigned bknew2-2-kktx5 to dhcp42-84.lab.eng.blr.redhat.com Normal SuccessfulMountVolume 5m kubelet, dhcp42-84.lab.eng.blr.redhat.com MountVolume.SetUp succeeded for volume "default-token-d8wh2" Warning FailedMount 4m kubelet, dhcp42-84.lab.eng.blr.redhat.com MountVolume.WaitForAttach failed for volume "pvc-d0bde07f-a8ee-11e8-adda-005056a52b66" : failed to mount the volume as "xfs", it already contains mpath_member. Mount error: mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 --scope -- mount -t xfs -o defaults /dev/disk/by-path/ip-10.70.41.217:3260-iscsi-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 Output: Running scope as unit run-101409.scope. mount: /dev/sdg is already mounted or /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.41.217:3260-iqn.2016-12.org.gluster-block:35226dea-29a3-4194-818b-8bce0b662e5b-lun-0 busy Warning FailedMount 1m (x2 over 3m) kubelet, dhcp42-84.lab.eng.blr.redhat.com Unable to mount volumes for pod "bknew2-2-kktx5_glusterfs(73a215ab-a9ed-11e8-adda-005056a52b66)": timeout expired waiting for volumes to attach/mount for pod "glusterfs"/"bknew2-2-kktx5". list of unattached/unmounted volumes=[foo-vol] """""""""""""""""""""""""""""""""""""""""""""""""""" """"""""""""""""""""""""""" bk5-2-z89ph Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default-scheduler Successfully assigned bk5-2-z89ph to dhcp41-217.lab.eng.blr.redhat.com Normal SuccessfulMountVolume 2m kubelet, dhcp41-217.lab.eng.blr.redhat.com MountVolume.SetUp succeeded for volume "default-token-d8wh2" Warning FailedMount 10s kubelet, dhcp41-217.lab.eng.blr.redhat.com Unable to mount volumes for pod "bk5-2-z89ph_glusterfs(307c2364-a9d8-11e8-adda-005056a52b66)": timeout expired waiting for volumes to attach/mount for pod "glusterfs"/"bk5-2-z89ph". list of unattached/unmounted volumes=[foo-vol] """""""""""""""""""""""""""""" 11. The block devices from the BHV-vol_4805e6aa548e1db5713a46c55ca51f95 which has enough free space are successfully mounted on app pods. bkafter-up-1-bp5nq 1/1 Running 0 5h 10.128.2.43 dhcp41-217.lab.eng.blr.redhat.com bknew3-1-wkks5 1/1 Running 0 5h 10.128.2.47 dhcp41-217.lab.eng.blr.redhat.com I ssue: Since the block devices are pre-allocated with the size requested during creation, can less space on respective BHV's affect proper mounting of those devices? ========================================== Space utilization of the bricks ----------------------------- [root@dhcp42-137 ~]# for i in `oc get pods -o wide| grep glusterfs|cut -d " " -f1` ; do echo $i; echo +++++++++++++++++++++++; oc exec $i -- df -kh|grep heketi; done glusterfs-storage-59jt5 +++++++++++++++++++++++ /dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_42d291a6d1f16dfecc8f9c321ec89c31 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_42d291a6d1f16dfecc8f9c321ec89c31 /dev/mapper/vg_49a8f6746a305fd3696f9aa9c1f32d6f-brick_9011b8f81ccc4c9cebf2a74e88ae70d6 100G 100G 180K 100% /var/lib/heketi/mounts/vg_49a8f6746a305fd3696f9aa9c1f32d6f/brick_9011b8f81ccc4c9cebf2a74e88ae70d6 /dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_a52072845f7ae8361e359a6f907f20cf 120G 120G 931M 100% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_a52072845f7ae8361e359a6f907f20cf /dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_c5be1ff405a61f424ed2205d45c5d360 100G 99G 2.0G 99% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_c5be1ff405a61f424ed2205d45c5d360 glusterfs-storage-7s5h6 +++++++++++++++++++++++ /dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_f21f7bd09d1c078c6f3191e811158bb2 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_f21f7bd09d1c078c6f3191e811158bb2 /dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_4584491f8c346580734d7b1e710b08de 100G 100G 204K 100% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_4584491f8c346580734d7b1e710b08de /dev/mapper/vg_556c81099602cc71dca84fbe4926c6b0-brick_dad0c9b47e85ac26c7ca09ea1832bedc 120G 120G 931M 100% /var/lib/heketi/mounts/vg_556c81099602cc71dca84fbe4926c6b0/brick_dad0c9b47e85ac26c7ca09ea1832bedc /dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_a3f1f0d91f9ab6735b35d605e5ec7d98 100G 99G 2.0G 99% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_a3f1f0d91f9ab6735b35d605e5ec7d98 glusterfs-storage-r9b56 +++++++++++++++++++++++ /dev/mapper/vg_4264459401ed48d0939685f2bd5472fb-brick_8724d3d50a9fc4c74f022e9e54b86874 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_4264459401ed48d0939685f2bd5472fb/brick_8724d3d50a9fc4c74f022e9e54b86874 /dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_7728805e80a10df134c6991134dee9f0 100G 100G 212K 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_7728805e80a10df134c6991134dee9f0 /dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_2fc66410598a5f3819d686d28fe3882d 120G 120G 931M 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_2fc66410598a5f3819d686d28fe3882d /dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_c5a42e8b391e4594c274e9255b65a6a8 100G 99G 2.0G 99% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_c5a42e8b391e4594c274e9255b65a6a8 [root@dhcp42-137 ~]# Version-Release number of selected component (if applicable): ++++++++++++++++++++++++ Before upgrade ============ Gluster Version before upgrade = -3.8.4-54.8 heketi verison before upgrade - 6.0.0-7.4 gluster-block-0.2.1-14.1.el7rhgs.x86_64 After Upgrade ================ Current gluster verison = 3.12.2-17 gluster-block-0.2.1-25.el7rhgs.x86_64 heketi = heketi-7.0.0-6 How reproducible: ++++++++++++++++++++++++ This testing is performed once Actual results: ++++++++++++++++++++++++ None of the block dveices which belong to BHV's which are full are now getting mounted on app pods. Expected results: ++++++++++++++++++++++++ Since the block devices are pre-allocated with space, shouldn't they always get mounted if the block devices itself have enough space.
updated the doc text field. kindly review.
changed it to raw space.
Doc Text looks OK
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2686