Bug 1621884 - [GSS] heketi sets freesize value to large inaccurate number for block volumes
Summary: [GSS] heketi sets freesize value to large inaccurate number for block volumes
Keywords:
Status: CLOSED DUPLICATE of bug 1584123
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: heketi
Version: cns-3.9
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: John Mulligan
QA Contact: Prasanth
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-23 19:14 UTC by Anton Mark
Modified: 2021-12-10 17:07 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-10 19:10:46 UTC
Embargoed:


Attachments (Terms of Use)

Description Anton Mark 2018-08-23 19:14:51 UTC
Description of problem:
After a period of time, the freesize value gets set to a large value, like 1200, which is larger than the 500 GB volume backing the block storage. This is problematic because heketi will keep trying to provision block storage to this 500 GB volume since it thinks it has room. Over time the "1200" value grows larger and larger as heketi attempts to provision more blockvolumes to the volume.


It takes manual intervention to reset the freesize to 10. To reflect the proper state in gluster.

Version-Release number of selected component (if applicable):
heketi-6.0.0-7.4.el7rhgs.x86_64                             Wed May 30 08:51:41 2018


How reproducible:
Customer is able to reproduce this after enough time.


Steps to Reproduce:
Let heketi provision storage for OpenShift cluster as normal part of it's operations. Eventually the condition will occur and cause any new block volume request to fail because it believes there is more space than actually exists.

Actual results:
Provisioning new block volumes fails.

Expected results:
Provisioning new block volume should succeed.

Additional info:
./gluster-block/sosreport-gnekic-20180809093958/var/log/gluster-block/gluster-block-configshell.log:[INFO] 2018-07-02 17:08:37 Created user-backed storage object blockvol_fb53e9b4f2150143835ae53386bd9e29 size 214748364800.
./gluster-block/sosreport-gnekic-20180809093958/var/log/gluster-block/gluster-blockd.log:[2018-07-02 22:08:37.286513] INFO: create request, volume=vol_08ae3dad229fba230ddcb4e0526c1b51 blockname=blockvol_fb53e9b4f2150143835ae53386bd9e29 blockhosts=10.128.33.170,10.128.33.171,10.128.33.172 filename=78c03684-3027-440d-a4a4-73c404ed0b22 authmode=1 passwd=627d2d7e-68e7-4389-8d19-6fa217ed5d9c size=214748364800 [at block_svc_routines.c+2087 :<block_create_1_svc_st>]
./gluster-block/sosreport-gnekic-20180809093958/var/log/gluster-block/tcmu-runner.log:2018-07-03 12:39:40.925 8950 [ERROR] reconfig_device:168 blockvol_fb53e9b4f2150143835ae53386bd9e29: Reconfiguration is not supported with this device.
./gluster-block/sosreport-gnekic-20180809093958/var/log/gluster-block/tcmu-runner.log:2018-07-11 02:05:42.266 55820 [ERROR] reconfig_device:168 blockvol_fb53e9b4f2150143835ae53386bd9e29: Reconfiguration is not supported with this device.
./gluster-block/sosreport-gnekic-20180809093958/var/log/gluster-block/tcmu-runner.log:2018-08-03 22:25:04.211 11339 [ERROR] tcmu_create_glfs_object:448 blockvol_fb53e9b4f2150143835ae53386bd9e29: glfs_init failed: Input/output error
./gluster-block/sosreport-gnekic-20180809093958/var/log/gluster-block/tcmu-runner.log:2018-08-03 22:25:05.583 11339 [ERROR] tcmu_glfs_open:513 blockvol_fb53e9b4f2150143835ae53386bd9e29: tcmu_create_glfs_object failed



***COLLAB-SHELL SOSREPORTS:
[amark@collab-shell sosreport-20180809-104358]$ pwd
/cases/02158924/sosreport-20180809-104358
[amark@collab-shell sosreport-20180809-104358]$ ls -l
total 16
drwxrwsrwx+ 14 amark support 4096 Aug 21 15:44 hoecprvhek01.na.xom.com
drwxrwsrwx+ 17 amark support 4096 Aug 21 15:44 hoecpsyglu04.na.xom.com
drwxrwsrwx+ 17 amark support 4096 Aug 21 15:45 hoecpsyglu05.na.xom.com
drwxrwsrwx+ 17 amark support 4096 Aug 21 15:45 hoecpsyglu06.na.xom.com


Note You need to log in before you can comment on or make changes to this bug.