Bug 1468882

Summary:	heketi allows to create block volumes exceeding the capacity of block hosting volume
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	krishnaram Karthick <kramdoss>
Component:	heketi	Assignee:	Mohamed Ashiq <mliyazud>
Status:	CLOSED ERRATA	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	cns-3.6	CC:	hchiramm, jkaur, madam, mliyazud, olim, rhs-bugs, storage-qa-internal, vinug
Target Milestone:	---
Target Release:	CNS 3.6
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	heketi-5.0.0-5 rhgs-volmanager-docker-3.3.0-7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-10-11 07:07:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1445448

Description krishnaram Karthick 2017-07-09 10:43:46 UTC

Description of problem:

Block volume creation should fail when the total capacity of block hosting volume is exhausted.

For ex, when block hosting volume is 500 GB, the total size of all block devices that could be created should be limited to 500 GB and any attempt to create a block device over and above that mark should fail. 

Gluster allows to create block devices above the capacity of volume it is hosting on as only sparse files are created by default in gluster. so heketi cannot depend on gluster command to fail, rather the request to create block volume should not proceed from heketi to gluster if the capacity is exceeded.

[root@dhcp47-10 ~]# heketi-cli blockvolume create --size=480
Name: blockvol_be664f518291761bf14fd5741362b5b8
Size: 480
Volume Id: be664f518291761bf14fd5741362b5b8
Cluster Id: 6bed12159b987cf70338ac6d1d3d0720
Hosts: [10.70.46.248 10.70.47.49 10.70.47.72]
IQN: iqn.2016-12.org.gluster-block:15695855-a4ab-4f20-9670-6eb9901ba1a1
LUN: 0
Hacount: 3
Username: 
Password: 
Block Hosting Volume: 78dfd43a5abacb9e8365500d54d4f6e5

[root@dhcp47-10 ~]# heketi-cli blockvolume create --size=30
Name: blockvol_0ced9f74bb5cb6d364a35f2752603c6c
Size: 30
Volume Id: 0ced9f74bb5cb6d364a35f2752603c6c
Cluster Id: 6bed12159b987cf70338ac6d1d3d0720
Hosts: [10.70.46.248 10.70.47.49 10.70.47.72]
IQN: iqn.2016-12.org.gluster-block:cfa768eb-723a-4a30-8b41-fdc56dfab6a9
LUN: 0
Hacount: 3
Username: 
Password: 
Block Hosting Volume: 78dfd43a5abacb9e8365500d54d4f6e5
[root@dhcp47-10 ~]# heketi-cli blockvolume create --size=30
Name: blockvol_b7d4e9737f20d25f0b305a10dd519076
Size: 30
Volume Id: b7d4e9737f20d25f0b305a10dd519076
Cluster Id: 6bed12159b987cf70338ac6d1d3d0720
Hosts: [10.70.46.248 10.70.47.49 10.70.47.72]
IQN: iqn.2016-12.org.gluster-block:04a8d622-314d-49bf-80c4-28ad2137ded6
LUN: 0
Hacount: 3
Username: 
Password: 
Block Hosting Volume: 78dfd43a5abacb9e8365500d54d4f6e5


[root@dhcp47-10 ~]# heketi-cli blockvolume list
Id:0ced9f74bb5cb6d364a35f2752603c6c    Cluster:6bed12159b987cf70338ac6d1d3d0720    Name:blockvol_0ced9f74bb5cb6d364a35f2752603c6c
Id:a3b7006bde73028b651d7362f6ba78a1    Cluster:6bed12159b987cf70338ac6d1d3d0720    Name:blockvol_a3b7006bde73028b651d7362f6ba78a1
Id:b7d4e9737f20d25f0b305a10dd519076    Cluster:6bed12159b987cf70338ac6d1d3d0720    Name:blockvol_b7d4e9737f20d25f0b305a10dd519076
Id:be664f518291761bf14fd5741362b5b8    Cluster:6bed12159b987cf70338ac6d1d3d0720    Name:blockvol_be664f518291761bf14fd5741362b5b8
[root@dhcp47-10 ~]# heketi-cli volume list
Id:78dfd43a5abacb9e8365500d54d4f6e5    Cluster:6bed12159b987cf70338ac6d1d3d0720    Name:vol_78dfd43a5abacb9e8365500d54d4f6e5 [block]
Id:fee4f0b4b98d1c468d1ebac9ed5acd44    Cluster:6bed12159b987cf70338ac6d1d3d0720    Name:heketidbstorage


[root@dhcp47-10 ~]# heketi-cli volume info 78dfd43a5abacb9e8365500d54d4f6e5
Name: vol_78dfd43a5abacb9e8365500d54d4f6e5
Size: 500
Volume Id: 78dfd43a5abacb9e8365500d54d4f6e5
Cluster Id: 6bed12159b987cf70338ac6d1d3d0720
Mount: 10.70.46.248:vol_78dfd43a5abacb9e8365500d54d4f6e5
Mount Options: backup-volfile-servers=10.70.47.49,10.70.47.72
Block: true
Free Size: -50  ---->>> SIZE GONE NEGATIVE
Block Volumes: [0ced9f74bb5cb6d364a35f2752603c6c a3b7006bde73028b651d7362f6ba78a1 b7d4e9737f20d25f0b305a10dd519076 be664f518291761bf14fd5741362b5b8]
Durability Type: replicate
Distributed+Replica: 3

Version-Release number of selected component (if applicable):

heketi-client-5.0.0-4.el7rhgs.x86_64
cns-deploy-5.0.0-6.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. create a block hosting device of 500GB
2. Try to create multiple block devices over above 500 GB mark

Actual results:
block vol creation succeeds oversubscribing from 500GB

Expected results:
block volume creation should fail

Additional info:

Comment 6 Mohamed Ashiq 2017-07-10 11:22:37 UTC

Fixed in upstream:

https://github.com/heketi/heketi/pull/793/commits/ca6a846df3200a0a11aaf87b79648c29b303559f

Comment 7 krishnaram Karthick 2017-09-01 16:08:21 UTC

Verified the fix in build - cns-deploy-5.0.0-25.el7rhgs.x86_64

Heketi now fails appropriately when a block device of size greater than that of block hosting volume is requested.

Access Modes:   
Events:
  FirstSeen     LastSeen        Count   From                                                            SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                                                            -------------   --------        ------                  -------
  11s           11s             1       gluster.org/glusterblock 695d0b4b-8ed9-11e7-be38-0a580a8100dd                   Normal          Provisioning            External provisioner is provisioning volume for claim "storage-project/test-04"
  10s           10s             1       gluster.org/glusterblock 695d0b4b-8ed9-11e7-be38-0a580a8100dd                   Warning         ProvisioningFailed      Failed to provision volume with StorageClass "glusterblockip": glusterblock: failed to create volume: [heketi] error creating volume No space
  11s           9s              3       persistentvolume-controller                                                     Normal          ExternalProvisioning    cannot find provisioner "gluster.org/glusterblock", expecting that a volume for the claim is provisioned either manually or via external software

Moving the bug to verified.

Comment 8 errata-xmlrpc 2017-10-11 07:07:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2879