Bug 1275360 - [Cinder] When cinder conf allows X volumes, engine actually allows X-1 cinder volumes to be created
[Cinder] When cinder conf allows X volumes, engine actually allows X-1 cinder...
Status: CLOSED CURRENTRELEASE
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage (Show other bugs)
3.6.0.2
Unspecified Unspecified
medium Severity medium (vote)
: ovirt-3.6.2
: 3.6.2
Assigned To: Daniel Erez
Natalie Gavrielov
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-26 12:17 EDT by Natalie Gavrielov
Modified: 2016-03-10 02:24 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-18 06:12:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.6.z+
ylavi: planning_ack+
rule-engine: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)
snapshot, engine.log, vdsm.log (1.03 MB, application/x-gzip)
2015-10-26 12:17 EDT, Natalie Gavrielov
no flags Details
engine.log (3.68 MB, application/x-gzip)
2015-12-15 12:40 EST, Natalie Gavrielov
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 47780 master MERGED core: fix cinder volumes limits calculation Never
oVirt gerrit 47790 ovirt-engine-3.6 MERGED core: fix cinder volumes limits calculation Never

  None (edit)
Description Natalie Gavrielov 2015-10-26 12:17:53 EDT
Created attachment 1086551 [details]
snapshot, engine.log, vdsm.log

Description of problem:

Cinder default configuration allows 10 volumes to exist.
When configuring 10 cinder disks, the 10th operation fails with an error message saying "volumes allowed (10) exceeded" (this message should appear when creating the 11th cinder disk..). 

Version-Release number of selected component (if applicable):

rhevm-3.6.0.2-0.1.el6.noarch


How reproducible:
100%

Steps to Reproduce:

Create 10 cinder disks.


Actual results:

The 10th operation fails with the following message:
Error while executing action: Cannot add Virtual Machine Disk. Maximum number of volumes allowed (10) exceeded - could not create Cinder disks on Storage Domain cinder.

engine.log:
2015-10-26 15:13:58,190 WARN  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-3) [11fb6cdd] CanDoAction of action 'AddDisk' failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM_DISK,CANNOT_ADD_CINDER_DISK_VOLUME_LIMIT_EXCEEDED,$maxTotalVolumes 10,$storageName cinder

cinder list shows it has 9 disks:
[root@natalie-cinder ~(keystone_natalie)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 440b9fe1-2f74-470f-b996-ed101c842593 | available |    test9     |  9   |     ceph    |  false   |             |
| 52d61e51-96a3-488c-a9bd-821f892b24d2 | available |    test5     |  3   |     ceph    |  false   |             |
| 56927133-12fb-420b-af58-fe18303e95f2 | available |    test8     |  11  |     ceph    |  false   |             |
| 87162c3c-4e1b-49bc-827e-8310250147a8 | available |    test7     |  14  |     ceph    |  false   |             |
| 9b73bdb1-f837-4c93-b212-e3e23e046bf3 | available |    test3     |  7   |     ceph    |  false   |             |
| b33d08f0-0589-4609-b3ab-f1df7c6eecc3 | available |    test1     |  6   |     ceph    |  false   |             |
| c166eba5-ffc2-4f5f-9b67-1aaa8e26feb2 | available |    test2     |  5   |     ceph    |  false   |             |
| cc54fb02-ccc3-4762-bf4f-8906ea8b2534 | available |    test4     |  4   |     ceph    |  false   |             |
| ee227b67-7e38-4758-90e8-3ce83bd9aa77 | available |    test6     |  12  |     ceph    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

Configuration shows the max volumes number is 10:

MariaDB [cinder]> select * from quota_classes;
+---------------------+------------+------------+---------+----+------------+-------------------+------------+
| created_at          | updated_at | deleted_at | deleted | id | class_name | resource          | hard_limit |
+---------------------+------------+------------+---------+----+------------+-------------------+------------+
| 2015-10-20 16:34:30 | NULL       | NULL       |       0 |  1 | default    | volumes           |         10 |
| 2015-10-20 16:34:30 | NULL       | NULL       |       0 |  2 | default    | snapshots         |         10 |
| 2015-10-20 16:34:30 | NULL       | NULL       |       0 |  3 | default    | gigabytes         |       1000 |
| 2015-10-20 16:34:30 | NULL       | NULL       |       0 |  4 | default    | consistencygroups |         10 |
+---------------------+------------+------------+---------+----+------------+-------------------+------------+
4 rows in set (0.00 sec)

Expected results:

The 10th creation of a cinder disk should pass successfully.
Only the 11th should fail. 

Additional info:
Cinder version 1.1.1
vdsm-4.17.8-1.el7ev.noarch
Comment 1 Yaniv Lavi (Dary) 2015-10-29 08:53:30 EDT
In oVirt testing is done on single release by default. Therefore I'm removing the 4.0 flag. If you think this bug must be tested in 4.0 as well, please re-add the flag. Please note we might not have testing resources to handle the 4.0 clone.
Comment 2 Natalie Gavrielov 2015-12-15 12:38:05 EST
The number of volumes I defined a long time ago was 10000, in order to verify this issue I changed the number of volumes to 10, the following way:

cinder  quota-class-update --volumes 10 --snapshots 10 --gigabytes 102400 default
ERROR: 'NoneType' object has no attribute '_info'

despite the error I get above.. the value in the db changes:

MariaDB [cinder]> select * from quota_classes;
+---------------------+---------------------+------------+---------+----+------------+-------------------+------------+
| created_at          | updated_at          | deleted_at | deleted | id | class_name | resource          | hard_limit |
+---------------------+---------------------+------------+---------+----+------------+-------------------+------------+
| 2015-11-15 15:43:34 | 2015-12-15 16:43:18 | NULL       |       0 |  1 | default    | volumes           |         10 |
| 2015-11-15 15:43:34 | 2015-12-15 16:43:18 | NULL       |       0 |  2 | default    | snapshots         |         10 |
| 2015-11-15 15:43:34 | 2015-11-29 15:31:05 | NULL       |       0 |  3 | default    | gigabytes         |     102400 |
| 2015-11-15 15:43:42 | NULL                | NULL       |       0 |  4 | default    | consistencygroups |         10 |
+---------------------+---------------------+------------+---------+----+------------+-------------------+------------+

number of existing cinder disks is 3:
[root@natalie-cinder ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+----------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |  Display Name  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+----------------+------+-------------+----------+-------------+
| 1de77a07-edcc-4626-b7f4-b61237a857bd | available | cinder-disk-10 |  10  |     ceph    |  false   |             |
| 2b3ab636-e951-4007-91aa-3f8c56ca9af8 | available | cinder-disk-10 |  10  |     ceph    |  false   |             |
| 8c9b1566-9ba9-46b7-8c3b-212ff24d56f5 | available | cinder-disk-10 |  10  |     ceph    |  false   |             |
+--------------------------------------+-----------+----------------+------+-------------+----------+-------------+


When trying to create a new cinder disk UI displays the following error:

Error while executing action: Cannot add Virtual Machine Disk. Maximum number of volumes allowed (10) exceeded - could not create Cinder disks on Storage Domain cinder.

and engine.log:
2015-12-15 16:39:18,197 WARN  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-10) [68d0f71a] CanDoAction of action 'AddDisk' failed for user jdoe@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM_DISK,CANNOT_ADD_CINDER_DISK_VOLUME_LIMIT_EXCEEDED,$maxTotalVolumes 10,$storageName cinder

When I changed back the number of volumes to 10000, I was able to create cinder volumes.

I'll attach engine.log in a few
Comment 3 Red Hat Bugzilla Rules Engine 2015-12-15 12:38:07 EST
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 4 Natalie Gavrielov 2015-12-15 12:40 EST
Created attachment 1106114 [details]
engine.log
Comment 5 Eyal Edri 2015-12-17 08:51:23 EST
ovirt 3.6.1 is GA.
moving all open bugs to 3.6.2
Comment 6 Eyal Edri 2015-12-17 08:59:43 EST
ovirt 3.6.1 was GA'd, all open bugs should move to 3.6.2.
Comment 7 Daniel Erez 2015-12-20 08:15:41 EST
Hi Natalie,

Can you please verify that the Username/Tenant that are used for Cinder DB credentials are identical to the ones in WebAdmin. E.g. both are using 'admin' username and 'admin' Tenant Name. As it seems that the number of volumes that is fetched for validation is different than the output you've specified.

Thanks!
Daniel
Comment 8 Natalie Gavrielov 2015-12-21 09:15:03 EST
(In reply to Daniel Erez from comment #7)
> Hi Natalie,
> 
> Can you please verify that the Username/Tenant that are used for Cinder DB
> credentials are identical to the ones in WebAdmin. E.g. both are using
> 'admin' username and 'admin' Tenant Name. As it seems that the number of
> volumes that is fetched for validation is different than the output you've
> specified.
> 
> Thanks!
> Daniel

I use username:admin, tenant:admin, on the engine side.
On cinder side:
[root@natalie-cinder ~(keystone_admin)]# cat keystonerc_admin
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PASSWORD=qum5net
export OS_AUTH_URL=http://10.35.161.93:5000/v2.0/
export OS_REGION_NAME=RegionOne
export PS1='[\u@\h \W(keystone_admin)]\$ '

I use the script above before running any cinder related commands.

Also, if I was using different users, I would expect a different behaviour for this section from comment #2:
> When I changed back the number of volumes to 10000, I was able to create 
> cinder volumes.
Comment 9 Natalie Gavrielov 2015-12-22 12:05:12 EST
Verified rhevm-3.6.1.3-0.1.el6.noarch (build 3.6.1-7)
cinder version kilo (openstack-cinder-2015.1.1-1.el7.noarch)

Scenario:

1. Cheking the max number of cinder volumes that can be created.
MariaDB [cinder]> select * from quota_classes;
+---------------------+------------+------------+---------+----+------------+-------------------+------------+
| created_at          | updated_at | deleted_at | deleted | id | class_name | resource          | hard_limit |
+---------------------+------------+------------+---------+----+------------+-------------------+------------+
| 2015-12-21 20:20:28 | NULL       | NULL       |       0 |  1 | default    | volumes           |         10 |
| 2015-12-21 20:20:28 | NULL       | NULL       |       0 |  2 | default    | snapshots         |         10 |
| 2015-12-21 20:20:28 | NULL       | NULL       |       0 |  3 | default    | gigabytes         |       1000 |
| 2015-12-21 20:20:34 | NULL       | NULL       |       0 |  4 | default    | consistencygroups |         10 |
+---------------------+------------+------------+---------+----+------------+-------------------+------------+

2. Create 10 cinder volumes using ovirt-remote/UI.

Result: 10th volume is created (11 fails as should).

Note You need to log in before you can comment on or make changes to this bug.