Bug 1279547 - cinder-api is consuming around 10GB RAM when running a "snapshot-list" command on the large set of data
cinder-api is consuming around 10GB RAM when running a "snapshot-list" comman...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
7.0 (Kilo)
All Linux
urgent Severity high
: async
: 7.0 (Kilo)
Assigned To: Gorka Eguileor
lkuchlan
: ZStream
: 1279562 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-09 12:13 EST by Eric Harney
Modified: 2016-04-26 11:29 EDT (History)
11 users (show)

See Also:
Fixed In Version: openstack-cinder-2015.1.3-4.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, Cinder calculated filters, limits, and offset locally instead of in the database. This would result in large amounts of memory being required to perform these calculations because all non-deleted entries in the database would need to be retrieved. With this update, these calculations are now performed in the database, and only the data required for listing the output is retrieved.
Story Points: ---
Clone Of: 1278576
Environment:
Last Closed: 2016-04-26 11:29:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eric Harney 2015-11-09 12:13:11 EST
+++ This bug was initially created as a clone of Bug #1278576 +++

Description of problem:

Cinder is performing an inefficient query: when looking
for information about snapshots associated with a particular volume,
it loads all the rows from the database and then filters them in
memory.  


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Create a lot of snapshots
2. Run cinder snapshot-list command, use volume_id to return only snapshots related to that specific volume

Actual results:

1. cinder-api is consuming a lot of memory after performing cinder snapshot-list
2. query returns a lot of unrelated records from the database (snapshots of a different volumes)


Expected results:

memory usage should be normal, 
query should return only relevant records


Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2015-11-05 15:21:53 EST ---

Since this issue was entered in bugzilla without a release flag set, rhos-8.0? has been automatically added to ensure that it is properly evaluated for this release.

--- Additional comment from Flavio Percoco on 2015-11-06 08:10:19 EST ---

Added a reference to the patch. This is the link to the scratch build:

https://brewweb.devel.redhat.com/taskinfo?taskID=10073456
Comment 1 Jon Schlueter 2015-11-09 13:27:21 EST
*** Bug 1279562 has been marked as a duplicate of this bug. ***
Comment 3 Eric Harney 2015-12-04 16:38:24 EST
This fix is not working correctly, causes the problem seen in bug 1287621.
Comment 6 lkuchlan 2016-04-03 09:58:27 EDT
Tested using:
openstack-cinder-2015.1.2-5.el7ost.noarch
python-cinder-2015.1.2-5.el7ost.noarch
python-cinderclient-1.2.1-1.el7ost.noarch

Verification flow: 

* Create a volume

[root@cougar01 ~(keystone_admin)]# cinder create 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2016-04-03T09:36:00.148098      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | f784b4bb-5f92-4016-8cc8-7dbae227e462 |
|       metadata      |                  {}                  |
|     multiattach     |                false                 |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

* Insert lots of snapshots related to that specific volume, under snapshots table in cinder database

[root@cougar01 ~(keystone_admin)]# for i in {1..10000}; do mysql cinder -e "INSERT INTO snapshots(id,deleted,status,volume_id,user_id,project_id ) values($i,0,'available','f784b4bb-5f92-4016-8cc8-7dbae227e462','635bff15a7d744458779a181f451bbdf','fdb3fb93b0474349afae4bcffe7855c1');"; done 

* Run cinder snapshot-list command, use volume_id


[root@cougar01 ~(keystone_admin)]# cinder snapshot-list
+-------+--------------------------------------+-----------+--------------+------+
|   ID  |              Volume ID               |   Status  | Display Name | Size |
+-------+--------------------------------------+-----------+--------------+------+
|   1   | f784b4bb-5f92-4016-8cc8-7dbae227e462 | available |      -       |  -   |
|   10  | f784b4bb-5f92-4016-8cc8-7dbae227e462 | available |      -       |  -   |
|  100  | f784b4bb-5f92-4016-8cc8-7dbae227e462 | available |      -       |  -   |
|  1000 | f784b4bb-5f92-4016-8cc8-7dbae227e462 | available |      -       |  -   |
| 10000 | f784b4bb-5f92-4016-8cc8-7dbae227e462 | available |      -       |  -   |
.
.
.


[root@cougar01 ~]# top | grep cinder-api
12277 cinder    20   0  449884  63836   7580 S   5.3  0.1   6:43.27 cinder-api                                                                                                                
12277 cinder    20   0  449884  63836   7580 S   1.7  0.1   6:43.32 cinder-api                                                                                                                
12277 cinder    20   0  449884  63836   7580 S   2.0  0.1   6:43.38 cinder-api                                                                                                                
12277 cinder    20   0  449884  63836   7580 S   2.0  0.1   6:43.44 cinder-api                                                                                                                
12277 cinder    20   0  449884  63836   7580 S   1.7  0.1   6:43.49 cinder-api
Comment 8 errata-xmlrpc 2016-04-26 11:29:37 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0688.html

Note You need to log in before you can comment on or make changes to this bug.