Bug 1243301

Summary: nova API cannot allocate memory due to horizon os-simple-tenant-usage calls
Product: Red Hat OpenStack Reporter: Martin Schuppert <mschuppe>
Component: python-django-horizonAssignee: Matthias Runge <mrunge>
Status: CLOSED ERRATA QA Contact: Ido Ovadia <iovadia>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: aortega, athomas, dmaley, eglynn, mrunge, pablo.iranzo, rdopiera, sgordon, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-django-horizon-2014.2.3-6.1.el7ost Doc Type: Bug Fix
Doc Text:
Cause: The Nova API call to get the simple tenant usage data is currently very costly in terms of memory used. Repeated calls may cause out of memory errors, especially when there are a lot of deleted nodes in the database. Consequence: Nova crashes with "out of memory" error. Workaround: Disable displaying of the simple tenant usage statistics on the Horizon side, by blacklisting the nova extension using the following configuration: OPENSTACK_NOVA_EXTENSIONS_BLACKLIST = ["SimpleTenantUsage"] Result: The simple tenant usage statistics will no longer be displayed in Horizon.
 Story Points: ---
Clone Of:
: 1244106 (view as bug list) Environment:
Last Closed: 2015-08-24 20:16:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1244106    

Description Martin Schuppert 2015-07-15 07:25:29 UTC 
Description of problem:

When logging into horizon, horizon sends an API request to nova inquiring about usage statistics.
Among the api requests is os-simple-tenant-usage. We see that, if there are thousands of deleted VMs in the nova DB, this api call makes the nova-api process memory consumption skyrocket. If there are enough deleted VMs in the DB the nova-api eventually crashes due to lack of memory.

I was able to reproduce this in a lab setup (by running a script that creates many VMs and then deletes them continuously). I was also able to make the problem go away by deleting the VM entries from the nova database. 

Version-Release number of selected component (if applicable):
OSP6 

Proposed fix - https://review.openstack.org/#/c/201221/

~~~
Add an OPENSTACK_SIMPLE_TENANT_USAGE_ENABLED settings option


The users may want to disable the simple tenant usage display for
efficiency reasons -- currently this gets very slow and may run out of
memory if there are a lot of deleted machines in the Nova database.

In the 2.0 API we can disable it on the Nova side, but it won't be
possible anymore with the 2.1 API.
~~~



Additional info:
related to BZ1154237
Comment 2 Radomir Dopieralski 2015-07-15 08:16:58 UTC 
We came up with a more generic solution:

~~~
Add OPENSTACK_NOVA_EXTENSIONS_BLACKLIST option to settings

This lets us disable any Nova extension we like, not just the
simple tenant usage.
~~~
Comment 10 Radomir Dopieralski 2015-08-14 07:14:29 UTC 
The patch has been merged upstream.
Comment 12 Ido Ovadia 2015-08-19 15:01:33 UTC 
Verified
========
openstack-dashboard-2014.2.3-6.1.el7ost.noarch
python-django-horizon-2014.2.3-6.1.el7ost.noarch
Comment 14 errata-xmlrpc 2015-08-24 20:16:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1679.html