Bug 1243301 – nova API cannot allocate memory due to horizon os-simple-tenant-usage calls

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1243301 - nova API cannot allocate memory due to horizon os-simple-tenant-usage calls

Summary:	nova API cannot allocate memory due to horizon os-simple-tenant-usage calls

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-django-horizon (Show other bugs)
Sub Component:
Version:	6.0 (Juno)
Hardware:	Unspecified Unspecified

Priority	high Severity high
Target Milestone:	---
Target Release:	6.0 (Juno)
Assigned To:	Matthias Runge
QA Contact:	Ido Ovadia
Docs Contact:

URL:
Whiteboard:
Keywords:	ZStream

Depends On:
Blocks:	1244106
	Show dependency tree / graph

Reported:	2015-07-15 03:25 EDT by Martin Schuppert
Modified:	2018-02-08 05:46 EST (History)
CC List:	10 users (show)

See Also:
Fixed In Version:	python-django-horizon-2014.2.3-6.1.el7ost
Doc Type:	Bug Fix
Doc Text:	Cause: The Nova API call to get the simple tenant usage data is currently very costly in terms of memory used. Repeated calls may cause out of memory errors, especially when there are a lot of deleted nodes in the database. Consequence: Nova crashes with "out of memory" error. Workaround: Disable displaying of the simple tenant usage statistics on the Horizon side, by blacklisting the nova extension using the following configuration: OPENSTACK_NOVA_EXTENSIONS_BLACKLIST = ["SimpleTenantUsage"] Result: The simple tenant usage statistics will no longer be displayed in Horizon.
Story Points:	---
Clone Of:
Clones:	1244106 (view as bug list)
Environment:
Last Closed:	2015-08-24 16:16:22 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Launchpad	1474241	None	None	None	Never
Red Hat Knowledge Base (Solution)	1530763	None	None	None	Never
OpenStack gerrit	201221	None	None	None	Never
Red Hat Product Errata	RHSA-2015:1679	normal	SHIPPED_LIVE	Moderate: python-django-horizon security and bug fix update	2015-08-24 20:15:52 EDT

Groups: None (edit)

Description Martin Schuppert 2015-07-15 03:25:29 EDT

Description of problem:

When logging into horizon, horizon sends an API request to nova inquiring about usage statistics.
Among the api requests is os-simple-tenant-usage. We see that, if there are thousands of deleted VMs in the nova DB, this api call makes the nova-api process memory consumption skyrocket. If there are enough deleted VMs in the DB the nova-api eventually crashes due to lack of memory.

I was able to reproduce this in a lab setup (by running a script that creates many VMs and then deletes them continuously). I was also able to make the problem go away by deleting the VM entries from the nova database. 

Version-Release number of selected component (if applicable):
OSP6 

Proposed fix - https://review.openstack.org/#/c/201221/

~~~
Add an OPENSTACK_SIMPLE_TENANT_USAGE_ENABLED settings option


The users may want to disable the simple tenant usage display for
efficiency reasons -- currently this gets very slow and may run out of
memory if there are a lot of deleted machines in the Nova database.

In the 2.0 API we can disable it on the Nova side, but it won't be
possible anymore with the 2.1 API.
~~~



Additional info:
related to BZ1154237

Comment 2 Radomir Dopieralski 2015-07-15 04:16:58 EDT

We came up with a more generic solution:

~~~
Add OPENSTACK_NOVA_EXTENSIONS_BLACKLIST option to settings

This lets us disable any Nova extension we like, not just the
simple tenant usage.
~~~

Comment 10 Radomir Dopieralski 2015-08-14 03:14:29 EDT

The patch has been merged upstream.

Comment 12 Ido Ovadia 2015-08-19 11:01:33 EDT

Verified
========
openstack-dashboard-2014.2.3-6.1.el7ost.noarch
python-django-horizon-2014.2.3-6.1.el7ost.noarch

Comment 14 errata-xmlrpc 2015-08-24 16:16:22 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1679.html

Note You need to log in before you can comment on or make changes to this bug.