Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1721068

Summary:	allocations database is not properly cleaned
Product:	Red Hat OpenStack	Reporter:	Eduard Barrera <ebarrera>
Component:	openstack-nova	Assignee:	Sylvain Bauza <sbauza>
Status:	CLOSED ERRATA	QA Contact:	Paras Babbar <pbabbar>
Severity:	medium	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	akaris, bbowen, cgaynor, chrisbro, dasmith, eglynn, igarciam, jhakimra, jhardee, jthomas, kchamart, lyarwood, mbooth, pbabbar, rdiwakar, rsunog, sbauza, sgordon, stephenfin, tvainio, vcojot
Target Milestone:	z13	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-17.0.13-16.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-28 18:32:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eduard Barrera 2019-06-17 09:08:09 UTC

Description of problem:

nova_api is groving huge, it seems never cleaned.

MariaDB [nova_api]> select count(*) from request_specs;
|  1126585 |

MariaDB [nova_api]> select count(*) from consumers;
|   444235 |

We have "nova-manage db archive_deleted_rows" running daily on each of our clouds, but the following seem to pile up regardless.

I've been looking at similar finding yesterday and today between several clouds and seems there's 2 culprits that leave allocations behind that effectively "consume" cloud resources but are not there:

- migrations -- sometimes the migration makes a reservation, fails, the instance gets back up but the reservation does not get cleared
- upgrades / maintenance / crashes on DB or rabbitmq -- high amount of allocations seem to correlate with clouds recently upgraded from RHOSP11 to 13 (upgrade is with our own installer following upstream guidelines of disabling services, doing offline data migrations and bringing the service back up)

And consumers/request_specs date back all the way to the original creation of the cloud. Seems these never get cleared.


Can we have a query or tool to properly clear the database of unused intried ?

Version-Release number of selected component (if applicable):
OSP13

How reproducible:
unsure

Steps to Reproduce:
unsure

Actual results:
Entries in the database are left there for ever

Expected results:
Entries removed or a way to properly clean it

Additional info:

Comment 4 Sylvain Bauza 2019-06-19 15:13:03 UTC

There are multiple concerns in this RFE, all of them relating to the Placement DB. I'll try to summarize :

1/ the 'request_specs' DB record related to an instance UUID is not deleted when the instance is itself deleted. That's a known behaviour and discussed with the community where some operators wanted to keep the original request even if the instance is deleted. That said, starting with Queens (and backported to Pike), archiving instances is now deleting corresponding instance_mapping and request_spec records [1].
For your case, just run the "nova-manage db archive_deleted_rows" command (since OSP13 == Queens) and you should see corresponding request_specs records be deleted.

2/ the 'consumers' DB record related to an instance UUID is not deleted when the instance is itself deleted. That's not really how it works. Consumer records are deleted when there are no left allocations against them. That's done by [2] but this is only Rocky code and newer releases. Fixing this would require both a backport to Queens/OSP13 and some SQL call for only deleting 'consumers' records not related to existing allocations.

3/ the 'allocations DB records (there are many per instance) related to an instance UUID are not deleted when a instance is deleted. That sounds a bug to me but there could be many possibilities why it doesn't work.
Could you please reply to me with some private comment about telling me whether those existing allocations are against instance UUIDs *or* migration UUIDs (ie. from the 'migrations' table records with the UUID field) ?
Ideally, I'd also see some allocations records, please.
We could possibly have a problem that is actually fixed by [3] but even if this change was recently merged, it's neither in a release nor packaged yet.

1/ and 2/ aren't root cause of the capacity issue you mention and can be purged anytime you want. 3/ requires some help hence the NEEDINFO.

Thanks.
-Sylvain

[1] https://review.opendev.org/#/c/515034/
[2] https://review.opendev.org/#/c/581086/
[3] https://review.opendev.org/#/c/661349/

Comment 11 Sylvain Bauza 2019-09-13 14:50:49 UTC

*** Bug 1749077 has been marked as a duplicate of this bug. ***

Comment 27 Sylvain Bauza 2020-02-03 10:15:50 UTC

The patch [1] is currently waiting for reviews with already one acception (+2). I already asked other core folks to look at my change, some of them told me they will do it for this week.
[1] https://review.opendev.org/#/c/670112/

> Customer's question ss how do I correlate between a provider allocation and an instance (by uuid, name or any other unique key )? If I can't do that I'll have no way of knowing which allocations can be deleted. Take into account that I > have hundreds of VMs running with probably thousands of allocations in place.  

Yeah, by looking at the customer_uuid indeed for each allocation.

Comment 50 errata-xmlrpc 2020-10-28 18:32:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openstack-nova bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4393