Cloned from launchpad blueprint https://blueprints.launchpad.net/oslo/+spec/db-oslo-purge-engine.

Description:

There are several points:

1) Nova & other core projects never delete data from the DBs and uses only soft deleting (There is special column in each table that indicates that entry is deleted).

2) To purge soft deleted records you have to get direct access to DB and do it manually by own script. Or use db-archiving feature that allows you to move soft deleted records from main to shadow tables. But db-archiving:
a) works very slow and as there is a complex logic around moving records from one table to another (we are not able to improve this)
b) db-archiving doesn’t take in account relationships between Tables.
c) db-archiving was implemented in Grizzly, during havana we found and fixed a lot of critical bugs, and didn’t see any bug reports form other, seems like nobody use it in production.
d) We didn’t find use cases for db-archiving.

3) Everybody that use OpenStack in production have some db purge script.

Due to these facts we should implement one purge engine that will be available out of box. This purge engine will cleanup considering of tables relationship. Implementation will be based on sql requests that do all work inside DB (without downloading all DB data) to improve deletions speed.

Purge engine's part of tables relationship control could be used next time to improve shadow tables.

This functionality will be available for administrators through the REST api. And probably we should think about periodic task that will cleanup DB.

Specification URL (additional information):

None