Description of problem: The purge_deleted function of heat-manage does not clean up raw_template entries that are no longer referenced by the stack table (fields: raw_template_id or prev_raw_template_id). OSP Director deployments creates unreferenced raw_template entries and they continue to build up over time causing this table to potentially consume significant space. Looking at the code, the current purge_deleted function only operates on raw_template IDs that are being purged from the stack table. Here is an example SQL that will find these unreferenced entries: SELECT x_id AS Abandoned_Raw_Template_IDs FROM (SELECT a.id AS x_id FROM raw_template a LEFT JOIN stack b ON (a.id = b.raw_template_id) WHERE b.raw_template_id IS NULL) AS x, (SELECT a.id AS y_id FROM raw_template a LEFT JOIN stack b ON (a.id = b.prev_raw_template_id) WHERE b.prev_raw_template_id IS NULL) AS y WHERE x.x_id = y.y_id; Version-Release number of selected component (if applicable): openstack-heat-common-5.0.1-6.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. On a fresh OSP-d undercloud; deploy a test overcloud. 2. Delete the overcloud heat stack. 3. Wait >1 min after stack delete completes. 4. Run: sudo heat-manage purge_deleted -g minutes 1 5. Verify from the database that the stack table is empty but entries remain in the raw_template table: select count(1) from stack; select count(1) from raw_template; 6. These entries will continue to build up as the Overcloud stack is redeployed or updated. Actual results: raw_template table grows over time as Overcloud stack is redeployed or updated. Expected results: unreferenced raw_template IDs are purged. Additional info: Quick test showing the details: Prior to the deployment we have zero entries in stack and raw_template table: $ echo 'select count(1) from stack' |sudo mysql heat count(1) 0 $ echo 'select count(1) from raw_template' |sudo mysql heat count(1) 0 Deploy & delete overcloud $ openstack overcloud deploy \ .... $ heat stack-delete overcloud Wait at least 1 min Verify the number of entries in stack and raw_template tables: $ echo 'select count(1) from stack' |sudo mysql heat count(1) 168 $ echo 'select count(1) from raw_template' |sudo mysql heat count(1) 262 Use the SQL from above to get the number of unreferenced raw_template IDs $ echo 'SELECT count(x_id) AS Abandoned_Raw_IDs_count FROM (SELECT a.id AS x_id FROM raw_template a LEFT JOIN stack b ON (a.id = b.raw_template_id) WHERE b.raw_template_id IS NULL) AS x, (SELECT a.id AS y_id FROM raw_template a LEFT JOIN stack b ON (a.id = b.prev_raw_template_id) WHERE b.prev_raw_template_id IS NULL) AS y WHERE x.x_id = y.y_id' |sudo mysql heat Abandoned_Raw_IDs_count 47 Run the purge_deleted function $ sudo /bin/heat-manage purge_deleted -g minutes 1 List remaining counts from stack and raw_template tables to show the 47 abandoned entries remain. $ echo 'select count(1) from stack' |sudo mysql heat count(1) 0 $ echo 'select count(1) from raw_template' |sudo mysql heat count(1) 47
It may be related to https://bugs.launchpad.net/heat/+bug/1506077 . Unfortunately it wasn't backported yet.
Yes, I tested that fix and it does resolve the issue; could we backport please?
Upstream backport in progress.
Hey guys, I have a customer that is asking for the fix. It looks like the fix already got merged upstream to heat (stable/liberty) on 2016-07-04. Do we have any information on when it's going to be available in OSP 8? Best, Irina
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1871.html