Bug 1349111 - heat-manage purge_deleted doesn't clean up unreferenced IDs in raw_template table
Summary: heat-manage purge_deleted doesn't clean up unreferenced IDs in raw_template t...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 8.0 (Liberty)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Thomas Hervé
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks: 1360777
TreeView+ depends on / blocked
 
Reported: 2016-06-22 17:33 UTC by Matt Flusche
Modified: 2019-12-16 05:58 UTC (History)
10 users (show)

Fixed In Version: openstack-heat-5.0.1-7.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1360777 (view as bug list)
Environment:
Last Closed: 2016-09-14 14:41:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1506077 0 None None None 2016-06-23 11:46:56 UTC
OpenStack gerrit 333544 0 None MERGED Delete previous template upon update 2020-11-11 19:05:23 UTC
Red Hat Product Errata RHBA-2016:1871 0 normal SHIPPED_LIVE openstack-heat bug fix advisory 2016-09-14 18:23:30 UTC

Description Matt Flusche 2016-06-22 17:33:44 UTC
Description of problem:

The purge_deleted function of heat-manage does not clean up raw_template entries that are no longer referenced by the stack table (fields: raw_template_id or prev_raw_template_id).

OSP Director deployments creates unreferenced raw_template entries and they continue to build up over time causing this table to potentially consume significant space.

Looking at the code, the current purge_deleted function only operates on raw_template IDs that are being purged from the stack table.

Here is an example SQL that will find these unreferenced entries:

SELECT x_id AS Abandoned_Raw_Template_IDs
FROM
  (SELECT a.id AS x_id
   FROM raw_template a 
   LEFT JOIN stack b 
   ON (a.id = b.raw_template_id)
   WHERE b.raw_template_id IS NULL) AS x,
  (SELECT a.id AS y_id
   FROM raw_template a 
   LEFT JOIN stack b 
   ON (a.id = b.prev_raw_template_id)
   WHERE b.prev_raw_template_id IS NULL) AS y
WHERE x.x_id = y.y_id;


Version-Release number of selected component (if applicable):

openstack-heat-common-5.0.1-6.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. On a fresh OSP-d undercloud; deploy a test overcloud.
2. Delete the overcloud heat stack.
3. Wait >1 min after stack delete completes.
4. Run:  sudo heat-manage purge_deleted -g minutes 1
5. Verify from the database that the stack table is empty but entries remain in the raw_template table:
  select count(1) from stack;
  select count(1) from raw_template;
6. These entries will continue to build up as the Overcloud stack is redeployed or updated.

Actual results:

raw_template table grows over time as Overcloud stack is redeployed or updated.

Expected results:

unreferenced raw_template IDs are purged.

Additional info:

Quick test showing the details:

Prior to the deployment we have zero entries in stack and raw_template table:

$ echo 'select count(1) from stack' |sudo mysql heat
count(1)
0

$ echo 'select count(1) from raw_template' |sudo mysql heat
count(1)
0

Deploy & delete overcloud

$ openstack overcloud deploy \
....

$ heat stack-delete overcloud

Wait at least 1 min

Verify the number of entries in stack and raw_template tables:

$ echo 'select count(1) from stack' |sudo mysql heat
count(1)
168
$ echo 'select count(1) from raw_template' |sudo mysql heat
count(1)
262

Use the SQL from above to get the number of unreferenced raw_template IDs

$ echo 'SELECT count(x_id) AS Abandoned_Raw_IDs_count
FROM
  (SELECT a.id AS x_id
   FROM raw_template a 
   LEFT JOIN stack b 
   ON (a.id = b.raw_template_id)
   WHERE b.raw_template_id IS NULL) AS x,
  (SELECT a.id AS y_id
   FROM raw_template a 
   LEFT JOIN stack b 
   ON (a.id = b.prev_raw_template_id)
   WHERE b.prev_raw_template_id IS NULL) AS y
WHERE x.x_id = y.y_id' |sudo mysql heat

Abandoned_Raw_IDs_count
47

Run the purge_deleted function
 
$ sudo /bin/heat-manage purge_deleted -g minutes 1 

List remaining counts from stack and raw_template tables to show the 47 abandoned entries remain.

$ echo 'select count(1) from stack' |sudo mysql heat
count(1)
0

$ echo 'select count(1) from raw_template' |sudo mysql heat
count(1)
47

Comment 1 Thomas Hervé 2016-06-23 11:46:27 UTC
It may be related to https://bugs.launchpad.net/heat/+bug/1506077 . Unfortunately it wasn't backported yet.

Comment 2 Matt Flusche 2016-06-23 14:57:19 UTC
Yes, I tested that fix and it does resolve the issue; could we backport please?

Comment 3 Thomas Hervé 2016-06-24 13:51:15 UTC
Upstream backport in progress.

Comment 4 Irina Petrova 2016-07-19 10:24:47 UTC
Hey guys,

I have a customer that is asking for the fix.

It looks like the fix already got merged upstream to heat (stable/liberty) on 2016-07-04.

Do we have any information on when it's going to be available in OSP 8?

Best,
Irina

Comment 10 errata-xmlrpc 2016-09-14 14:41:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1871.html


Note You need to log in before you can comment on or make changes to this bug.