Bug 1306444 - DB Purge of raw_template fails on constraint from resources.current_template_id
DB Purge of raw_template fails on constraint from resources.current_template_id
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: z4
: 7.0 (Kilo)
Assigned To: Crag Wolfe
Amit Ugol
: ZStream
Depends On:
Blocks: 1310890
  Show dependency treegraph
Reported: 2016-02-10 16:37 EST by Crag Wolfe
Modified: 2016-04-26 14:00 EDT (History)
7 users (show)

See Also:
Fixed In Version: openstack-heat-2015.1.2-9.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the "heat-manage purge_delete" cron job on the undercloud failed when attempting to purge the raw_template table. This patch fixes raw_template purge query on MySQL, and handles stack tags before removing stacks which resolves the issue. This also removes a bunch of race conditions where incorrect data was being deleted.
Story Points: ---
Clone Of:
: 1310890 (view as bug list)
Last Closed: 2016-02-18 11:43:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 278466 None None None 2016-02-10 16:58 EST

  None (edit)
Description Crag Wolfe 2016-02-10 16:37:43 EST
Description of problem:

The "heat-manage purge_delete" cron job on the undercloud fails when attempting to purge the raw_template table.

Version-Release number of selected component (if applicable):

How reproducible:
Not clear.

Steps to Reproduce:
1. Run /bin/heat-manage purge_deleted manually

Actual results:
A stack trace and the raw_template table is left un-purged.

Expected results:
A purged raw_template table.
Comment 2 Zane Bitter 2016-02-11 12:53:48 EST
Since DB migrations convergence were introduced in Kilo, we've been storing a reference to the raw_template for each resource in the current_template_id column of the resource table. It appears that prior to the fix for https://bugs.launchpad.net/heat/+bug/1524387 we were deleting too many raw_template entries - i.e. ones that might have been referenced by prev_raw_template_id in a stack, or current_template_id in a resource, resulting in this error:

ERROR: (_mysql_exceptions.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`resource`, CONSTRAINT `current_template_fkey_ref` FOREIGN KEY (`current_template_id`) REFERENCES `raw_template` (`id`))') [SQL: u'DELETE FROM raw_template WHERE raw_template.id NOT IN (SELECT stack.raw_template_id \nFROM stack)']

The patch avoids deleting raw_template entries that are still referenced by prev_raw_template_id, but that field is not actually populated in Kilo (it was also introduced for convergence, although we also use it for other purposes in the legacy code path in Liberty). So it's unlikely this is having an effect.

I think the real reason it appears to work is that it now cleans up only raw_templates from the stacks that it had just deleted... and in this case all of the old *stacks* had already been removed by the previous command (which later failed attempting to clean up non-orphaned raw_templates). So we'll never be able to clean up those templates with this code.
Comment 3 Zane Bitter 2016-02-11 13:04:52 EST
That said, the fact that we were trying to clean up all non-referenced raw_templates (not just the ones that were previously referenced by the stacks we just deleted) and doing an inadequate check for existing references was probably the reason for the error in the first place. So the check to only delete raw_templates associated with the stacks we just removed is probably a correct fix. But this at least explains why none of the old raw_templates were actually removed when purge_deleted was run with the patch, even though there was no more error.
Comment 5 Amit Ugol 2016-02-16 05:29:54 EST
no issues running it manually now.
Comment 7 errata-xmlrpc 2016-02-18 11:43:40 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.