Bug 1596866

Summary:	Heat database error: 'Cannot delete or update a parent row: a foreign key constraint fails ...
Product:	Red Hat OpenStack	Reporter:	Zane Bitter <zbitter>
Component:	openstack-heat	Assignee:	Zane Bitter <zbitter>
Status:	CLOSED ERRATA	QA Contact:	Ronnie Rasouli <rrasouli>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	astupnik, ebarrera, joflynn, jschluet, lars, mburns, rrasouli, sbaker, shardy, srevivo, therve, zbitter
Target Milestone:	z2	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-heat-10.0.1-2.el7ost	Doc Type:	Bug Fix
Doc Text:	When a stack has more than 1,000 past events associated with it, Heat begins to purge a portion of existing events from the database, with some probability of this occurring each time a new event is created. However, if the stack has had previous updates with convergence disabled (the default on the overcloud prior to OSP10, and on the undercloud prior to OSP13) some events might reference resource properties data from the backup stack. Purging events may fail with a foreign key constraint error: 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))' or 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`resource`, CONSTRAINT `rsrc_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))' This prevents the new event from being stored and the stack update fails as a result. Heat now ignores foreign key constraint errors when attempting to purge events. If this situation occurs, events are not purged until any backup stacks that hold common references have been purged. New events are stored and the operation of the stack continues.	Story Points:	---
Clone Of:	1577874	Environment:
Last Closed:	2018-08-29 16:21:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1577874
Bug Blocks:

Description Zane Bitter 2018-06-29 20:22:33 UTC

+++ This bug was initially created as a clone of Bug #1577874 +++

Description of problem:

This error can be seen in a OSP12 deployment:

2018-05-14 04:38:02Z [overcloud-Controller-irffjxb5zlgh-2-u56vlld5py3g]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-14 04:38:03Z [overcloud-Controller-irffjxb5zlgh.2]: UPDATE_COMPLETE  state changed
2018-05-14 04:38:03Z [overcloud-Controller-irffjxb5zlgh]: UPDATE_FAILED  resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') 
2018-05-14 04:38:04Z [Controller]: UPDATE_FAILED  resources.Controller: resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_prope
2018-05-14 04:42:05Z [Compute]: UPDATE_FAILED  UPDATE aborted
2018-05-14 04:42:05Z [overcloud]: UPDATE_FAILED  resources.Controller: resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_prope

 Stack overcloud UPDATE_FAILED 

overcloud.Controller.0:
  resource_type: OS::TripleO::Controller
  physical_resource_id: 7173f444-fe34-417d-bfeb-c853ee85bc06
  status: UPDATE_FAILED
  status_reason: |
    resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s, %(id_6)s, %(id_7)s, %(id_8)s, %(id_9)s, %(id_10)s)'] [parameters: {u'id_10': 6781, u'id_2': 6785, u'id_3': 7014, u'id_1': 7008, u'id_6': 6962, u'id_7': 6964, u'id_4': 6731, u'id_5': 6736, u'id_8': 6871, u'id_9': 6868}]
overcloud.Compute:
  resource_type: OS::Heat::ResourceGroup
  physical_resource_id: cb7ed374-46dd-4121-a9a5-e982100e3123
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted


It seems like the following upstream bug [1] solved in penstack/heat 9.0.0.0b2    but seen in  openstack-heat-engine-9.0.1-3

[1] https://bugs.launchpad.net/heat/+bug/1681772

Version-Release number of selected component (if applicable):


How reproducible:
Unsure


Steps to Reproduce:
1. Scale up the overcloud
2.
3.

Actual results:
Database error

Expected results:
no database error

Additional info:

--- Additional comment from Zane Bitter on 2018-06-22 23:04:24 EDT ---

I found the root cause:

https://storyboard.openstack.org/#!/story/2002643#comment-90440

This affects all non-convergence stacks in OSP11 and up (although in practice that only means TripleO).

--- Additional comment from Zane Bitter on 2018-06-29 16:19:20 EDT ---

We're in the process of backporting a patch to ignore (but log) the error upstream.

The root cause is that entries in the resource properties data can be shared between events/resources in the current stack and those in the backup stack (for non-covergence stacks, including TripleO prior to OSP13). This will be fully fixed on master.

If anybody encounters this in the meantime, a good workaround to try would be to purge deleted stacks using 'heat-manage purge_deleted', since the unwanted references may be in now-deleted backup stacks. There's no guarantee, however, since the references may in fact be from a current backup stack.

Comment 8 Ronnie Rasouli 2018-08-22 07:25:42 UTC

modified the config.py as

https://review.openstack.org/#/c/577432/13/heat/common/config.py

Scaled out compute node successfully

Comment 10 errata-xmlrpc 2018-08-29 16:21:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2590