Bug 1568004 - FFU: fast_forward_upgrade_playbook.yaml fails while running gnocchi-upgrade: cannot delete or update a parent row: a foreign key constraint fails (`gnocchi`.`resource_history`, CONSTRAINT `fk_rh_id_resource_id` FOREIGN KEY (`id`)
Summary: FFU: fast_forward_upgrade_playbook.yaml fails while running gnocchi-upgrade: ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-gnocchi
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: zstream
: 11.0 (Ocata)
Assignee: Julien Danjou
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-16 14:57 UTC by Marius Cornea
Modified: 2018-06-27 20:13 UTC (History)
14 users (show)

Fixed In Version: openstack-gnocchi-3.1.16-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 20:12:59 UTC
Target Upstream Version:


Attachments (Terms of Use)
gnocchi db dump at failure time (63.51 KB, text/x-vhdl)
2018-04-23 17:35 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github gnocchixyz gnocchi pull 879 0 None None None 2018-04-24 15:23:36 UTC
Red Hat Product Errata RHBA-2018:2098 0 None None None 2018-06-27 20:13:05 UTC

Description Marius Cornea 2018-04-16 14:57:39 UTC
Description of problem:
FFU: fast_forward_upgrade_playbook.yaml fails while running gnocchi-upgrade: 

2018-04-15 08:23:59.510 607232 ERROR gnocchi DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`gnocchi`.`resource_history`, CONSTRAINT `fk_rh_id_resource_id` FOREIGN KEY (`id`) REFERENCES `resource` (`id`) ON DELETE CASCADE)') [SQL: u'UPDATE resource SET id=%(param_1)s, original_resource_id=%(original_resource_id)s WHERE resource.id = %(id_1)s'] [parameters: {'original_resource_id': 'instance-00000051-7d482130-b873-4873-b303-2eda83f3ab6a-tapadffcf31-e3', u'id_1': bytearray(b')\x01V\xff7\xcb_\x06\xb9K;kQ\x8bC\xfd'), u'param_1': 'H\xff\xde\xe9\x1c|^\xe3\x81\x8e\xe8\xde)\xd9]l'}]

Note: the initial deployment was OSP10 z4 which was later updated to latest OSP10 on RHEL 7.5

Version-Release number of selected component (if applicable):
openstack-gnocchi-statsd-3.1.12-1.el7ost.noarch
openstack-gnocchi-common-3.1.12-1.el7ost.noarch
python-gnocchi-3.1.12-1.el7ost.noarch
openstack-gnocchi-metricd-3.1.12-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-3.1.12-1.el7ost.noarch
puppet-gnocchi-9.5.0-3.el7ost.noarch
python-gnocchiclient-2.8.2-2.el7ost.noarch
openstack-gnocchi-api-3.1.12-1.el7ost.noarch


How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 Z4
2. Minor update to OSP10 latest
3. Upgrade undercloud to OSP11/12/13
4. Update stack outputs
5. Run fast_forward_upgrade_playbook.yaml

Actual results:
Failure while running /bin/gnocchi-upgrade at OSP11.

Expected results:
No failure.

Additional info:
The issue is not reproducing when starting from OSP10 latest for initial deployment.

Comment 2 Marius Cornea 2018-04-23 17:35:57 UTC
Created attachment 1425706 [details]
gnocchi db dump at failure time

Attaching gnocchi db dump at failure time.

Comment 3 Marius Cornea 2018-04-23 17:37:00 UTC
Note: if I try re-running /bin/gnocchi-upgrade after the failure I get a different failure in /var/log/gnocchi/gnocchi-upgrade.log:

2018-04-23 17:33:04.392 85915 ERROR gnocchi DBNonExistentConstraint: (pymysql.err.InternalError) (1025, u"Error on rename of './gnocchi/metric' to './gnocchi/#sql2-199d-22d14' (errno: 152)") [SQL: u'ALTER TABLE metric DROP FOREIGN KEY fk_metric_resource_id_resource_id']

Comment 5 Julien Danjou 2018-04-23 18:26:33 UTC
(In reply to Marius Cornea from comment #3)
> Note: if I try re-running /bin/gnocchi-upgrade after the failure I get a
> different failure in /var/log/gnocchi/gnocchi-upgrade.log:
> 
> 2018-04-23 17:33:04.392 85915 ERROR gnocchi DBNonExistentConstraint:
> (pymysql.err.InternalError) (1025, u"Error on rename of './gnocchi/metric'
> to './gnocchi/#sql2-199d-22d14' (errno: 152)") [SQL: u'ALTER TABLE metric
> DROP FOREIGN KEY fk_metric_resource_id_resource_id']

That's normal: MySQL DDL is not transactional, so when the migration script fails, there's no way to rollback. That leaves the migration in a invalid state, and it's a mess to recover. Blame MySQL.

Comment 6 Julien Danjou 2018-04-23 18:33:09 UTC
The problem seems to be that resource_history.id foreign key constraint on resource.id (fk_rh_id_resource_id) is not dropped by the migration script `397987e38570_no_more_slash_and_reencode.py` and therefore the update on the new id is blocked.

This probably has not been tested neither upstream/downstream, as I imagine what has been tested was resources without history.

Comment 17 errata-xmlrpc 2018-06-27 20:12:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2098


Note You need to log in before you can comment on or make changes to this bug.