Description of problem: FFU: fast_forward_upgrade_playbook.yaml fails while running gnocchi-upgrade: 2018-04-15 08:23:59.510 607232 ERROR gnocchi DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`gnocchi`.`resource_history`, CONSTRAINT `fk_rh_id_resource_id` FOREIGN KEY (`id`) REFERENCES `resource` (`id`) ON DELETE CASCADE)') [SQL: u'UPDATE resource SET id=%(param_1)s, original_resource_id=%(original_resource_id)s WHERE resource.id = %(id_1)s'] [parameters: {'original_resource_id': 'instance-00000051-7d482130-b873-4873-b303-2eda83f3ab6a-tapadffcf31-e3', u'id_1': bytearray(b')\x01V\xff7\xcb_\x06\xb9K;kQ\x8bC\xfd'), u'param_1': 'H\xff\xde\xe9\x1c|^\xe3\x81\x8e\xe8\xde)\xd9]l'}] Note: the initial deployment was OSP10 z4 which was later updated to latest OSP10 on RHEL 7.5 Version-Release number of selected component (if applicable): openstack-gnocchi-statsd-3.1.12-1.el7ost.noarch openstack-gnocchi-common-3.1.12-1.el7ost.noarch python-gnocchi-3.1.12-1.el7ost.noarch openstack-gnocchi-metricd-3.1.12-1.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-3.1.12-1.el7ost.noarch puppet-gnocchi-9.5.0-3.el7ost.noarch python-gnocchiclient-2.8.2-2.el7ost.noarch openstack-gnocchi-api-3.1.12-1.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP10 Z4 2. Minor update to OSP10 latest 3. Upgrade undercloud to OSP11/12/13 4. Update stack outputs 5. Run fast_forward_upgrade_playbook.yaml Actual results: Failure while running /bin/gnocchi-upgrade at OSP11. Expected results: No failure. Additional info: The issue is not reproducing when starting from OSP10 latest for initial deployment.
Created attachment 1425706 [details] gnocchi db dump at failure time Attaching gnocchi db dump at failure time.
Note: if I try re-running /bin/gnocchi-upgrade after the failure I get a different failure in /var/log/gnocchi/gnocchi-upgrade.log: 2018-04-23 17:33:04.392 85915 ERROR gnocchi DBNonExistentConstraint: (pymysql.err.InternalError) (1025, u"Error on rename of './gnocchi/metric' to './gnocchi/#sql2-199d-22d14' (errno: 152)") [SQL: u'ALTER TABLE metric DROP FOREIGN KEY fk_metric_resource_id_resource_id']
(In reply to Marius Cornea from comment #3) > Note: if I try re-running /bin/gnocchi-upgrade after the failure I get a > different failure in /var/log/gnocchi/gnocchi-upgrade.log: > > 2018-04-23 17:33:04.392 85915 ERROR gnocchi DBNonExistentConstraint: > (pymysql.err.InternalError) (1025, u"Error on rename of './gnocchi/metric' > to './gnocchi/#sql2-199d-22d14' (errno: 152)") [SQL: u'ALTER TABLE metric > DROP FOREIGN KEY fk_metric_resource_id_resource_id'] That's normal: MySQL DDL is not transactional, so when the migration script fails, there's no way to rollback. That leaves the migration in a invalid state, and it's a mess to recover. Blame MySQL.
The problem seems to be that resource_history.id foreign key constraint on resource.id (fk_rh_id_resource_id) is not dropped by the migration script `397987e38570_no_more_slash_and_reencode.py` and therefore the update on the new id is blocked. This probably has not been tested neither upstream/downstream, as I imagine what has been tested was resources without history.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2098