Bug 1565460

Summary: cinder-manage db purge fails
Product: Red Hat OpenStack Reporter: Nilesh <nchandek>
Component: openstack-cinderAssignee: Rajat Dhasmana <rdhasman>
Status: CLOSED DUPLICATE QA Contact: Avi Avraham <aavraham>
Severity: high Docs Contact: RHOS Documentation Team <rhos-docs>
Priority: high    
Version: 10.0 (Newton)CC: dciabrin, eharney, jvisser, mbayer, pkesavar, srevivo
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-13 15:58:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1777906, 1794693, 1796144    

Description Nilesh 2018-04-10 04:47:00 UTC
# cinder-manage db purge 300
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
Purge command failed, check cinder-manage logs for more details.

# tail /var/log/cinder/cinder-manage.log
2018-03-29 15:39:37.255 454259 ERROR cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] DBError detected when purging from volumes: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_attachment`, CONSTRAINT `volume_attachment_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))') [SQL: u'DELETE FROM volumes WHERE volumes.deleted_at < %(deleted_at_1)s'] [parameters: {u'deleted_at_1': datetime.datetime(2017, 6, 2, 13, 39, 37, 226381)}].

The bug is "fixed": https://bugs.launchpad.net/cinder/+bug/1542169
unfortunately the bugfix only fixed the error log message and not the actual problem.


* ERROR
	~~~
	2018-03-29 15:39:30.500 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volume_glance_metadata
	2018-03-29 15:39:30.987 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=snapshot_metadata
	2018-03-29 15:39:31.572 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=attachment_specs
	2018-03-29 15:39:32.048 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volume_type_projects
	2018-03-29 15:39:32.517 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volume_type_extra_specs
	2018-03-29 15:39:33.101 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volume_metadata
	2018-03-29 15:39:33.578 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volume_attachment
	2018-03-29 15:39:34.170 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volume_admin_metadata
	2018-03-29 15:39:34.644 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=transfers
	2018-03-29 15:39:35.110 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=snapshots
	2018-03-29 15:39:35.684 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=group_volume_type_mapping
	2018-03-29 15:39:36.187 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=workers
	2018-03-29 15:39:36.766 454259 INFO cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] Purging deleted rows older than age=300 days from table=volumes
	2018-03-29 15:39:37.255 454259 ERROR cinder.db.sqlalchemy.api [req-a5a8a546-c6d5-4646-b202-3bf118711594 - - - - -] DBError detected when purging from volumes: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_attachment`, CONSTRAINT `volume_attachment_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))') [SQL: u'DELETE FROM volumes WHERE volumes.deleted_at < %(deleted_at_1)s'] [parameters: {u'deleted_at_1': datetime.datetime(2017, 6, 2, 13, 39, 37, 226381)}].
	(END)
	~~~

Comment 3 Michael Bayer 2018-04-10 15:34:28 UTC
the issue as described so far seems to be that you are unsatisfied with the use case not being covered from https://bugs.launchpad.net/cinder/+bug/1542169, and you would prefer that they implement option "2. Explicitly delete attachment records" as stated at https://bugs.launchpad.net/cinder/+bug/1542169/comments/1.  This is not a mariadb issue.

Comment 4 Eric Harney 2018-04-10 15:40:03 UTC
It looks like this is or was a bug in Cinder:  the fix for https://bugs.launchpad.net/cinder/+bug/1542169 was not actually a fix, it just generated a better error message.

Comment 12 Eric Harney 2019-04-17 13:27:02 UTC
I think we could fix this by adjusting

https://git.openstack.org/cgit/openstack/cinder/tree/cinder/db/sqlalchemy/api.py?h=14.0.0#n6636

to skip any volumes that still have attachment records.  (Since they aren't supposed to be in that state and it isn't clear what to do with them.)

Issuing a warning for the admin in that case might also be helpful, but not having "cinder-manage" fail would be a good improvement in itself.

Comment 13 Eric Harney 2019-12-13 15:58:48 UTC

*** This bug has been marked as a duplicate of bug 1777906 ***