Bug 1722332
| Summary: | [RHOS15-CEPH] Unable to create cinder incremental backup | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bkopilov <bkopilov> | ||||
| Component: | openstack-cinder | Assignee: | Sofia Enriquez <senrique> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | ||||
| Severity: | urgent | Docs Contact: | Tana <tberry> | ||||
| Priority: | high | ||||||
| Version: | 15.0 (Stein) | CC: | abishop, pgrist, senrique, tdados, tshefi | ||||
| Target Milestone: | z2 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 15.0 (Stein) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-cinder-14.0.3-0.20191203220448.ea67131.el8ost | Doc Type: | No Doc Update | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-03-05 11:54:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
bkopilov
2019-06-20 05:37:00 UTC
Created attachment 1582521 [details]
controller0 logs
This is very strange. Something low-level bombs out as soon as the incremental backup is initiated (signal 6 is SIGABRT): 2019-06-20 05:33:03.913 9271 DEBUG cinder.backup.drivers.ceph [req-f8240da3-bdad-442b-8de8-29231f716951 4905e8e0d2d540c79619376e40e22764 84000d76c19f4307a0fe1722f7c350a1 - default default] Starting backup of volume='edbf9b37-1ac4-4253-9aef-6c07eec9d98f'. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:935 2019-06-20 05:33:03.914 9271 DEBUG cinder.backup.drivers.ceph [req-f8240da3-bdad-442b-8de8-29231f716951 4905e8e0d2d540c79619376e40e22764 84000d76c19f4307a0fe1722f7c350a1 - default default] Volume file is RBD: attempting incremental backup. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:944 2019-06-20 05:33:05.934 8 INFO oslo_service.service [req-16c3a67b-2ef3-4a8f-841f-f6d12c229473 - - - - -] Child 9271 killed by signal 6 @Sofi, can you take a look at this? @Benny, we may need to you leave us a system on which we can debug this @Benny, I assume this is covered by automation. Are there other runs that show a similar failure? Hello chaps, I also experience the same issue with cinder-backup incremental using ceph as backend. Have the same log output as above and also when you do "openstack volume backup show" it mentions: | fail_reason | incomplete backup reset on manager restart | Any hints, appreciated During the incremental backups creation, unexpectedly the '_cleanup_incomplete_backup_operations' method is called aborting the backup: DEBUG cinder.backup.drivers.ceph Volume file is RBD: attempting incremental backup. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:955 INFO cinder.service [-] Starting cinder-backup node (version 14.0.1) DEBUG oslo_concurrency.lockutils Lock "cleanup_incomplete_backups_1" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:327 INFO cinder.backup.manager Cleaning up incomplete backup operations. This happens when using ceph mimic with Python 3.x. The connections with Librbd[1] suddenly abort connections before accessing the data. [2] How to reproduce: >>> import rbd >>> import rados >>> cluster = rados.Rados(conffile='/etc/ceph/ceph.conf') >>> cluster.connect() >>> ioctx = cluster.open_ioctx('backups') >>> image = rbd.Image(ioctx, 'volume-2d2c563d-c567-431a-9b25-ea06af83c598.backup.base') >>> snaps = image.list_snaps() >>> image.close() >>> list(snaps) Aborted (core dumped) Expected output: snapshots list >>> list(snaps) [{'id': 4, 'size': 1073741824, 'name': 'backup.0dbaab38-595e-4e8a-8ecd-7ab60a879903.snap.1563912753.3520913', 'namespace': 0}, {'id': 5, 'size': 1073741824, 'name': 'backup.c9aaaf88-9ce1-4502-aa23-1dddfb1d45bb.snap.1564609930.8021717', 'namespace': 0}] Actual output: core dumped [1] https://docs.ceph.com/docs/giant/rbd/librbdpy/ [2] https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/drivers/ceph.py#L649 Fix merged upstream stable/stein : https://review.opendev.org/#/c/676195/. Sofia, Do you happen to know which fixed-in version includes this fix? Sorry Tzach, I added the fixed-in version. P1 -> "RHOS_TRUNK-15.0-RHEL-8-20191115.n.0". still provided a pre-fixed-in version 14.0.3 > openstack-cinder-14.0.2-0.20190812230446.a71fa76.el8ost.noarch Can't verify just yet, waiting for a newer build. Build date: RHOS_TRUNK-15.0-RHEL-8-20191212.n.0 Still produced a pre-fixed-in 14.0.3-0.201911... < 14.0.3-0.201912... Waiting for a newer build. Verified on: openstack-cinder-14.0.3-0.20191214220454.1d8efc5.el8ost.noarch This passed automation cinder incremental tests. Also ran manual testing: Create volA write data, create full backup of volA. Check that volume backup is of type full (is_incremental false). Restore backup to new volB compare data. Write more data on VolA Created an incremental backup (--incremental) Verify that resulting type is incremental backup (is_incremental true) Restored incremental backup to VolC Compare VolA and VolC Create a third "full" backup, without issuing "--incremental" As this is ceph backed despite requesting a "full backup" resulting backup is of type incremental see [0] When I check result as expected (is_incremental true) Restore this last back to volD Compare volA with VolD [0] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/block_storage_backup_guide/index#doing-incremental-backups-using-backup-service Ceph backups currently ignore the --incremental option: Ceph backups always try to perform incremental backups when the source is a Ceph volume. Looks good to verify. If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0712 |