Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1722332

Summary: [RHOS15-CEPH] Unable to create cinder incremental backup
Product: Red Hat OpenStack Reporter: bkopilov <bkopilov>
Component: openstack-cinderAssignee: Sofia Enriquez <senrique>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: urgent Docs Contact: Tana <tberry>
Priority: high    
Version: 15.0 (Stein)CC: abishop, pgrist, senrique, tdados, tshefi
Target Milestone: z2Keywords: Triaged, ZStream
Target Release: 15.0 (Stein)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-cinder-14.0.3-0.20191203220448.ea67131.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-05 11:54:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
controller0 logs none

Description bkopilov 2019-06-20 05:37:00 UTC
Description of problem:
openstack setup with ceph as backend for nova, glance and cinder.
rhos15.

Unable to create an incremental backup .

Create a volume - pass
Create a backup from the volume - pass
Create an incremental backup from the volume  - Fails.

Attaching logs.

Thanks, 
Benny

Comment 1 bkopilov 2019-06-20 05:41:42 UTC
Created attachment 1582521 [details]
controller0 logs

Comment 2 Alan Bishop 2019-06-25 14:15:16 UTC
This is very strange. Something low-level bombs out as soon as the incremental backup is initiated (signal 6 is SIGABRT):

2019-06-20 05:33:03.913 9271 DEBUG cinder.backup.drivers.ceph [req-f8240da3-bdad-442b-8de8-29231f716951 4905e8e0d2d540c79619376e40e22764 84000d76c19f4307a0fe1722f7c350a1 - default default] Starting backup of volume='edbf9b37-1ac4-4253-9aef-6c07eec9d98f'. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:935
2019-06-20 05:33:03.914 9271 DEBUG cinder.backup.drivers.ceph [req-f8240da3-bdad-442b-8de8-29231f716951 4905e8e0d2d540c79619376e40e22764 84000d76c19f4307a0fe1722f7c350a1 - default default] Volume file is RBD: attempting incremental backup. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:944
2019-06-20 05:33:05.934 8 INFO oslo_service.service [req-16c3a67b-2ef3-4a8f-841f-f6d12c229473 - - - - -] Child 9271 killed by signal 6

@Sofi, can you take a look at this?
@Benny, we may need to you leave us a system on which we can debug this

Comment 8 Alan Bishop 2019-07-02 13:14:47 UTC
@Benny, I assume this is covered by automation. Are there other runs that show a similar failure?

Comment 16 Anastasios 2019-08-02 10:51:40 UTC
Hello chaps,
I also experience the same issue with cinder-backup incremental using ceph as backend. Have the same log output as above and also when you do "openstack volume backup show" it mentions:
| fail_reason           | incomplete backup reset on manager restart |

Any hints, appreciated

Comment 17 Sofia Enriquez 2019-08-02 14:43:18 UTC
During the incremental backups creation, unexpectedly the '_cleanup_incomplete_backup_operations' method is called aborting the backup:

DEBUG cinder.backup.drivers.ceph Volume file is RBD: attempting incremental backup. backup /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:955

INFO cinder.service [-] Starting cinder-backup node (version 14.0.1)

DEBUG oslo_concurrency.lockutils Lock "cleanup_incomplete_backups_1" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:327

INFO cinder.backup.manager Cleaning up incomplete backup operations.

This happens when using ceph mimic with Python 3.x. The connections with Librbd[1] suddenly abort connections before accessing the data. [2]

How to reproduce:

>>> import rbd
>>> import rados
>>> cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
>>> cluster.connect()
>>> ioctx = cluster.open_ioctx('backups')
>>> image = rbd.Image(ioctx, 'volume-2d2c563d-c567-431a-9b25-ea06af83c598.backup.base')
>>> snaps = image.list_snaps()
>>> image.close()
>>> list(snaps)
Aborted (core dumped)

Expected output: snapshots list
>>> list(snaps)
[{'id': 4, 'size': 1073741824, 'name': 'backup.0dbaab38-595e-4e8a-8ecd-7ab60a879903.snap.1563912753.3520913', 'namespace': 0}, {'id': 5, 'size': 1073741824, 'name': 'backup.c9aaaf88-9ce1-4502-aa23-1dddfb1d45bb.snap.1564609930.8021717', 'namespace': 0}]

Actual output: core dumped

[1] https://docs.ceph.com/docs/giant/rbd/librbdpy/
[2] https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/drivers/ceph.py#L649

Comment 18 Sofia Enriquez 2019-08-23 19:43:58 UTC
Fix merged upstream stable/stein : https://review.opendev.org/#/c/676195/.

Comment 20 Tzach Shefi 2019-12-04 13:54:22 UTC
Sofia, 
Do you happen to know which fixed-in version includes this fix?

Comment 21 Sofia Enriquez 2019-12-04 16:57:27 UTC
Sorry Tzach, I added the fixed-in version.

Comment 22 Tzach Shefi 2019-12-10 10:20:06 UTC
P1 -> "RHOS_TRUNK-15.0-RHEL-8-20191115.n.0". 
still provided a pre-fixed-in version 14.0.3 > openstack-cinder-14.0.2-0.20190812230446.a71fa76.el8ost.noarch

Can't verify just yet, waiting for a newer build.

Comment 24 Tzach Shefi 2019-12-16 09:49:12 UTC
Build date:
RHOS_TRUNK-15.0-RHEL-8-20191212.n.0

Still produced a pre-fixed-in 
14.0.3-0.201911...  < 14.0.3-0.201912...

Waiting for a newer build.

Comment 25 Tzach Shefi 2019-12-26 08:35:31 UTC
Verified on:
openstack-cinder-14.0.3-0.20191214220454.1d8efc5.el8ost.noarch

This passed automation cinder incremental tests. 

Also ran manual testing:
Create volA  write data, create full backup of volA.
Check that volume backup is of type full (is_incremental false).
Restore backup to new volB compare data. 

Write more data on VolA
Created an incremental backup (--incremental)
Verify that resulting type is incremental backup (is_incremental true)
Restored incremental backup to VolC
Compare VolA and VolC 

Create a third "full" backup, without issuing "--incremental" 
As this is ceph backed despite requesting a "full backup" resulting backup is of type incremental see [0]
When I check result as expected (is_incremental true)

Restore this last back to volD
Compare volA with VolD

[0]
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/block_storage_backup_guide/index#doing-incremental-backups-using-backup-service 
Ceph backups currently ignore the --incremental option: Ceph backups always try to perform incremental backups when the source is a Ceph volume. 


Looks good to verify.

Comment 26 Alex McLeod 2020-02-19 12:48:56 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.

Comment 28 errata-xmlrpc 2020-03-05 11:54:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0712