Bug 1468753 - cinder backup is_incremental flag is True even if incremental backup fails [NEEDINFO]
cinder backup is_incremental flag is True even if incremental backup fails
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
medium Severity medium
: ---
: 12.0 (Pike)
Assigned To: Chaynika Saikia
Avi Avraham
: TestOnly, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-07 16:13 EDT by Chaynika Saikia
Modified: 2018-02-15 17:55 EST (History)
7 users (show)

See Also:
Fixed In Version: openstack-cinder-11.0.1-1.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the cinder backup service did not verify whether incremental backup completed successfully, and did not update the is_incremental and the parent_id flags correctly. This means that the service did not report when an incremental backup or full backup was completed. This fix adds verification to the incremental backup process, and now the Ceph RBD backup updates the flags and reports the state and type of backup correctly.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-01-30 15:13:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
tshefi: needinfo? (csaikia)


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1703013 None None None 2018-01-25 11:22 EST
OpenStack gerrit 483089 None master: MERGED cinder: cinder backup sets incorrect parent_id (I22deff68d9376c2ecc88bd2d9d0e1b445d793123) 2018-01-25 15:26 EST

  None (edit)
Description Chaynika Saikia 2017-07-07 16:13:00 EDT
Description of problem:

cinder incremental backup sets is_incremental as True even if it does a forceful full backup

[csaikia@localhost ~]$ cinder backup-create ebf16d3c-deb8-4168-9dfd-50d386860d42 --force --incremental
+-----------+--------------------------------------+
| Property  | Value                                |
+-----------+--------------------------------------+
| id        | 7f27f402-31cb-49fc-ad75-a52d6a5252aa |
| name      | None                                 |
| volume_id | ebf16d3c-deb8-4168-9dfd-50d386860d42 |
+-----------+--------------------------------------+


$ cinder backup-show 7f27f402-31cb-49fc-ad75-a52d6a5252aa
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| availability_zone     | nova                                 |
| container             | backups                              |
| created_at            | 2017-07-07T19:49:26.000000           |
| data_timestamp        | 2017-07-07T19:49:26.000000           |
| description           | None                                 |
| fail_reason           | None                                 |
| has_dependent_backups | False                                |
| id                    | 7f27f402-31cb-49fc-ad75-a52d6a5252aa |
| is_incremental        | True                                 |
| name                  | None                                 |
| object_count          | 0                                    |
| size                  | 1                                    |
| snapshot_id           | None                                 |
| status                | available                            |
| updated_at            | 2017-07-07T19:50:06.000000           |
| volume_id             | ebf16d3c-deb8-4168-9dfd-50d386860d42 |
+-----------------------+--------------------------------------+

While in the logs, one can see that there is a full backup done for that backup ID.

9206 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Forcing full backup of volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) backup /opt/stack/     cinder/cinder/backup/drivers/ceph.py:905}}
9207 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Creating backup base image='volume-ebf16d3c-deb8-4168-9dfd-50d386860d42.backup.7f27f402-31cb-49fc-ad     75-a52d6a5252aa' for volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _full_backup /opt/stack/cinder/cinder/backup/d     rivers/ceph.py:736}}
9208 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Copying data from volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _full_backup /opt/stack     /cinder/cinder/backup/drivers/ceph.py:745}}
9209 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferring data between 'volume-ebf16d3c-deb8-4168-9dfd-50d386860d42' and 'volume-ebf16d3c-deb8-41     68-9dfd-50d386860d42.backup.7f27f402-31cb-49fc-ad75-a52d6a5252aa' {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup     /drivers/ceph.py:338}}
9210 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] 8 chunks of 134217728 bytes to be transferred {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/     backup/drivers/ceph.py:342}}
9211 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 1 of 8 (286576K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dr     ivers/ceph.py:364}}
9212 Jul 07 15:49:32 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 2 of 8 (24954K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9213 Jul 07 15:49:37 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 3 of 8 (25152K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9214 Jul 07 15:49:43 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 4 of 8 (25191K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9215 Jul 07 15:49:48 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 5 of 8 (25293K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9216 Jul 07 15:49:53 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 6 of 8 (24709K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9217 Jul 07 15:49:58 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 7 of 8 (25041K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9218 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Transferred chunk 8 of 8 (24935K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri     vers/ceph.py:364}}
9219 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] Getting metadata type 'volume-base-metadata' {{(pid=23880) _save_vol_base_meta /opt/stack/cinder/cinder/ba     ckup/driver.py:80}}
9220 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] Completed fetching metadata type 'volume-base-metadata' {{(pid=23880) _save_vol_base_meta /opt/stack/cinde     r/cinder/backup/driver.py:97}}
9221 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] Getting metadata type 'volume-metadata' {{(pid=23880) _save_vol_meta /opt/stack/cinder/cinder/backup/drive     r.py:108}}
9222 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] No metadata type 'volume-metadata' available {{(pid=23880) _save_vol_meta /opt/stack/cinder/cinder/backup/     driver.py:122}}
9223 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] Getting metadata type 'volume-glance-metadata' {{(pid=23880) _save_vol_glance_meta /opt/stack/cinder/cinde     r/backup/driver.py:131}}
9224 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] No metadata type 'volume-glance-metadata' available {{(pid=23880) _save_vol_glance_meta /opt/stack/cinder/     cinder/backup/driver.py:146}}
9225 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Backing up metadata for volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _backup_metadata      /opt/stack/cinder/cinder/backup/drivers/ceph.py:871}}
9226 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Backup '7f27f402-31cb-49fc-ad75-a52d6a5252aa' of volume ebf16d3c-deb8-4168-9dfd-50d386860d42 finishe     d. {{(pid=23880) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:927}}
9227 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG os_brick.initiator.connectors.rbd [None req-e8adeca3-4dc7-42     7f-8a77-78d2280621ed admin None] ==> disconnect_volume: call u"{'connection_properties': {u'secret_type': u'ceph', u'name': u'     volumes/volume-ebf16d3c-deb8-4168-9dfd-50d386860d42', u'encrypted': False, u'keyring': None, u'cluster_name': u'ceph', u'secre     t_uuid': u'***', u'qos_specs': None, u'auth_enabled': True, u'hosts': [u'10.13.57.200'], u'volume_id': u'ebf16d3c-deb8-4168-9d     fd-50d386860d42', u'discard': True, u'access_mode': u'rw', u'auth_username': u'cinder', u'ports': [u'6789']}, 'self': <os_bric     k.initiator.connectors.rbd.RBDConnector object at 0x7fe02831ff50>, 'force': False, 'device_info': {'path': <os_brick.initiator     .linuxrbd.RBDVolumeIOWrapper object at 0x7fdff059c090>}, 'ignore_errors': False}" {{(pid=23880) trace_logging_wrapper /opt/sta     ck/os-brick/os_brick/utils.py:146}}
9228 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG os_brick.initiator.connectors.rbd [None req-e8adeca3-4dc7-42     7f-8a77-78d2280621ed admin None] <== disconnect_volume: return (2ms) None {{(pid=23880) trace_logging_wrapper /opt/stack/os-br     ick/os_brick/utils.py:170}}
9229 Jul 07 15:50:06 localhost.localdomain cinder-backup[23880]: WARNING oslo.service.loopingcall [None req-e8adeca3-4dc7-427f-8a77     -78d2280621ed admin None] Function 'cinder.service.Service.report_state' run outlasted interval by 18.95 sec
9230 Jul 07 15:50:07 localhost.localdomain cinder-backup[23880]: INFO cinder.backup.manager [None req-e8adeca3-4dc7-427f-8a77-78d22     80621ed admin None] Create backup finished. backup: 7f27f402-31cb-49fc-ad75-a52d6a5252aa.  

Version-Release number of selected component (if applicable):
[csaikia@localhost cinder]$ cinder --version
2.2.1
[csaikia@localhost cinder]$ openstack --version
openstack 3.11.0

How reproducible:
Every time

Expected results:
is_incremental flag should be set to False if the full backup is done when the incremental backup fails
Comment 3 Alan Bishop 2018-01-25 11:17:54 EST
Assigning to Chaynika because she fixed this in another upstream patch.
Comment 5 Jon Schlueter 2018-01-25 15:29:09 EST
above mentioned patch is on stable/pike and master and in 11.0.0 tag so would have been part of OSP 12 at GA
Comment 8 Jon Schlueter 2018-01-26 10:50:02 EST
According to our records, this should be resolved by openstack-cinder-11.0.1-1.el7ost.  This build is available now.
Comment 9 Tzach Shefi 2018-01-28 16:09:28 EST
Hi Chaynika, 

Assistance with repo steps, osp12 (ceph backed) with Cinder backup configured. 
Fixed-in included -> openstack-cinder-11.0.1-2.el7ost.noarch

You mentioned on expected results ->  is_incremental flag should be set to False if the full backup is done when the incremental backup fails

How may I cause an incremental backup to fail? 

I mean if I try to create an incremental without a full one before I get an expected error:

$ cinder backup-create b73e9cd2-e196-4934-aceb-9266ae5a1da7 --incremental
ERROR: Invalid backup: No backups available to do an incremental backup. (HTTP 400) (Request-ID: req-45e680ca-7730-437c-a53f-b459f7e1efa5)
Comment 13 errata-xmlrpc 2018-01-30 15:13:01 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0244

Note You need to log in before you can comment on or make changes to this bug.