Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1802263

Summary: [RHOS 16.1] cinder backup-import is removing entries from DB when the volume backup exists.
Product: Red Hat OpenStack Reporter: Darin Sorrentino <dsorrent>
Component: openstack-cinderAssignee: Brian Rosmaita <brian.rosmaita>
Status: CLOSED ERRATA QA Contact: Evelina Shames <eshames>
Severity: medium Docs Contact: Andy Stillman <astillma>
Priority: medium    
Version: 16.0 (Train)CC: brian.rosmaita, jelynch, ltoscano, senrique
Target Milestone: z9Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-15.4.0-1.20221003203219.58f0e73.el8ost Doc Type: Bug Fix
Doc Text:
Before this update, if you imported a backup record for a backup ID that currently existed, the import operation would correctly fail, but the existing backup record would incorrectly be deleted. With this update, the existing backup record is not deleted under this scenario.
 Story Points: ---
Clone Of:
: 2131386 (view as bug list) Environment:
Last Closed: 2022-12-07 20:24:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2131386    
Bug Blocks: 1802255    

Description Darin Sorrentino 2020-02-12 18:16:02 UTC 
Description of problem:

After successfully using cinder backup-import to recover a backup that exists in Ceph but not in the DB, if I re-run the backup-import command a second time, it deletes the backup from the DB.  In the below example, I am trying to import a backup called 'attached-full':

# Show that attached-full does not exist

(overcloud) [stack@director16 ~]$ openstack volume backup list
+--------------------------------------+---------------+-------------+-----------+------+
| ID                                   | Name          | Description | Status    | Size |
+--------------------------------------+---------------+-------------+-----------+------+
| 1b1b2d06-6dc1-4a34-9f3b-7dd852ad3fa6 | None          | None        | available |    1 |
| 6ada70b0-ed1a-47de-9661-37517844cea1 | detached-full | None        | available |    1 |
| 55904715-fe70-4e4b-ac0d-458ceb4c9633 | None          | snap-backup | available |    1 |
| 5ca9ed1b-e719-47fa-bbfd-a57d5f1b877b | None          | None        | available |    1 |
+--------------------------------------+---------------+-------------+-----------+------+

# Import the backup data

(overcloud) [stack@director16 ~]$ cinder backup-import cinder.backup.drivers.ceph.CephBackupDriver eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiODU1YWFhYTUtZTU5MS00M2I2LWJlMjAtYTcxNTBiOWRjOTM0IiwgInVzZXJfaWQiOiAiMWUwM2U0NDVlNWUzNDcyODhiNWViNTMxMjljZWU5ZTkiLCAicHJvamVjdF9pZCI6ICJhY2NkYzk2ZmFjMjU0YjJkOTVhYTNhODY2Yzg5NjliMCIsICJ2b2x1bWVfaWQiOiAiY2VlOWIyZDgtNGJhMS00MmIwLTlmMzYtNjIxZjFmYTk3YTgwIiwgImhvc3QiOiAib3ZlcmNsb3VkLWNvbnRyb2xsZXItMCIsICJhdmFpbGFiaWxpdHlfem9uZSI6ICJub3ZhIiwgImNvbnRhaW5lciI6ICJiYWNrdXBzIiwgInBhcmVudF9pZCI6IG51bGwsICJwYXJlbnQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJhdHRhY2hlZC1mdWxsIiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLWNlZTliMmQ4LTRiYTEtNDJiMC05ZjM2LTYyMWYxZmE5N2E4MC5iYWNrdXAuODU1YWFhYTUtZTU5MS00M2I2LWJlMjAtYTcxNTBiOWRjOTM0XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMC0wMi0xMlQxNzowMjozOVoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMC0wMi0xMlQxNzowMjozOVoiLCAidXBkYXRlZF9hdCI6ICIyMDIwLTAyLTEyVDE3OjAyOjU1WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| id       | 855aaaa5-e591-43b6-be20-a7150b9dc934 |
| name     | attached-full                        |
+----------+--------------------------------------+

# Show backup attached-full exists

(overcloud) [stack@director16 ~]$ openstack volume backup list
+--------------------------------------+---------------+-------------+-----------+------+
| ID                                   | Name          | Description | Status    | Size |
+--------------------------------------+---------------+-------------+-----------+------+
| 1b1b2d06-6dc1-4a34-9f3b-7dd852ad3fa6 | None          | None        | available |    1 |
| 855aaaa5-e591-43b6-be20-a7150b9dc934 | attached-full | None        | available |    1 |
| 6ada70b0-ed1a-47de-9661-37517844cea1 | detached-full | None        | available |    1 |
| 55904715-fe70-4e4b-ac0d-458ceb4c9633 | None          | snap-backup | available |    1 |
| 5ca9ed1b-e719-47fa-bbfd-a57d5f1b877b | None          | None        | available |    1 |
+--------------------------------------+---------------+-------------+-----------+------+

# Re-run the import which fails, correctly stating that the backup already exists

(overcloud) [stack@director16 ~]$ cinder backup-import cinder.backup.drivers.ceph.CephBackupDriver eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiODU1YWFhYTUtZTU5MS00M2I2LWJlMjAtYTcxNTBiOWRjOTM0IiwgInVzZXJfaWQiOiAiMWUwM2U0NDVlNWUzNDcyODhiNWViNTMxMjljZWU5ZTkiLCAicHJvamVjdF9pZCI6ICJhY2NkYzk2ZmFjMjU0YjJkOTVhYTNhODY2Yzg5NjliMCIsICJ2b2x1bWVfaWQiOiAiY2VlOWIyZDgtNGJhMS00MmIwLTlmMzYtNjIxZjFmYTk3YTgwIiwgImhvc3QiOiAib3ZlcmNsb3VkLWNvbnRyb2xsZXItMCIsICJhdmFpbGFiaWxpdHlfem9uZSI6ICJub3ZhIiwgImNvbnRhaW5lciI6ICJiYWNrdXBzIiwgInBhcmVudF9pZCI6IG51bGwsICJwYXJlbnQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJhdHRhY2hlZC1mdWxsIiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLWNlZTliMmQ4LTRiYTEtNDJiMC05ZjM2LTYyMWYxZmE5N2E4MC5iYWNrdXAuODU1YWFhYTUtZTU5MS00M2I2LWJlMjAtYTcxNTBiOWRjOTM0XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMC0wMi0xMlQxNzowMjozOVoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMC0wMi0xMlQxNzowMjozOVoiLCAidXBkYXRlZF9hdCI6ICIyMDIwLTAyLTEyVDE3OjAyOjU1WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=
ERROR: Invalid backup: Backup already exists in database. (HTTP 400) (Request-ID: req-6fa32aec-2c21-41f9-8845-c52d303771b5)

# Backup is now gone:

(overcloud) [stack@director16 ~]$ openstack volume backup list
+--------------------------------------+---------------+-------------+-----------+------+
| ID                                   | Name          | Description | Status    | Size |
+--------------------------------------+---------------+-------------+-----------+------+
| 1b1b2d06-6dc1-4a34-9f3b-7dd852ad3fa6 | None          | None        | available |    1 |
| 6ada70b0-ed1a-47de-9661-37517844cea1 | detached-full | None        | available |    1 |
| 55904715-fe70-4e4b-ac0d-458ceb4c9633 | None          | snap-backup | available |    1 |
| 5ca9ed1b-e719-47fa-bbfd-a57d5f1b877b | None          | None        | available |    1 |
+--------------------------------------+---------------+-------------+-----------+------+
(overcloud) [stack@director16 ~]$ 

 

Version-Release number of selected component (if applicable):

16

How reproducible:

To create the scenario in order to test the backup export/import:

Create backup of a volume, note backup ID
Export the DB data using cinder backup-export
SSH to overcloud controller
Use podman to enter the galera container
Use mysql to access the cinder database
Delete from cinder.backup where id='<backup ID>'
Exit back out to Director
Confirm the backup is gone by listing them and perform the import as detailed above.


Actual results:
Expected results:

Backup restored in database.

Additional info:
Comment 1 Luigi Toscano 2020-02-13 14:02:59 UTC 
(In reply to Darin Sorrentino from comment #0)

> 
> To create the scenario in order to test the backup export/import:

Isn't the scenario already described by the steps reported above, with all the openstack commands? If it is correct, what is the purpose of the following steps?

> 
> Create backup of a volume, note backup ID
> Export the DB data using cinder backup-export
> SSH to overcloud controller
> Use podman to enter the galera container
> Use mysql to access the cinder database
> Delete from cinder.backup where id='<backup ID>'
> Exit back out to Director
> Confirm the backup is gone by listing them and perform the import as
> detailed above.

Also, does it work if you use the "cinder" instead of openstack volume?
Comment 2 Darin Sorrentino 2020-02-17 20:38:08 UTC 
Luigi,
  No, it's not.  As far as I read, the purpose of the cinder backup is to restore the DB entries in the event of a catastrophic loss of data in the DB.  In order to "imitate a catastrophic loss of data", one must actually go into the DB and delete the record, which is the steps I provided:

> Create backup of a volume, note backup ID
> Export the DB data using cinder backup-export
> SSH to overcloud controller
> Use podman to enter the galera container
> Use mysql to access the cinder database
> Delete from cinder.backup where id='<backup ID>'
> Exit back out to Director
> Confirm the backup is gone by listing them and perform the import as
> detailed above.

Once you do that, THEN you can use the cinder backup restore procedure to restore the volume to the DB as detailed in the BZ description.

The import/export commands are "cinder" commands and not "openstack" commands.

Are you asking if "cinder list" has a different output from "openstack volume list"?  If that is what you're asking, no, it does not. Both show the same information.
Comment 3 Luigi Toscano 2020-02-28 11:28:51 UTC 
Thanks. Reproduced.
Comment 6 Brian Rosmaita 2022-10-03 21:04:41 UTC 
Fix is present in openstack-cinder-15.4.0-1.20221003203219.58f0e73.el8ost, which has the 'rhos-16.1-rhel-8-trunk-candidate' tag:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2192209
Comment 20 errata-xmlrpc 2022-12-07 20:24:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795