Bug 1567607

Summary: Cinder backup of an encrypted RBD volume fails
Product: Red Hat OpenStack Reporter: Tzach Shefi <tshefi>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: high Docs Contact: Kim Nylander <knylande>
Priority: medium    
Version: 13.0 (Queens)CC: abishop, eharney, pgrist, srevivo
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost Doc Type: If docs needed, set a value
Doc Text:
Don't think doc update is required because issue was fixed before feature released.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:51:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Cinder backup log, look at towards end of log, as head contains previous attempts wont fit above mentioned IDs none

Description Tzach Shefi 2018-04-15 12:10:48 UTC 
Created attachment 1422164 [details]
Cinder backup log, look at towards end of log, as head contains previous attempts wont fit above mentioned IDs

Description of problem: When I try to backup a none (or attached) RBD backed Cinder volume backup fails. Backup of a none encrypted volume works fine. 

Version-Release number of selected component (if applicable):
rhel 7.5
openstack-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch
puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch
python2-cinderclient-3.5.0-1.el7ost.noarch
python-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch



How reproducible:
Every time

Steps to Reproduce:
1. Create an encrypted RBD backed Cinder volume 

2. Try to backup that volume result back in error state. 

$cinder backup-create 84eae95b-efed-441a-880a-bd35af1aadcd --name BackupOfEncNoneAttVolFails
+-----------+--------------------------------------+
| Property  | Value                                |
+-----------+--------------------------------------+
| id        | fd55218a-f26c-45a0-abd6-03ccabbd448a |
| name      | BackupOfEncNoneAttVolFails           |
| volume_id | 84eae95b-efed-441a-880a-bd35af1aadcd |
+-----------+--------------------------------------+

cinder backup-show fd55218a-f26c-45a0-abd6-03ccabbd448a | grep status
| status                | error           

3.   Backup of a none encrypted volume works fine

 cinder backup-create 9a90e8dd-59c2-445c-961c-e2dd3413289b --force --name WorkingBackupOfNoneEncAttVol
+-----------+--------------------------------------+
| Property  | Value                                |
+-----------+--------------------------------------+
| id        | 781434ee-ade8-4f44-85de-be0cf11b84c4 |
| name      | WorkingBackupOfNoneEncAttVol         |
| volume_id | 9a90e8dd-59c2-445c-961c-e2dd3413289b |
+-----------+--------------------------------------+

$ cinder backup-show 781434ee-ade8-4f44-85de-be0cf11b84c4 | grep status
| status                | available 

Actual results:
Backup should complete 

Expected results:
Backup status error. 

Additional info:

On the same system Archit helped me get Barbican working with Cinder. I was failing to create encrypted volumes this is what what was suggested/done on my system -> 

Barbican is running on your setup on tigris01 now. The issue was that the auth_endpoint in cinder.conf [barbican] section was pointing to a versionless auth url and it was going to ip:5000/v2 while v2 is deprecated in RHOS-13.

This here backup might be more of the same. As backup log trackback also complains about keystone resource not being found. Was about the same problem I got before Archit's fix. 

6
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server [req-70762a72-90d4-42f5-8a60-7b3c160254a4 a2707f862dba4a43a5fb3f5698f0ed64 c662a91a23e44faab47f1b319495a73f - - -] Exception during message handling: NotFound: (http://172.17.1.12:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-0cb44c9a-7a6c-4d74-a313-0eb6a170a0ec)
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server
Comment 6 Tzach Shefi 2018-05-06 07:59:14 UTC 
Verified on:
openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost.noarch


Two encrypted volume attached and none attached:

(overcloud) [stack@undercloud-0 ~]$ cinder list                                                                      
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to              
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------
| 0435c548-b32b-40ee-a4f5-53b7255c55e4 | in-use    | vol4 | 1    | LUKS        | false    | 22e141f4-76d0-4580-a800-0
| c26d8b98-d16f-45bf-b1af-30af8233e689 | available | vol1 | 1    | LUKS        | false    |                          

Backup of unattached enc volume

(overcloud) [stack@undercloud-0 ~]$ cinder backup-create c26d8b98-d16f-45bf-b1af-30af8233e689 --name backupOfEncVol
+-----------+--------------------------------------+
| Property  | Value                                |
+-----------+--------------------------------------+
| id        | c5d6f817-a4dc-4036-a2db-1c6f15a17a53 |
| name      | backupOfEncVol                       |
| volume_id | c26d8b98-d16f-45bf-b1af-30af8233e689 |
+-----------+--------------------------------------+

Backup of attached enc volume
cinder backup-create 0435c548-b32b-40ee-a4f5-53b7255c55e4 --name backupOfEncAttVol --force
+-----------+--------------------------------------+
| Property  | Value                                |
+-----------+--------------------------------------+
| id        | 002ccc8d-4857-4e23-bce5-81ccb863c233 |
| name      | backupOfEncAttVol                    |
| volume_id | 0435c548-b32b-40ee-a4f5-53b7255c55e4 |
+-----------+--------------------------------------+


Both backups created successfully / available, verified. 
cinder backup-list
+--------------------------------------+--------------------------------------+-----------+-------------------+------+--------------+---------------+
| ID                                   | Volume ID                            | Status    | Name              | Size | Object Count | Container     |
+--------------------------------------+--------------------------------------+-----------+-------------------+------+--------------+---------------+
| 002ccc8d-4857-4e23-bce5-81ccb863c233 | 0435c548-b32b-40ee-a4f5-53b7255c55e4 | available | backupOfEncAttVol | 1    | 22           | volumebackups |

| c5d6f817-a4dc-4036-a2db-1c6f15a17a53 | c26d8b98-d16f-45bf-b1af-30af8233e689 | available | backupOfEncVol    | 1    | 22           | volumebackups |
+--------------------------------------+--------------------------------------+-----------+-------------------+------+--------------+---------------+
Comment 9 errata-xmlrpc 2018-06-27 13:51:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086