Bug 1567614 - Reattaching an encrypted(Barbican) Cinder (RBD) volume to an instance fails
Summary: Reattaching an encrypted(Barbican) Cinder (RBD) volume to an instance fails
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-barbican
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Linux
medium
high
Target Milestone: ---
: ---
Assignee: Ade Lee
QA Contact: Prasanth Anbalagan
URL:
Whiteboard:
Depends On:
Blocks: 1230405
TreeView+ depends on / blocked
 
Reported: 2018-04-15 13:27 UTC by Tzach Shefi
Modified: 2019-09-09 13:15 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-11 18:58:45 UTC
Target Upstream Version:


Attachments (Terms of Use)
Nova compute log (1.65 MB, text/plain)
2018-04-15 13:27 UTC, Tzach Shefi
no flags Details
Requested logs (2.06 MB, application/zip)
2018-04-17 06:13 UTC, Tzach Shefi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1764125 0 None None None 2018-04-15 13:52:35 UTC

Description Tzach Shefi 2018-04-15 13:27:11 UTC
Created attachment 1422192 [details]
Nova compute log

Description of problem: An attached encrypted (Barbican) RBD Cinder volume was attached to instance, write data to it. 
Then volume was detached, when trying to reattach the volume to same instance volume fails to attach. Odd errors on attached nova-compute.log 

2018-04-15 13:14:06.274 1 ERROR nova.compute.manager [instance: 923c5318-8502-4f85-a215-78afc4fd641b]     uuid=managed_object_id)
2018-04-15 13:14:06.274 1 ERROR nova.compute.manager [instance: 923c5318-8502-4f85-a215-78afc4fd641b] ManagedObjectNotFoundError: Key not found, uuid: 7912eac8-2652-4c92-b53f-3db4ecca7bc7

2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment c17e2b89-5a36-4e7e-8c71-b975f2f5ccb3 could not be found.
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server 



Version-Release number of selected component (if applicable):
rhel7.5 


How reproducible:
Unsure look every time I try

Steps to Reproduce:
1. Boot an instance
2. Create an encrypted(Barbican) backed Cinder(RBD) volume, attach to instance write data.  
3. Detach volume from instance
4. Try to reattach same volume to same instance. 

$nova volume-attach 923c5318-8502-4f85-a215-78afc4fd641b 16584072-ef78-4a80-91ab-cbd47e9bc70d auto

5. Volume fails to attach
No error volume remains unattached
cinder list 
+--------------------------------------+-----------+-------------+------+----------------------------+----------+--------------------------------------+
| ID                                   | Status    | Name        | Size | Volume Type                | Bootable | Attached to                          |
+--------------------------------------+-----------+-------------+------+----------------------------+----------+--------------------------------------+
| 16584072-ef78-4a80-91ab-cbd47e9bc70d | available | 2-Encrypted | 1    | LuksEncryptor-Template-256 | false    |                                      |


Actual results:
Volume fails to attach. 


Expected results:
Volume should successfully reattach. 

Additional info:

Comment 1 Tzach Shefi 2018-04-15 13:46:36 UTC
Forgot to mention Cinder/Nova versions: 

openstack-nova-conductor-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
python-nova-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
python-novaclient-9.1.1-1.el7ost.noarch
openstack-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch
openstack-nova-scheduler-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-console-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch
openstack-nova-compute-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
python2-cinderclient-3.5.0-1.el7ost.noarch
python-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch
openstack-nova-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-novncproxy-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
puppet-nova-12.3.1-0.20180319062741.9db79a6.el7ost.noarch
openstack-nova-common-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-migration-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-placement-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch

Comment 3 Tzach Shefi 2018-04-15 14:04:21 UTC
Doesn't show up in logs post update. 

A new encrypted RBD volume was created:
#cinder create 1 --volume-type LuksEncryptor-Template-256 --name NewEncVol

It successfully attached:

| ac8045f2-a8d6-4567-8476-40f7e0f63dcf | in-use    | NewEncVol   | 1    | LuksEncryptor-Template-256 | false    | 923c5318-8502-4f85-a215-78afc4fd641b |


Volume detach works
#nova volume-detach 923c5318-8502-4f85-a215-78afc4fd641b ac8045f2-a8d6-4567-8476-40f7e0f63dcf

Cinder list -> 
| ac8045f2-a8d6-4567-8476-40f7e0f63dcf | available | NewEncVol   | 1    | LuksEncryptor-Template-256 | false    |                                      |

Re-attaching this new volume, now worked.  
 nova volume-attach 923c5318-8502-4f85-a215-78afc4fd641b ac8045f2-a8d6-4567-8476-40f7e0f63dcf auto

| ac8045f2-a8d6-4567-8476-40f7e0f63dcf | in-use    | NewEncVol   | 1    | LuksEncryptor-Template-256 | false    | 923c5318-8502-4f85-a215-78afc4fd641b |


Now I'm not sure if this bug is a one off / fluke, or maybe only original volume had an issue.

Comment 4 Lee Yarwood 2018-04-15 16:44:36 UTC
Odd, can you confirm if the 7912eac8-2652-4c92-b53f-3db4ecca7bc7 secret was removed from Barbican by checking the API logs? If it was it might suggest a bug in Cinder when deleting the initial attachment, if it wasn't then it suggests Barbican somehow lost track of the secret.

Comment 5 Tzach Shefi 2018-04-17 06:13:47 UTC
Created attachment 1422913 [details]
Requested logs

Lee, 
I wasn't sure which logs so I grepped for 7912eac8-2652-4c92-b53f-3db4ecca7bc7

Attached are the logs I found on controller 0 where ^ ID was found. 

/var/log/containers/cinder/cinder-volume.log.1 /var/log/containers/barbican/main.log
/var/log/containers/httpd/barbican-api/barbican_wsgi_main_access.log.1

Comment 6 Lee Yarwood 2018-04-19 15:56:31 UTC
(In reply to Tzach Shefi from comment #5)
> Created attachment 1422913 [details]
> Requested logs
> 
> Lee, 
> I wasn't sure which logs so I grepped for
> 7912eac8-2652-4c92-b53f-3db4ecca7bc7
> 
> Attached are the logs I found on controller 0 where ^ ID was found. 
> 
> /var/log/containers/cinder/cinder-volume.log.1
> /var/log/containers/barbican/main.log
> /var/log/containers/httpd/barbican-api/barbican_wsgi_main_access.log.1

Yeah I only see GETs against the secret in these logs, something is clearly wrong on the barbarian backend here if we can't suddenly find the secret, I can't see anything obvious in the main.log file so reassigning for review by the Security DFG.

Comment 7 Tzach Shefi 2018-04-22 08:46:06 UTC
FYI

Failed to reproduce this bug on a new system, issue might have been a fluke. 

I've attached an encrypted volume to an instance wrote data. 
Unattached and successfully reattached volume, including data integrity check. 

That being said whist also not having the original system,
not sure if it's worth pursuing.

Comment 8 Ade Lee 2018-04-24 19:13:02 UTC
Are any other logs from the original system available?  From the logs here, I see nothing that came into the barbican API to delete the secret.

The only situation that I can think of where a secret would be "deleted" is if
a) the secret expired.  ie. we created a secret with a very short expiration time for some reason.
b) the keystone project was deleted.  In this case, the barbican-keystone-listener would delete the barbican internal project associated with the keystone project and "delete" the associated secrets.

I use "delete" in quotes because what actually happens is a soft delete - where a field is_deleted is set to True, making that particular record inaccessible via the api.

It would be useful to see any barbican-keystone-listener logs and/or mysql logs to see what transactions took place.

Also, if this recurs, it would be useful to check the db and see if the secret is still there albeit soft deleted.

Comment 9 Tzach Shefi 2018-04-25 08:25:10 UTC
Nope sorry no more logs available
system was reused, logs sank into oblivion :( 

Should it ever happen again, I'll update bz with requested info. 

Probably safe to close bz. 
Or have it linger for a little longer then close, 
as we can't reproduce or supply more logs. 

Tzach

Comment 10 Harry Rybacki 2018-05-11 18:58:45 UTC
Closing per comment#9 -- please re-open if the issue re-occurs and provide logs requested by Ade per comment#8.


Note You need to log in before you can comment on or make changes to this bug.