Bug 1567614
| Summary: | Reattaching an encrypted(Barbican) Cinder (RBD) volume to an instance fails | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Tzach Shefi <tshefi> | ||||||
| Component: | openstack-barbican | Assignee: | Ade Lee <alee> | ||||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Prasanth Anbalagan <panbalag> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 13.0 (Queens) | CC: | berrange, dasmith, eglynn, hrybacki, jhakimra, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, tshefi, vromanso | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-05-11 18:58:45 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1230405 | ||||||||
| Attachments: |
|
||||||||
Forgot to mention Cinder/Nova versions: openstack-nova-conductor-17.0.2-0.20180323024604.0390d5f.el7ost.noarch python-nova-17.0.2-0.20180323024604.0390d5f.el7ost.noarch python-novaclient-9.1.1-1.el7ost.noarch openstack-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch openstack-nova-scheduler-17.0.2-0.20180323024604.0390d5f.el7ost.noarch openstack-nova-console-17.0.2-0.20180323024604.0390d5f.el7ost.noarch puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch openstack-nova-compute-17.0.2-0.20180323024604.0390d5f.el7ost.noarch python2-cinderclient-3.5.0-1.el7ost.noarch python-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch openstack-nova-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch openstack-nova-novncproxy-17.0.2-0.20180323024604.0390d5f.el7ost.noarch puppet-nova-12.3.1-0.20180319062741.9db79a6.el7ost.noarch openstack-nova-common-17.0.2-0.20180323024604.0390d5f.el7ost.noarch openstack-nova-migration-17.0.2-0.20180323024604.0390d5f.el7ost.noarch openstack-nova-placement-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch Doesn't show up in logs post update. A new encrypted RBD volume was created: #cinder create 1 --volume-type LuksEncryptor-Template-256 --name NewEncVol It successfully attached: | ac8045f2-a8d6-4567-8476-40f7e0f63dcf | in-use | NewEncVol | 1 | LuksEncryptor-Template-256 | false | 923c5318-8502-4f85-a215-78afc4fd641b | Volume detach works #nova volume-detach 923c5318-8502-4f85-a215-78afc4fd641b ac8045f2-a8d6-4567-8476-40f7e0f63dcf Cinder list -> | ac8045f2-a8d6-4567-8476-40f7e0f63dcf | available | NewEncVol | 1 | LuksEncryptor-Template-256 | false | | Re-attaching this new volume, now worked. nova volume-attach 923c5318-8502-4f85-a215-78afc4fd641b ac8045f2-a8d6-4567-8476-40f7e0f63dcf auto | ac8045f2-a8d6-4567-8476-40f7e0f63dcf | in-use | NewEncVol | 1 | LuksEncryptor-Template-256 | false | 923c5318-8502-4f85-a215-78afc4fd641b | Now I'm not sure if this bug is a one off / fluke, or maybe only original volume had an issue. Odd, can you confirm if the 7912eac8-2652-4c92-b53f-3db4ecca7bc7 secret was removed from Barbican by checking the API logs? If it was it might suggest a bug in Cinder when deleting the initial attachment, if it wasn't then it suggests Barbican somehow lost track of the secret. Created attachment 1422913 [details]
Requested logs
Lee,
I wasn't sure which logs so I grepped for 7912eac8-2652-4c92-b53f-3db4ecca7bc7
Attached are the logs I found on controller 0 where ^ ID was found.
/var/log/containers/cinder/cinder-volume.log.1 /var/log/containers/barbican/main.log
/var/log/containers/httpd/barbican-api/barbican_wsgi_main_access.log.1
(In reply to Tzach Shefi from comment #5) > Created attachment 1422913 [details] > Requested logs > > Lee, > I wasn't sure which logs so I grepped for > 7912eac8-2652-4c92-b53f-3db4ecca7bc7 > > Attached are the logs I found on controller 0 where ^ ID was found. > > /var/log/containers/cinder/cinder-volume.log.1 > /var/log/containers/barbican/main.log > /var/log/containers/httpd/barbican-api/barbican_wsgi_main_access.log.1 Yeah I only see GETs against the secret in these logs, something is clearly wrong on the barbarian backend here if we can't suddenly find the secret, I can't see anything obvious in the main.log file so reassigning for review by the Security DFG. FYI Failed to reproduce this bug on a new system, issue might have been a fluke. I've attached an encrypted volume to an instance wrote data. Unattached and successfully reattached volume, including data integrity check. That being said whist also not having the original system, not sure if it's worth pursuing. Are any other logs from the original system available? From the logs here, I see nothing that came into the barbican API to delete the secret. The only situation that I can think of where a secret would be "deleted" is if a) the secret expired. ie. we created a secret with a very short expiration time for some reason. b) the keystone project was deleted. In this case, the barbican-keystone-listener would delete the barbican internal project associated with the keystone project and "delete" the associated secrets. I use "delete" in quotes because what actually happens is a soft delete - where a field is_deleted is set to True, making that particular record inaccessible via the api. It would be useful to see any barbican-keystone-listener logs and/or mysql logs to see what transactions took place. Also, if this recurs, it would be useful to check the db and see if the secret is still there albeit soft deleted. Nope sorry no more logs available system was reused, logs sank into oblivion :( Should it ever happen again, I'll update bz with requested info. Probably safe to close bz. Or have it linger for a little longer then close, as we can't reproduce or supply more logs. Tzach |
Created attachment 1422192 [details] Nova compute log Description of problem: An attached encrypted (Barbican) RBD Cinder volume was attached to instance, write data to it. Then volume was detached, when trying to reattach the volume to same instance volume fails to attach. Odd errors on attached nova-compute.log 2018-04-15 13:14:06.274 1 ERROR nova.compute.manager [instance: 923c5318-8502-4f85-a215-78afc4fd641b] uuid=managed_object_id) 2018-04-15 13:14:06.274 1 ERROR nova.compute.manager [instance: 923c5318-8502-4f85-a215-78afc4fd641b] ManagedObjectNotFoundError: Key not found, uuid: 7912eac8-2652-4c92-b53f-3db4ecca7bc7 2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request 2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body) 2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment c17e2b89-5a36-4e7e-8c71-b975f2f5ccb3 could not be found. 2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server Version-Release number of selected component (if applicable): rhel7.5 How reproducible: Unsure look every time I try Steps to Reproduce: 1. Boot an instance 2. Create an encrypted(Barbican) backed Cinder(RBD) volume, attach to instance write data. 3. Detach volume from instance 4. Try to reattach same volume to same instance. $nova volume-attach 923c5318-8502-4f85-a215-78afc4fd641b 16584072-ef78-4a80-91ab-cbd47e9bc70d auto 5. Volume fails to attach No error volume remains unattached cinder list +--------------------------------------+-----------+-------------+------+----------------------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+----------------------------+----------+--------------------------------------+ | 16584072-ef78-4a80-91ab-cbd47e9bc70d | available | 2-Encrypted | 1 | LuksEncryptor-Template-256 | false | | Actual results: Volume fails to attach. Expected results: Volume should successfully reattach. Additional info: