Bug 1555271 - cinder encrypted volumes fail - configured to use barbican which itself is not configured
Summary: cinder encrypted volumes fail - configured to use barbican which itself is no...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-packstack
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: beta
: 13.0 (Queens)
Assignee: Alfredo Moralejo
QA Contact: Frank Jansen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-14 11:23 UTC by Pavel Sedlák
Modified: 2018-08-03 18:33 UTC (History)
6 users (show)

Fixed In Version: openstack-packstack-12.0.0-0.20180327155808.90cd9d5.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:36:14 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 553317 'None' MERGED Revert "Define key_manager/api_class to a class full path" 2020-09-25 13:37:00 UTC
OpenStack gerrit 554214 'None' MERGED Revert "Define key_manager/api_class to a class full path" 2020-09-25 13:37:04 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:37:23 UTC

Description Pavel Sedlák 2018-03-14 11:23:33 UTC
In OSP 13 packstack allinone deployment,
there is failing tempest test tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks.

> 2018-03-05 16:22:23,659 29901 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 400 POST http://172.16.1.21:8776/v3/41beae394ac34e29902ebbdf8f85514e/volumes 0.428s
> 2018-03-05 16:22:23,660 29901 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
>         Body: {"volume": {"snapshot_id": null, "display_name": "tempest-TestVolumeBootPattern-volume-174359571", "imageRef": null, "volume_type": "tempest-scenario-type-luks-424487280", "size": 1}}
>     Response - Headers: {'status': '400', u'content-length': '61', 'content-location': 'http://172.16.1.21:8776/v3/41beae394ac34e29902ebbdf8f85514e/volumes', u'x-compute-request-id': 'req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e', u'vary': 'OpenStack-API-Version', u'openstack-api-version': 'volume 3.0', u'connection': 'close', u'date': 'Mon, 05 Mar 2018 21:22:23 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e'}
>         Body: {"badRequest": {"message": "Key manager error", "code": 400}}

openstack-packstack.noarch           1:12.0.0-0.20180221132607.14db552.el7ost @rhelosp-13.0-puddle    
openstack-packstack-puppet.noarch    1:12.0.0-0.20180221132607.14db552.el7ost @rhelosp-13.0-puddle    
python2-barbicanclient.noarch        4.6.0-0.20180211180442.262025b.el7ost @rhelosp-13.0-puddle    
openstack-cinder.noarch              1:12.0.0-0.20180227162609.7d27804.el7ost @rhelosp-13.0-puddle    
puppet-cinder.noarch                 12.3.1-0.20180222074326.18152ac.el7ost @rhelosp-13.0-puddle    
python-cinder.noarch                 1:12.0.0-0.20180227162609.7d27804.el7ost @rhelosp-13.0-puddle    
python2-cinderclient.noarch          3.5.0-0.20180211213738.1de605c.el7ost @rhelosp-13.0-puddle    


Looks like that in cinder.conf key_manager was switched to barbican:

> [key_manager]
> ...
> # Specify the key manager implementation. Options are "barbican" and "vault".
> # Default is  "barbican". Will support the  values earlier set using
> # [key_manager]/api_class for some time. (string value)
> # Deprecated group/name - [key_manager]/api_class
> #backend = barbican
> backend=castellan.key_manager.barbican_key_manager.BarbicanKeyManager
> api_class = cinder.keymgr.conf_key_mgr.ConfKeyManager
> ... // rest of this section opt commented out

but at the same time barbican section isn't configured (correctly/at-all)
(notice e.g. auth_endpoint):

> [barbican]
> 
> #
> # From castellan.config
> #
> 
> # Use this endpoint to connect to Barbican, for example:
> # "http://localhost:9311/" (string value)
> #barbican_endpoint = <None>
> 
> # Version of the Barbican API, for example: "v1" (string value)
> #barbican_api_version = <None>
> 
> # Use this endpoint to connect to Keystone (string value)
> # Deprecated group/name - [key_manager]/auth_url
> #auth_endpoint = http://localhost/identity/v3
> 
> # Number of seconds to wait before retrying poll for key creation completion
> # (integer value)
> #retry_delay = 1
> 
> # Number of times to retry poll for key creation completion (integer value)
> #number_of_retries = 60
> 
> # Specifies if insecure TLS (https) requests. If False, the server's
> # certificate will not be validated (boolean value)
> #verify_ssl = true


There seems to be possibly multiple issues related to this,
in cinder/volume.log unable to load BarbicanKeyManager:
> 2018-03-05 15:55:15.228 13266 WARNING stevedore.named [req-bcdce443-d692-4c20-9dae-122d03ec2361 - - - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager: ServiceNotFound: Service cinder-volume could not be found on host 2bd37abf7f332e6fc2ad17d7861df7ec-aio-0@lvm.
> 2018-03-05 15:55:15.228 13266 WARNING castellan.key_manager [req-bcdce443-d692-4c20-9dae-122d03ec2361 - - - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager'

also in cinder/api.log are failures:
- contacting identity at incorrect endpoint (as seen in barbican section above)
- then unable to get suitable url for plugin (don't know if consequence or parallel?)

> 2018-03-05 16:22:23.642 26605 WARNING keystoneauth.identity.generic.base [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Failed to discover available identity versions when contacting http://localhost/identity/v3. Attempting to parse version from URL.: NotFound: Not Found (HTTP 404)
> 2018-03-05 16:22:23.643 26602 INFO cinder.api.openstack.wsgi [req-d3f60681-8aea-4337-b30e-ef326b1cd97b 7e3ca040ba294f979af0e8e3c604c328 7f965db5b64d422fbdd7b3369f12646b - default default] http://172.16.1.21:8776/v2/7f965db5b64d422fbdd7b3369f12646b/volumes/c53fac3c-8482-4cce-9362-f4485f6bd761 returned with HTTP 202
> 2018-03-05 16:22:23.643 26605 ERROR castellan.key_manager.barbican_key_manager [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Error creating Barbican client: Could not determine a suitable URL for the plugin: DiscoveryFailure: Could not determine a suitable URL for the plugin
> 2018-03-05 16:22:23.644 26602 INFO eventlet.wsgi.server [req-d3f60681-8aea-4337-b30e-ef326b1cd97b 7e3ca040ba294f979af0e8e3c604c328 7f965db5b64d422fbdd7b3369f12646b - default default] 172.16.1.21 "DELETE /v2/7f965db5b64d422fbdd7b3369f12646b/volumes/c53fac3c-8482-4cce-9362-f4485f6bd761 HTTP/1.1" status: 202  len: 206 time: 0.0996070
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Key manager error: KeyManagerError: Key manager error: Could not determine a suitable URL for the plugin
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils Traceback (most recent call last):
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils   File "/usr/lib/python2.7/site-packages/cinder/volume/utils.py", line 903, in create_encryption_key
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils     length=length)
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils   File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 229, in create_key
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils     barbican_client = self._get_barbican_client(context)
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils   File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 132, in _get_barbican_client
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils     raise exception.KeyManagerError(reason=e)
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils KeyManagerError: Key manager error: Could not determine a suitable URL for the plugin
> 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils 
> 2018-03-05 16:22:23.649 26605 WARNING cinder.volume.api [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Task 'cinder.volume.flows.api.create_volume.ExtractVolumeRequestTask;volume:create' (54166d95-1fc2-4ebc-9882-5276dc6e9840) transitioned into state 'FAILURE' from state 'RUNNING'
> 1 predecessors (most recent first):
>   Flow 'volume_create_api': Invalid: Key manager error
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api Traceback (most recent call last):
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api     result = task.execute(**arguments)
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/api/create_volume.py", line 465, in execute
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api     image_meta)
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/api/create_volume.py", line 401, in _get_encryption_key_id
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api     volume_type_id)
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api   File "/usr/lib/python2.7/site-packages/cinder/volume/utils.py", line 909, in create_encryption_key
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api     raise exception.Invalid(message="Key manager error")
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api Invalid: Key manager error
> 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api 


Fyi in case of nova.conf there is still prev used:
> [key_manager]
> backend=nova.keymgr.conf_key_mgr.ConfKeyManager
> api_class = nova.keymgr.conf_key_mgr.ConfKeyManager

Expected would be that either way it is configured in a way it works (at most with providing enc key).
As in osp 12 it used 'api_class = cinder.keymgr.conf_key_mgr.ConfKeyManager'.
Or that the barbican part would be configured for working state.

Comment 2 Alan Bishop 2018-03-14 16:21:10 UTC
I tracked the problem to infrared's packstack plugin, specifically the
configure_services_post_install.yml playbook. There are tasks that configure
Nova and Cinder to use the ConfKeyManager, but they're configuring the
key_manager's api_class ([1] [2]), and that option has been deprecated. The
tasks should be configuring the new 'backend' option.

[1] https://github.com/redhat-openstack/infrared/blob/master/plugins/packstack/configure_services_post_install.yml#L33
[2] https://github.com/redhat-openstack/infrared/blob/master/plugins/packstack/configure_services_post_install.yml#L54

The puppet-cinder and puppet-nova code both default to setting the 'backend'
to the ConfKeyManager [3], [4]. However, packstack overrides the 'backend'
cinder [4].

[3] https://github.com/openstack/puppet-cinder/blob/master/manifests/api.pp#L210
[4] https://github.com/openstack/puppet-nova/blob/master/manifests/compute.pp#L186
[5] https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/cinder.pp#L40

This leads to a situation where nova.conf has both api_class and backend
options specifying the ConfKeyManager. But in cinder.conf, the api_class is
ConfKeyManager and the backend is Barbican. The backend option supersedes the
api_class, so that's how Cinder ends up trying to use Barbican.

There is another problem with the way infrared is configuring encryption in
cinder.conf. Like Nova, Cinder's ConfKeyManager requires the fixed_key be
set. This is done for Nova [6] but it is not done for Cinder.

[6] https://github.com/redhat-openstack/infrared/blob/master/plugins/packstack/configure_services_post_install.yml#L25

To summarize, the following changes need to be made to infrared's packstack
plugin. The configure_services_post_install.yml playbook needs to be updated
to:

- Set the key_manager 'backend' and not the 'api_class'
- Set the key_manager 'fixed_key' in cinder.conf (must be same as Nova value)

Comment 3 Pavel Sedlák 2018-03-14 18:27:02 UTC
Thanks for amazingly detailed answer even including infrared, will prepare patch for it.

So tempest failure in our case is user misconfiguration (esp due to using deprecated api_class).

But then, if default is barbican (and just for cinder?) should rest of it be configured correctly by packstack cinder manifest too?

Comment 4 Alan Bishop 2018-03-14 19:10:37 UTC
Different projects have their own notion of defaults. Castellan's key_manager defaults the backend to 'barbican' [7]. 

[7] https://github.com/openstack/castellan/blob/master/castellan/key_manager/__init__.py#L26

However, puppet-cinder and puppet-nova assign the ConfKeyManager as their default [3], [4].

Packstack relies on the puppet modules, but overrides the cinder configuration to specify barbican [5].

Then, infrared comes along and makes its own decision to use the ConfKeyManager [1], [2].

Packstack and infrared's packstack plugin are certainly free to use Barbican. However, additional work will be required to fully configure the whole deployment. As we see in this BZ's tempest logs, it's not a matter of simply configuring Nova's and Cinder's key_manager backend. The rest of the Barbican service would need to be configured, and the scope of that effort goes well beyond this BZ.

Comment 5 Alfredo Moralejo 2018-03-14 19:51:28 UTC
Packstack does not support barbican, so it should never configure barbican backend for encryption. I'll fix in packstack to use ConfKeyManager.

Comment 6 Pavel Sedlák 2018-03-15 08:32:32 UTC
Thanks for both answers.

That's kind of why I was asking - infrared generally should NOT touch any of such configs - unless they are out of scope of installer (regardless which), what was obviously case of fixed_key in nova.conf.

And so I wanted to have it clarified if now infrared should start configuring required-user-inputs for barbican for cinder (and so request most of these to work by default) and such.
If packstack is not supposed to support barbican by default and it will be changed back then that is what this bug is about then.

(in addition to updating infrared for correct api_class=>backend + fixed_key in cinder.conf (when >=13 as afaik worked for <13) thanks again for that info)

Comment 7 Alfredo Moralejo 2018-03-15 10:29:41 UTC
https://review.openstack.org/#/c/553317 should change key_manager/backend to cinder.keymgr.conf_key_mgr.ConfKeyManager

Comment 17 errata-xmlrpc 2018-06-27 13:36:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.