In OSP 13 packstack allinone deployment, there is failing tempest test tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks. > 2018-03-05 16:22:23,659 29901 INFO [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 400 POST http://172.16.1.21:8776/v3/41beae394ac34e29902ebbdf8f85514e/volumes 0.428s > 2018-03-05 16:22:23,660 29901 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} > Body: {"volume": {"snapshot_id": null, "display_name": "tempest-TestVolumeBootPattern-volume-174359571", "imageRef": null, "volume_type": "tempest-scenario-type-luks-424487280", "size": 1}} > Response - Headers: {'status': '400', u'content-length': '61', 'content-location': 'http://172.16.1.21:8776/v3/41beae394ac34e29902ebbdf8f85514e/volumes', u'x-compute-request-id': 'req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e', u'vary': 'OpenStack-API-Version', u'openstack-api-version': 'volume 3.0', u'connection': 'close', u'date': 'Mon, 05 Mar 2018 21:22:23 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e'} > Body: {"badRequest": {"message": "Key manager error", "code": 400}} openstack-packstack.noarch 1:12.0.0-0.20180221132607.14db552.el7ost @rhelosp-13.0-puddle openstack-packstack-puppet.noarch 1:12.0.0-0.20180221132607.14db552.el7ost @rhelosp-13.0-puddle python2-barbicanclient.noarch 4.6.0-0.20180211180442.262025b.el7ost @rhelosp-13.0-puddle openstack-cinder.noarch 1:12.0.0-0.20180227162609.7d27804.el7ost @rhelosp-13.0-puddle puppet-cinder.noarch 12.3.1-0.20180222074326.18152ac.el7ost @rhelosp-13.0-puddle python-cinder.noarch 1:12.0.0-0.20180227162609.7d27804.el7ost @rhelosp-13.0-puddle python2-cinderclient.noarch 3.5.0-0.20180211213738.1de605c.el7ost @rhelosp-13.0-puddle Looks like that in cinder.conf key_manager was switched to barbican: > [key_manager] > ... > # Specify the key manager implementation. Options are "barbican" and "vault". > # Default is "barbican". Will support the values earlier set using > # [key_manager]/api_class for some time. (string value) > # Deprecated group/name - [key_manager]/api_class > #backend = barbican > backend=castellan.key_manager.barbican_key_manager.BarbicanKeyManager > api_class = cinder.keymgr.conf_key_mgr.ConfKeyManager > ... // rest of this section opt commented out but at the same time barbican section isn't configured (correctly/at-all) (notice e.g. auth_endpoint): > [barbican] > > # > # From castellan.config > # > > # Use this endpoint to connect to Barbican, for example: > # "http://localhost:9311/" (string value) > #barbican_endpoint = <None> > > # Version of the Barbican API, for example: "v1" (string value) > #barbican_api_version = <None> > > # Use this endpoint to connect to Keystone (string value) > # Deprecated group/name - [key_manager]/auth_url > #auth_endpoint = http://localhost/identity/v3 > > # Number of seconds to wait before retrying poll for key creation completion > # (integer value) > #retry_delay = 1 > > # Number of times to retry poll for key creation completion (integer value) > #number_of_retries = 60 > > # Specifies if insecure TLS (https) requests. If False, the server's > # certificate will not be validated (boolean value) > #verify_ssl = true There seems to be possibly multiple issues related to this, in cinder/volume.log unable to load BarbicanKeyManager: > 2018-03-05 15:55:15.228 13266 WARNING stevedore.named [req-bcdce443-d692-4c20-9dae-122d03ec2361 - - - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager: ServiceNotFound: Service cinder-volume could not be found on host 2bd37abf7f332e6fc2ad17d7861df7ec-aio-0@lvm. > 2018-03-05 15:55:15.228 13266 WARNING castellan.key_manager [req-bcdce443-d692-4c20-9dae-122d03ec2361 - - - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager' also in cinder/api.log are failures: - contacting identity at incorrect endpoint (as seen in barbican section above) - then unable to get suitable url for plugin (don't know if consequence or parallel?) > 2018-03-05 16:22:23.642 26605 WARNING keystoneauth.identity.generic.base [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Failed to discover available identity versions when contacting http://localhost/identity/v3. Attempting to parse version from URL.: NotFound: Not Found (HTTP 404) > 2018-03-05 16:22:23.643 26602 INFO cinder.api.openstack.wsgi [req-d3f60681-8aea-4337-b30e-ef326b1cd97b 7e3ca040ba294f979af0e8e3c604c328 7f965db5b64d422fbdd7b3369f12646b - default default] http://172.16.1.21:8776/v2/7f965db5b64d422fbdd7b3369f12646b/volumes/c53fac3c-8482-4cce-9362-f4485f6bd761 returned with HTTP 202 > 2018-03-05 16:22:23.643 26605 ERROR castellan.key_manager.barbican_key_manager [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Error creating Barbican client: Could not determine a suitable URL for the plugin: DiscoveryFailure: Could not determine a suitable URL for the plugin > 2018-03-05 16:22:23.644 26602 INFO eventlet.wsgi.server [req-d3f60681-8aea-4337-b30e-ef326b1cd97b 7e3ca040ba294f979af0e8e3c604c328 7f965db5b64d422fbdd7b3369f12646b - default default] 172.16.1.21 "DELETE /v2/7f965db5b64d422fbdd7b3369f12646b/volumes/c53fac3c-8482-4cce-9362-f4485f6bd761 HTTP/1.1" status: 202 len: 206 time: 0.0996070 > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Key manager error: KeyManagerError: Key manager error: Could not determine a suitable URL for the plugin > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils Traceback (most recent call last): > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils File "/usr/lib/python2.7/site-packages/cinder/volume/utils.py", line 903, in create_encryption_key > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils length=length) > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 229, in create_key > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils barbican_client = self._get_barbican_client(context) > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 132, in _get_barbican_client > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils raise exception.KeyManagerError(reason=e) > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils KeyManagerError: Key manager error: Could not determine a suitable URL for the plugin > 2018-03-05 16:22:23.643 26605 ERROR cinder.volume.utils > 2018-03-05 16:22:23.649 26605 WARNING cinder.volume.api [req-3d0539fe-7f3a-4dad-9a55-e8c7ee48686e 496693321c524c6cb9868962e6b1dcca 41beae394ac34e29902ebbdf8f85514e - default default] Task 'cinder.volume.flows.api.create_volume.ExtractVolumeRequestTask;volume:create' (54166d95-1fc2-4ebc-9882-5276dc6e9840) transitioned into state 'FAILURE' from state 'RUNNING' > 1 predecessors (most recent first): > Flow 'volume_create_api': Invalid: Key manager error > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api Traceback (most recent call last): > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api result = task.execute(**arguments) > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api File "/usr/lib/python2.7/site-packages/cinder/volume/flows/api/create_volume.py", line 465, in execute > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api image_meta) > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api File "/usr/lib/python2.7/site-packages/cinder/volume/flows/api/create_volume.py", line 401, in _get_encryption_key_id > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api volume_type_id) > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api File "/usr/lib/python2.7/site-packages/cinder/volume/utils.py", line 909, in create_encryption_key > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api raise exception.Invalid(message="Key manager error") > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api Invalid: Key manager error > 2018-03-05 16:22:23.649 26605 ERROR cinder.volume.api Fyi in case of nova.conf there is still prev used: > [key_manager] > backend=nova.keymgr.conf_key_mgr.ConfKeyManager > api_class = nova.keymgr.conf_key_mgr.ConfKeyManager Expected would be that either way it is configured in a way it works (at most with providing enc key). As in osp 12 it used 'api_class = cinder.keymgr.conf_key_mgr.ConfKeyManager'. Or that the barbican part would be configured for working state.
I tracked the problem to infrared's packstack plugin, specifically the configure_services_post_install.yml playbook. There are tasks that configure Nova and Cinder to use the ConfKeyManager, but they're configuring the key_manager's api_class ([1] [2]), and that option has been deprecated. The tasks should be configuring the new 'backend' option. [1] https://github.com/redhat-openstack/infrared/blob/master/plugins/packstack/configure_services_post_install.yml#L33 [2] https://github.com/redhat-openstack/infrared/blob/master/plugins/packstack/configure_services_post_install.yml#L54 The puppet-cinder and puppet-nova code both default to setting the 'backend' to the ConfKeyManager [3], [4]. However, packstack overrides the 'backend' cinder [4]. [3] https://github.com/openstack/puppet-cinder/blob/master/manifests/api.pp#L210 [4] https://github.com/openstack/puppet-nova/blob/master/manifests/compute.pp#L186 [5] https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/cinder.pp#L40 This leads to a situation where nova.conf has both api_class and backend options specifying the ConfKeyManager. But in cinder.conf, the api_class is ConfKeyManager and the backend is Barbican. The backend option supersedes the api_class, so that's how Cinder ends up trying to use Barbican. There is another problem with the way infrared is configuring encryption in cinder.conf. Like Nova, Cinder's ConfKeyManager requires the fixed_key be set. This is done for Nova [6] but it is not done for Cinder. [6] https://github.com/redhat-openstack/infrared/blob/master/plugins/packstack/configure_services_post_install.yml#L25 To summarize, the following changes need to be made to infrared's packstack plugin. The configure_services_post_install.yml playbook needs to be updated to: - Set the key_manager 'backend' and not the 'api_class' - Set the key_manager 'fixed_key' in cinder.conf (must be same as Nova value)
Thanks for amazingly detailed answer even including infrared, will prepare patch for it. So tempest failure in our case is user misconfiguration (esp due to using deprecated api_class). But then, if default is barbican (and just for cinder?) should rest of it be configured correctly by packstack cinder manifest too?
Different projects have their own notion of defaults. Castellan's key_manager defaults the backend to 'barbican' [7]. [7] https://github.com/openstack/castellan/blob/master/castellan/key_manager/__init__.py#L26 However, puppet-cinder and puppet-nova assign the ConfKeyManager as their default [3], [4]. Packstack relies on the puppet modules, but overrides the cinder configuration to specify barbican [5]. Then, infrared comes along and makes its own decision to use the ConfKeyManager [1], [2]. Packstack and infrared's packstack plugin are certainly free to use Barbican. However, additional work will be required to fully configure the whole deployment. As we see in this BZ's tempest logs, it's not a matter of simply configuring Nova's and Cinder's key_manager backend. The rest of the Barbican service would need to be configured, and the scope of that effort goes well beyond this BZ.
Packstack does not support barbican, so it should never configure barbican backend for encryption. I'll fix in packstack to use ConfKeyManager.
Thanks for both answers. That's kind of why I was asking - infrared generally should NOT touch any of such configs - unless they are out of scope of installer (regardless which), what was obviously case of fixed_key in nova.conf. And so I wanted to have it clarified if now infrared should start configuring required-user-inputs for barbican for cinder (and so request most of these to work by default) and such. If packstack is not supposed to support barbican by default and it will be changed back then that is what this bug is about then. (in addition to updating infrared for correct api_class=>backend + fixed_key in cinder.conf (when >=13 as afaik worked for <13) thanks again for that info)
https://review.openstack.org/#/c/553317 should change key_manager/backend to cinder.keymgr.conf_key_mgr.ConfKeyManager
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086