Bug 1478161
Summary: | [OSP12] tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern was failed due to" Build of instance 53d6c154-6461-44dc-90d1-a5dc2a24776e aborted: keymgr.fixed_key not defined" | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
Component: | openstack-nova | Assignee: | Eoghan Glynn <eglynn> |
Status: | CLOSED DUPLICATE | QA Contact: | Joe H. Rahme <jhakimra> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 12.0 (Pike) | CC: | abishop, ahrechan, aschultz, berrange, bkopilov, dasmith, eglynn, jjoyce, jschluet, kchamart, lyarwood, m.andre, mbooth, pgrist, rhallise, sasha, sbauza, sferdjao, sgordon, slinaber, srevivo, tshefi, tvignaud, vromanso |
Target Milestone: | rc | Keywords: | AutomationBlocker, Reopened, Triaged |
Target Release: | 12.0 (Pike) | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-11-01 12:38:36 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Artem Hrechanychenko
2017-08-03 18:46:31 UTC
reproduced in new puddle puddle - OpenStack-12.0-RHEL-7-20170808.4 tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks testtools.testresult.real._StringException: Empty attachments: stderr stdout pythonlogging:'': {{{ 2017-08-09 04:23:50,505 18217 DEBUG [tempest.scenario.manager] Creating a volume type: tempest-scenario-type-luks-129707297 on backend None 2017-08-09 04:23:51,183 18217 INFO [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 200 POST https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types 0.677s 2017-08-09 04:23:51,184 18217 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: {"volume_type": {"extra_specs": {}, "name": "tempest-scenario-type-luks-129707297"}} Response - Headers: {'status': '200', u'content-length': '211', 'content-location': 'https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types', u'x-compute-request-id': 'req-795b30c2-5c6e-4e55-91a4-d8f72acac32b', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:50 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-795b30c2-5c6e-4e55-91a4-d8f72acac32b'} Body: {"volume_type": {"name": "tempest-scenario-type-luks-129707297", "extra_specs": {}, "os-volume-type-access:is_public": true, "is_public": true, "id": "657eda75-e270-4b87-87ed-374eee673acc", "description": null}} 2017-08-09 04:23:51,185 18217 DEBUG [tempest.scenario.manager] Creating an encryption type for volume type: 657eda75-e270-4b87-87ed-374eee673acc 2017-08-09 04:23:51,805 18217 INFO [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 200 POST https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc/encryption 0.620s 2017-08-09 04:23:51,806 18217 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: {"encryption": {"control_location": "front-end", "key_size": 256, "cipher": "aes-xts-plain64", "provider": "nova.volume.encryptors.luks.LuksEncryptor"}} Response - Headers: {'status': '200', u'content-length': '267', 'content-location': 'https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc/encryption', u'x-compute-request-id': 'req-6f1d73d7-2082-4304-b958-6e9cda87a949', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:51 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-6f1d73d7-2082-4304-b958-6e9cda87a949'} Body: {"encryption": {"volume_type_id": "657eda75-e270-4b87-87ed-374eee673acc", "control_location": "front-end", "encryption_id": "893b9db7-c2e3-4545-950c-ee6e40c59ca0", "key_size": 256, "provider": "nova.volume.encryptors.luks.LuksEncryptor", "cipher": "aes-xts-plain64"}} 2017-08-09 04:23:52,397 18217 INFO [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 400 POST https://10.0.0.101:13776/v2/49c7c931ea2e49a3b2efe6e3ed6b3c18/volumes 0.590s 2017-08-09 04:23:52,397 18217 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: {"volume": {"snapshot_id": null, "display_name": "tempest-TestVolumeBootPattern-volume-1124969670", "imageRef": null, "volume_type": "tempest-scenario-type-luks-129707297", "size": 1}} Response - Headers: {'status': '400', u'content-length': '61', 'content-location': 'https://10.0.0.101:13776/v2/49c7c931ea2e49a3b2efe6e3ed6b3c18/volumes', u'x-compute-request-id': 'req-6b3d228c-033b-4924-93de-cfe21c2fab31', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:51 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-6b3d228c-033b-4924-93de-cfe21c2fab31'} Body: {"badRequest": {"message": "Key manager error", "code": 400}} 2017-08-09 04:23:52,959 18217 INFO [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 202 DELETE https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc 0.557s 2017-08-09 04:23:52,960 18217 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'status': '202', u'content-length': '0', 'content-location': 'https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:52 GMT', u'content-type': 'text/html; charset=UTF-8', u'x-openstack-request-id': 'req-49dd2bc0-13dd-472b-b292-83cb64f1b54e'} Body: }}} Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tempest/test.py", line 103, in wrapper return f(self, *func_args, **func_kwargs) File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 224, in test_boot_server_from_encrypted_volume_luks volume_type='luks') File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 1278, in create_encrypted_volume return self.create_volume(volume_type=volume_type['name']) File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 227, in create_volume volume = self.volumes_client.create_volume(**kwargs)['volume'] File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/volumes_client.py", line 108, in create_volume resp, body = self.post('volumes', post_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 270, in post return self.request('POST', url, extra_headers, headers, body, chunked) File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request method, url, extra_headers, headers, body, chunked) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request self._error_checker(resp, resp_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 770, in _error_checker raise exceptions.BadRequest(resp_body, resp=resp) tempest.lib.exceptions.BadRequest: Bad request Details: {u'message': u'Key manager error', u'code': 400} Looks like new bz Alan chased something similar yesterday and I think this is the same one an infrared infrared patch went in today for. https://review.gerrithub.io/#/c/373428/ That was for a packstack based issue, so need to confirm this is the same issue or if we have another keymanager config issues in some downstream area. This BZ seemed related to bug #1479751 and I was hoping the fix for that would also fix this one. Unfortunately things didn't work out. Looking further, this error has been reported before, notably in bug #1294422. The error appears to be coming from Nova, so I'm reassigning to DFG:Compute. (In reply to Alan Bishop from comment #6) > This BZ seemed related to bug #1479751 and I was hoping the fix for that > would also fix this one. Unfortunately things didn't work out. Looking > further, this error has been reported before, notably in bug #1294422. The > error appears to be coming from Nova, so I'm reassigning to DFG:Compute. c#0 appears to be an n-api issue with keymgr.fixed_key not being set correctly : > Details: {u'message': u'Build of instance 53d6c154-6461-44dc-90d1-a5dc2a24776e aborted: keymgr.fixed_key not defined', u'code': 500, u'created': u'2017-08-03T09:36:59Z'} This isn't set by puppet-tripleo, puppet-nova or tripleo-heat-templates as it's unsupported in production envs. Upstream devstack sets this here : https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1244-L1254 I can only assume this is a tripleo-quickstart / infrared post deployment configuration issue. I don't think there's anything to be fixed in nova here. Also, c#2 is a separate issue with c-api/c-vol during the creation of an encrypted volume. Nova isn't called as part of this flow. *** Bug 1294422 has been marked as a duplicate of this bug. *** Any thoughts on my previous comment? This appears to be something that needs to be fixed in the post-deployment configuration of our CI envs and not by Nova directly, would you agree? I'd like to close / reassign this bug if possible. I agree it seems the issue is due to a problem with the deployment itself, and not something specific to cinder or nova. I'm not very familiar with infrared, but [1] looks like Barbican should have been deployed, and so I'm confused why there's _any_ reference to the fixed_key. [1] https://github.com/redhat-openstack/infrared/blob/master/plugins/tripleo-overcloud/files/roles/novacontrol/controller.yml#L83 Artem, would it be possible to get access to a deployment when it's in this state? Closing this out with INSUFFICIENT_DATA given the lack of response here, please feel free to reopen if you need any further advice around enabling the fixed_key manager here. Bug #1479751 comment #11 seems relevant. Looks like there's an issue getting the fixed_key properly configured in the nova_compute container. So need to move issue from closed *** This bug has been marked as a duplicate of bug 1487920 *** Upstream (RDO) is hitting this [1]. I gather it's a relatively new tempest test, and they may just blacklist the test in RDO Ocata, at least until Barbican becomes the default key manager in TripleO. [1] https://bugs.launchpad.net/tripleo/+bug/1733574 |