Bug 1478161

Summary: [OSP12] tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern was failed due to" Build of instance 53d6c154-6461-44dc-90d1-a5dc2a24776e aborted: keymgr.fixed_key not defined"
Product: Red Hat OpenStack Reporter: Artem Hrechanychenko <ahrechan>
Component: openstack-novaAssignee: Eoghan Glynn <eglynn>
Status: CLOSED DUPLICATE QA Contact: Joe H. Rahme <jhakimra>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: abishop, ahrechan, aschultz, berrange, bkopilov, dasmith, eglynn, jjoyce, jschluet, kchamart, lyarwood, m.andre, mbooth, pgrist, rhallise, sasha, sbauza, sferdjao, sgordon, slinaber, srevivo, tshefi, tvignaud, vromanso
Target Milestone: rcKeywords: AutomationBlocker, Reopened, Triaged
Target Release: 12.0 (Pike)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-01 12:38:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Artem Hrechanychenko 2017-08-03 18:46:31 UTC
OSP12 HA+OC_SSL+UC_SSL
3ctrl+2comp nodes

Description of problem:
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern

 Response - Headers: {'status': '200', u'content-length': '1269', 'content-location': 'https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2', u'x-compute-request-id': 'req-1f71bbc9-8044-4b12-b2fd-e2e4f3b5dab7', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 03 Aug 2017 09:41:56 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-1f71bbc9-8044-4b12-b2fd-e2e4f3b5dab7'}
        Body: {"volume": {"status": "in-use", "user_id": "97b72ee624034c57bc8dfd5535abe9bb", "attachments": [{"server_id": "53d6c154-6461-44dc-90d1-a5dc2a24776e", "attachment_id": "0bd39448-18b2-4e41-b69e-ef27a54f5da2", "attached_at": "2017-08-03T09:36:43.000000", "host_name": null, "volume_id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "device": "/dev/vda", "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2"}], "links": [{"href": "https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "self"}, {"href": "https://10.0.0.101:13776/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "bookmark"}], "availability_zone": "nova", "bootable": "true", "encrypted": true, "created_at": "2017-08-03T09:36:10.000000", "description": null, "os-vol-tenant-attr:tenant_id": "c9e72d50d5634ec989dbca8724e467d1", "updated_at": "2017-08-03T09:36:43.000000", "volume_type": "tempest-scenario-type-luks-651894320", "name": "tempest-TestVolumeBootPattern-volume-2088985538", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "size": 1}}
2017-08-03 05:41:59,383 15257 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 200 GET https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2 0.186s
2017-08-03 05:41:59,384 15257 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1269', 'content-location': 'https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2', u'x-compute-request-id': 'req-346458a1-c424-4367-afc2-d74a2980f0a8', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 03 Aug 2017 09:41:57 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-346458a1-c424-4367-afc2-d74a2980f0a8'}
        Body: {"volume": {"status": "in-use", "user_id": "97b72ee624034c57bc8dfd5535abe9bb", "attachments": [{"server_id": "53d6c154-6461-44dc-90d1-a5dc2a24776e", "attachment_id": "0bd39448-18b2-4e41-b69e-ef27a54f5da2", "attached_at": "2017-08-03T09:36:43.000000", "host_name": null, "volume_id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "device": "/dev/vda", "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2"}], "links": [{"href": "https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "self"}, {"href": "https://10.0.0.101:13776/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "bookmark"}], "availability_zone": "nova", "bootable": "true", "encrypted": true, "created_at": "2017-08-03T09:36:10.000000", "description": null, "os-vol-tenant-attr:tenant_id": "c9e72d50d5634ec989dbca8724e467d1", "updated_at": "2017-08-03T09:36:43.000000", "volume_type": "tempest-scenario-type-luks-651894320", "name": "tempest-TestVolumeBootPattern-volume-2088985538", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "size": 1}}
2017-08-03 05:42:00,855 15257 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 200 GET https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2 0.469s
2017-08-03 05:42:00,855 15257 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1269', 'content-location': 'https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2', u'x-compute-request-id': 'req-739a02d4-fe20-4812-9856-8996c2b01d31', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 03 Aug 2017 09:41:58 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-739a02d4-fe20-4812-9856-8996c2b01d31'}
        Body: {"volume": {"status": "in-use", "user_id": "97b72ee624034c57bc8dfd5535abe9bb", "attachments": [{"server_id": "53d6c154-6461-44dc-90d1-a5dc2a24776e", "attachment_id": "0bd39448-18b2-4e41-b69e-ef27a54f5da2", "attached_at": "2017-08-03T09:36:43.000000", "host_name": null, "volume_id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "device": "/dev/vda", "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2"}], "links": [{"href": "https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "self"}, {"href": "https://10.0.0.101:13776/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "bookmark"}], "availability_zone": "nova", "bootable": "true", "encrypted": true, "created_at": "2017-08-03T09:36:10.000000", "description": null, "os-vol-tenant-attr:tenant_id": "c9e72d50d5634ec989dbca8724e467d1", "updated_at": "2017-08-03T09:36:43.000000", "volume_type": "tempest-scenario-type-luks-651894320", "name": "tempest-TestVolumeBootPattern-volume-2088985538", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "size": 1}}
2017-08-03 05:42:02,122 15257 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 200 GET https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2 0.264s
2017-08-03 05:42:02,123 15257 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1269', 'content-location': 'https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2', u'x-compute-request-id': 'req-bd1e90e7-7ac5-4f27-8ead-c3e5d8e6601c', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 03 Aug 2017 09:41:59 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-bd1e90e7-7ac5-4f27-8ead-c3e5d8e6601c'}
        Body: {"volume": {"status": "in-use", "user_id": "97b72ee624034c57bc8dfd5535abe9bb", "attachments": [{"server_id": "53d6c154-6461-44dc-90d1-a5dc2a24776e", "attachment_id": "0bd39448-18b2-4e41-b69e-ef27a54f5da2", "attached_at": "2017-08-03T09:36:43.000000", "host_name": null, "volume_id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "device": "/dev/vda", "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2"}], "links": [{"href": "https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "self"}, {"href": "https://10.0.0.101:13776/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "bookmark"}], "availability_zone": "nova", "bootable": "true", "encrypted": true, "created_at": "2017-08-03T09:36:10.000000", "description": null, "os-vol-tenant-attr:tenant_id": "c9e72d50d5634ec989dbca8724e467d1", "updated_at": "2017-08-03T09:36:43.000000", "volume_type": "tempest-scenario-type-luks-651894320", "name": "tempest-TestVolumeBootPattern-volume-2088985538", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "size": 1}}
2017-08-03 05:42:03,438 15257 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 200 GET https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2 0.313s
2017-08-03 05:42:03,438 15257 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1269', 'content-location': 'https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2', u'x-compute-request-id': 'req-29e8c7f5-1ddf-4e9f-b19d-613e47f03885', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 03 Aug 2017 09:42:01 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-29e8c7f5-1ddf-4e9f-b19d-613e47f03885'}
        Body: {"volume": {"status": "in-use", "user_id": "97b72ee624034c57bc8dfd5535abe9bb", "attachments": [{"server_id": "53d6c154-6461-44dc-90d1-a5dc2a24776e", "attachment_id": "0bd39448-18b2-4e41-b69e-ef27a54f5da2", "attached_at": "2017-08-03T09:36:43.000000", "host_name": null, "volume_id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "device": "/dev/vda", "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2"}], "links": [{"href": "https://10.0.0.101:13776/v2/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "self"}, {"href": "https://10.0.0.101:13776/c9e72d50d5634ec989dbca8724e467d1/volumes/5bf75e2f-18ad-49f2-8894-9451641bedc2", "rel": "bookmark"}], "availability_zone": "nova", "bootable": "true", "encrypted": true, "created_at": "2017-08-03T09:36:10.000000", "description": null, "os-vol-tenant-attr:tenant_id": "c9e72d50d5634ec989dbca8724e467d1", "updated_at": "2017-08-03T09:36:43.000000", "volume_type": "tempest-scenario-type-luks-651894320", "name": "tempest-TestVolumeBootPattern-volume-2088985538", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "5bf75e2f-18ad-49f2-8894-9451641bedc2", "size": 1}}
2017-08-03 05:42:04,557 15257 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 400 DELETE https://10.0.0.101:13776/v2/848a250efab04310ba4d01d8a83f66a0/types/d578719b-1e5a-42ab-8600-5fc0d2af30af 1.117s
2017-08-03 05:42:04,558 15257 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: None
    Response - Headers: {'status': '400', u'content-length': '79', 'content-location': 'https://10.0.0.101:13776/v2/848a250efab04310ba4d01d8a83f66a0/types/d578719b-1e5a-42ab-8600-5fc0d2af30af', u'x-compute-request-id': 'req-ece7704b-896c-4727-b860-796483ad11f6', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 03 Aug 2017 09:42:01 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-ece7704b-896c-4727-b860-796483ad11f6'}
        Body: {"badRequest": {"message": "Target volume type is still in use.", "code": 400}}
}}}

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/volumes_client.py", line 136, in delete_volume
    resp, body = self.delete(url)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 301, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request
    method, url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 770, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.', u'code': 400}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 871, in wait_for_resource_deletion
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (TestVolumeBootPattern:_run_cleanups) Failed to delete volume 5bf75e2f-18ad-49f2-8894-9451641bedc2 within the required time (300 s).
}}}

traceback-3: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/types_client.py", line 88, in delete_volume_type
    resp, body = self.delete("types/%s" % volume_type_id)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 301, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 770, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Target volume type is still in use.', u'code': 400}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 103, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 232, in test_boot_server_from_encrypted_volume_luks
    delete_on_termination=False)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 70, in _boot_instance_from_resource
    return self.create_server(image_id='', **create_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 203, in create_server
    image_id=image_id, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 219, in create_test_server
    server['id'])
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 188, in create_test_server
    clients.servers_client, server['id'], wait_until)
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server 53d6c154-6461-44dc-90d1-a5dc2a24776e failed to build and is in ERROR status
Details: {u'message': u'Build of instance 53d6c154-6461-44dc-90d1-a5dc2a24776e aborted: keymgr.fixed_key not defined', u'code': 500, u'created': u'2017-08-03T09:36:59Z'}

Version-Release number of selected component (if applicable):
OSP12
openstack-nova-compute-16.0.0-0.20170721224719.fe46cfa.el7ost.noarch
puppet-openstacklib-11.2.0-0.20170721172003.a3cef1d.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170721190058.e003b07.el7ost.noarch
python-openstacksdk-0.9.17-0.20170621195806.7946243.el7ost.noarch
openstack-swift-object-2.14.1-0.20170721185434.5b10cf5.el7ost.noarch
openstack-heat-api-9.0.0-0.20170721154436.bdddeee.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170720143547.6c98f31.el7ost.noarch
openstack-nova-api-16.0.0-0.20170721224719.fe46cfa.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170721224719.fe46cfa.el7ost.noarch
openstack-glance-15.0.0-0.20170720002347.0752f7f.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170721140651.c389fae.el7ost.noarch
openstack-swift-container-2.14.1-0.20170721185434.5b10cf5.el7ost.noarch
openstack-keystone-12.0.0-0.20170719224216.025e844.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170721222602.9cb907c.el7ost.noarch
openstack-swift-proxy-2.14.1-0.20170721185434.5b10cf5.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170721154436.bdddeee.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170720220617.b908a30.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170720143547.6c98f31.el7ost.noarch
openstack-tripleo-validations-7.1.1-0.20170720215108.e6ebdd3.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170721224719.fe46cfa.el7ost.noarch
puppet-openstack_extras-11.2.0-0.20170721171851.6a9d2aa.el7ost.noarch
openstack-tripleo-common-containers-7.3.1-0.20170720003002.f0ef9ac.el7ost.noarch
python-openstackclient-3.11.0-0.20170613232431.c69304e.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170712081605.35068ac.el7ost.noarch
openstack-tripleo-common-7.3.1-0.20170720003002.f0ef9ac.el7ost.noarch
openstack-swift-account-2.14.1-0.20170721185434.5b10cf5.el7ost.noarch
openstack-neutron-11.0.0-0.20170721222602.9cb907c.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170721222602.9cb907c.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170721154436.bdddeee.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170721190058.e003b07.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170720143547.6c98f31.el7ost.noarch
openstack-tripleo-ui-7.1.1-0.20170720050235.3cb948f.el7ost.noarch
openstack-zaqar-5.0.0-0.20170721081135.9b966b4.el7ost.noarch
openstack-neutron-common-11.0.0-0.20170721222602.9cb907c.el7ost.noarch
openstack-heat-common-9.0.0-0.20170721154436.bdddeee.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170720143547.6c98f31.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170720143547.6c98f31.el7ost.noarch
openstack-selinux-0.8.7-2.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170721224719.fe46cfa.el7ost.noarch
openstack-nova-common-16.0.0-0.20170721224719.fe46cfa.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170721190058.e003b07.el7ost.noarch
openstack-tempest-16.1.1-0.20170721223940.e1e6e78.el7ost.noarch

images puddle:   2017-07-26.10

How reproducible:
always

Comment 2 Artem Hrechanychenko 2017-08-09 10:36:45 UTC
reproduced in new puddle
puddle - OpenStack-12.0-RHEL-7-20170808.4


tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks

testtools.testresult.real._StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2017-08-09 04:23:50,505 18217 DEBUG    [tempest.scenario.manager] Creating a volume type: tempest-scenario-type-luks-129707297 on backend None
2017-08-09 04:23:51,183 18217 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 200 POST https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types 0.677s
2017-08-09 04:23:51,184 18217 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: {"volume_type": {"extra_specs": {}, "name": "tempest-scenario-type-luks-129707297"}}
    Response - Headers: {'status': '200', u'content-length': '211', 'content-location': 'https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types', u'x-compute-request-id': 'req-795b30c2-5c6e-4e55-91a4-d8f72acac32b', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:50 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-795b30c2-5c6e-4e55-91a4-d8f72acac32b'}
        Body: {"volume_type": {"name": "tempest-scenario-type-luks-129707297", "extra_specs": {}, "os-volume-type-access:is_public": true, "is_public": true, "id": "657eda75-e270-4b87-87ed-374eee673acc", "description": null}}
2017-08-09 04:23:51,185 18217 DEBUG    [tempest.scenario.manager] Creating an encryption type for volume type: 657eda75-e270-4b87-87ed-374eee673acc
2017-08-09 04:23:51,805 18217 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 200 POST https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc/encryption 0.620s
2017-08-09 04:23:51,806 18217 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: {"encryption": {"control_location": "front-end", "key_size": 256, "cipher": "aes-xts-plain64", "provider": "nova.volume.encryptors.luks.LuksEncryptor"}}
    Response - Headers: {'status': '200', u'content-length': '267', 'content-location': 'https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc/encryption', u'x-compute-request-id': 'req-6f1d73d7-2082-4304-b958-6e9cda87a949', u'vary': 'Accept-Encoding', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:51 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-6f1d73d7-2082-4304-b958-6e9cda87a949'}
        Body: {"encryption": {"volume_type_id": "657eda75-e270-4b87-87ed-374eee673acc", "control_location": "front-end", "encryption_id": "893b9db7-c2e3-4545-950c-ee6e40c59ca0", "key_size": 256, "provider": "nova.volume.encryptors.luks.LuksEncryptor", "cipher": "aes-xts-plain64"}}
2017-08-09 04:23:52,397 18217 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:test_boot_server_from_encrypted_volume_luks): 400 POST https://10.0.0.101:13776/v2/49c7c931ea2e49a3b2efe6e3ed6b3c18/volumes 0.590s
2017-08-09 04:23:52,397 18217 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: {"volume": {"snapshot_id": null, "display_name": "tempest-TestVolumeBootPattern-volume-1124969670", "imageRef": null, "volume_type": "tempest-scenario-type-luks-129707297", "size": 1}}
    Response - Headers: {'status': '400', u'content-length': '61', 'content-location': 'https://10.0.0.101:13776/v2/49c7c931ea2e49a3b2efe6e3ed6b3c18/volumes', u'x-compute-request-id': 'req-6b3d228c-033b-4924-93de-cfe21c2fab31', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:51 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-6b3d228c-033b-4924-93de-cfe21c2fab31'}
        Body: {"badRequest": {"message": "Key manager error", "code": 400}}
2017-08-09 04:23:52,959 18217 INFO     [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 202 DELETE https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc 0.557s
2017-08-09 04:23:52,960 18217 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '&lt;omitted>'}
        Body: None
    Response - Headers: {'status': '202', u'content-length': '0', 'content-location': 'https://10.0.0.101:13776/v2/d4436f5dde0f4586afab9ab91ac95522/types/657eda75-e270-4b87-87ed-374eee673acc', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 09 Aug 2017 08:23:52 GMT', u'content-type': 'text/html; charset=UTF-8', u'x-openstack-request-id': 'req-49dd2bc0-13dd-472b-b292-83cb64f1b54e'}
        Body:
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 103, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 224, in test_boot_server_from_encrypted_volume_luks
    volume_type='luks')
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 1278, in create_encrypted_volume
    return self.create_volume(volume_type=volume_type['name'])
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 227, in create_volume
    volume = self.volumes_client.create_volume(**kwargs)['volume']
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/volumes_client.py", line 108, in create_volume
    resp, body = self.post('volumes', post_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 270, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request
    method, url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 770, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Key manager error', u'code': 400}

Comment 3 Artem Hrechanychenko 2017-08-09 10:38:23 UTC
Looks like new bz

Comment 4 Paul Grist 2017-08-09 14:10:11 UTC
Alan chased something similar yesterday and I think this is the same one an infrared infrared patch went in today for.  

https://review.gerrithub.io/#/c/373428/

That was for a packstack based issue, so need to confirm this is the same issue or if we have another keymanager config issues in some downstream area.

Comment 6 Alan Bishop 2017-08-21 14:07:47 UTC
This BZ seemed related to bug #1479751 and I was hoping the fix for that would also fix this one. Unfortunately things didn't work out. Looking further, this error has been reported before, notably in bug #1294422. The error appears to be coming from Nova, so I'm reassigning to DFG:Compute.

Comment 8 Lee Yarwood 2017-09-26 12:40:56 UTC
(In reply to Alan Bishop from comment #6)
> This BZ seemed related to bug #1479751 and I was hoping the fix for that
> would also fix this one. Unfortunately things didn't work out. Looking
> further, this error has been reported before, notably in bug #1294422. The
> error appears to be coming from Nova, so I'm reassigning to DFG:Compute.

c#0 appears to be an n-api issue with keymgr.fixed_key not being set correctly :

> Details: {u'message': u'Build of instance 53d6c154-6461-44dc-90d1-a5dc2a24776e aborted: keymgr.fixed_key not defined', u'code': 500, u'created': u'2017-08-03T09:36:59Z'}

This isn't set by puppet-tripleo, puppet-nova or tripleo-heat-templates as it's unsupported in production envs. Upstream devstack sets this here :

https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1244-L1254

I can only assume this is a tripleo-quickstart / infrared post deployment configuration issue. I don't think there's anything to be fixed in nova here.

Also, c#2 is a separate issue with c-api/c-vol during the creation of an encrypted volume. Nova isn't called as part of this flow.

Comment 9 Lee Yarwood 2017-09-27 11:20:07 UTC
*** Bug 1294422 has been marked as a duplicate of this bug. ***

Comment 10 Lee Yarwood 2017-09-29 12:54:58 UTC
Any thoughts on my previous comment? This appears to be something that needs to be fixed in the post-deployment configuration of our CI envs and not by Nova directly, would you agree? I'd like to close / reassign this bug if possible.

Comment 11 Alan Bishop 2017-10-03 14:34:39 UTC
I agree it seems the issue is due to a problem with the deployment itself, and not something specific to cinder or nova. I'm not very familiar with infrared, but [1] looks like Barbican should have been deployed, and so I'm confused why there's _any_ reference to the fixed_key.

[1] https://github.com/redhat-openstack/infrared/blob/master/plugins/tripleo-overcloud/files/roles/novacontrol/controller.yml#L83

Artem, would it be possible to get access to a deployment when it's in this state?

Comment 12 Lee Yarwood 2017-10-09 09:42:28 UTC
Closing this out with INSUFFICIENT_DATA given the lack of response here, please feel free to reopen if you need any further advice around enabling the fixed_key manager here.

Comment 13 Alan Bishop 2017-10-23 14:14:47 UTC
Bug #1479751 comment #11 seems relevant. Looks like there's an issue getting the fixed_key properly configured in the nova_compute container.

Comment 14 Artem Hrechanychenko 2017-10-23 15:14:18 UTC
So need to move issue from closed

Comment 16 Lee Yarwood 2017-11-01 12:38:36 UTC

*** This bug has been marked as a duplicate of bug 1487920 ***

Comment 17 Alan Bishop 2017-11-21 17:28:26 UTC
Upstream (RDO) is hitting this [1]. I gather it's a relatively new tempest test, and they may just blacklist the test in RDO Ocata, at least until Barbican becomes the default key manager in TripleO.

[1] https://bugs.launchpad.net/tripleo/+bug/1733574