Bug 1668858
| Summary: | Failure prepping block device., Code: 500 when deploying whole disk secure image | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | mlammon | |
| Component: | openstack-ironic | Assignee: | RHOS Maint <rhos-maint> | |
| Status: | CLOSED ERRATA | QA Contact: | mlammon | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 13.0 (Queens) | CC: | bfournie, ccopello, jkreger, mburns, pablo.iranzo, rhos-maint, slinaber | |
| Target Milestone: | z5 | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-ironic-10.1.7-1.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1668860 (view as bug list) | Environment: | ||
| Last Closed: | 2019-03-14 13:54:43 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1668860, 1687601 | |||
As seen on logs...
8da7-5d1376d8be91 3f9a3e1f897e43a698f0189e979f4c75 c3c11a00693048d4aad128f9ceb9665a - default default] [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] Failure prepping block device: InternalServerError: Unable to attach VIF 6ea0679f-2ee0-4d8d-8521-0829d6e7ed70 because Ironic can not update Neutron port 6ea0679f-2ee0-4d8d-8521-0829d6e7ed70 MAC address to match physical MAC address 52:54:00:b3:ab:30 (HTTP 500)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] Traceback (most recent call last):
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2194, in _build_resources
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] instance, network_info)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1841, in prepare_networks_before_block_device_mapping
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] instance=instance)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] self.force_reraise()
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] six.reraise(self.type_, self.value, self.tb)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1834, in prepare_networks_before_block_device_mapping
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] self.plug_vifs(instance, network_info)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1486, in plug_vifs
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] self._plug_vifs(node, instance, network_info)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1456, in _plug_vifs
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] self._plug_vif(node, port_id)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1420, in _plug_vif
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] port_id, retry_on_conflict=False)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py", line 158, in call
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] return self._multi_getattr(client, method)(*args, **kwargs)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/ironicclient/v1/node.py", line 391, in vif_attach
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] self.update(path, data, http_method="POST")
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/ironicclient/v1/node.py", line 336, in update
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] method=http_method)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/ironicclient/common/base.py", line 200, in _update
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] resp, body = self.api.json_request(method, url, **kwargs)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 645, in json_request
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] resp = self._http_request(url, method, **kwargs)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 247, in wrapper
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] return func(self, url, method, **kwargs)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 627, in _http_request
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] error_json.get('debuginfo'), method, url)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b] InternalServerError: Unable to attach VIF 6ea0679f-2ee0-4d8d-8521-0829d6e7ed70 because Ironic can not update Neutron port 6ea0679f-2ee0-4d8d-8521-0829d6e7ed70 MAC address to match physical MAC address 52:54:00:b3:ab:30 (HTTP 500)
2019-01-22 18:22:35.413 30494 ERROR nova.compute.manager [instance: 7faa0226-f2c0-4451-9134-aef22d3d043b]
It appears that this upstream patch has not been backported - https://review.openstack.org/#/c/621400. Environment Installed Packages Name : openstack-ironic-conductor Arch : noarch Epoch : 1 Version : 10.1.7 Release : 1.el7ost Size : 378 Repo : installed From repo : rhelosp-13.0-puddle Summary : The Ironic Conductor URL : http://www.openstack.org License : ASL 2.0 Description : Ironic Conductor for management and provisioning of physical machines This issue was no longer seen during the last regression test which originally found it. Hi,
I'm getting something similar when deploying baremetal instance in overcloud (alongside virtual instances):
| fault | {u'message': u'Build of instance 566100c8-87ca-48c1-9089-7d3e75a2dd90 aborted: Failure prepping block device.', u'code': 500, u'created': u'2019-03-07T11:58:07Z'} |
If I skip one of the networks and just leave another, it gets another kind of error (I reviewed the PR and seemed to be related with networks attached)
on OSP queens
*** Bug 1687599 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0559 |
Description of problem: Failure prepping block device., Code: 500 when deploying whole disk secure image Steps to Reproduce: 1. Deploy undercloud, images, introspect, tagging, etc (nodes uefi boot mode) 2. Build whole disk secure image 3. Deploy overcloud Actual results: Failed overcloud (undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud │····························· overcloud.Controller.1.Controller: │····························· resource_type: OS::TripleO::ControllerServer │····························· physical_resource_id: e5d83f79-ff6c-4223-aa19-3984908501ca │····························· status: CREATE_FAILED │····························· status_reason: | │····························· ResourceInError: resources.Controller: Went to status ERROR due to "Message: Build of instance e5d83f79-ff6c-4223-aa19-3984908501ca aborted: Failure prepping block device., Code: 500" │····························· overcloud.Controller.0.Controller: │····························· resource_type: OS::TripleO::ControllerServer │····························· physical_resource_id: 7faa0226-f2c0-4451-9134-aef22d3d043b │····························· status: CREATE_FAILED │····························· status_reason: | │····························· ResourceInError: resources.Controller: Went to status ERROR due to "Message: Build of instance 7faa0226-f2c0-4451-9134-aef22d3d043b aborted: Failure prepping block device., Code: 500" │····························· overcloud.Compute.0.NovaCompute: │····························· resource_type: OS::TripleO::ComputeServer │····························· physical_resource_id: 1f327355-5689-4b1b-9bf4-c2cc55e11348 │····························· status: CREATE_FAILED │····························· status_reason: | │····························· ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: Build of instance 1f327355-5689-4b1b-9bf4-c2cc55e11348 aborted: Failure prepping block device., Code: 500" openstack server show | fault | {u'message': u'Build of instance 7faa0226-f2c0-4451-9134-aef22d3d043b aborted: Failure prepping block device.', u'code': 500, u'details': u' File "/u│····························· sr/lib/python2.7/site-packages/nova/compute/manager.py", line 1841, in _do_build_and_run_instance\n filter_properties, request_spec)\n File "/usr/lib/python2.7/site-packages/nova/compute│····························· /manager.py", line 2063, in _build_and_run_instance\n bdms=block_device_mapping)\n File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_r│····························· eraise()\n File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n File "/usr/lib/python2.7/site-pack│····························· ages/nova/compute/manager.py", line 2015, in _build_and_run_instance\n block_device_mapping) as resources:\n File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__\n return │····························· self.gen.next()\n File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _build_resources\n reason=msg)\n', u'created': u'2019-01-22T23:22:37Z'} | │····························· | flavor | controller (71d79511-3802-46f6-a0bb-5ae14ac5f6a6) Expected results: Successful overcloud deployment of nodes with whole disk image Dev Suspect patch not back-ported to osp 13 https://review.openstack.org/#/c/621400/