Hide Forgot
Description of problem: After upgrading to osp13. $ openstack compute service list +----+----------------+---------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+----------------+---------------------+----------+---------+-------+----------------------------+ | 1 | nova-cert | undercloud | internal | enabled | down | 2019-06-17T21:45:59.000000 | | 2 | nova-scheduler | undercloud | internal | enabled | up | 2019-07-24T18:50:46.000000 | | 3 | nova-conductor | undercloud | internal | enabled | up | 2019-07-24T18:50:43.000000 | | 6 | nova-compute | undercloud | nova | enabled | up | 2019-07-24T18:50:45.000000 | $ openstack compute service delete 1 Failed to delete compute service with ID '1': Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-fc30d427-db1a-4496-9d51-e45643395f04) 1 of 1 compute services failed to delete. from nova-api.log 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi [req-fc30d427-db1a-4496-9d51-e45643395f04 65dd7dce219249f48a4fc92dc6173d52 502217386c5c4e60b03a6564e47d0e68 - default default] Unexpected exception in API method: ValueError: Field value nova-cert is invalid 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 260, in delete 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi self.host_api.service_delete(context, id) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4925, in service_delete 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi self._service_delete(context, service_id) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4921, in _service_delete 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi service.destroy() 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi return fn(self, *args, **kwargs) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 409, in destroy 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi self._send_notification(fields.NotificationAction.DELETE) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 399, in _send_notification 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi self), 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 164, in from_service_obj 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi return cls(host=service.host, source=source) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 159, in __init__ 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi self.source = source 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi field_value = field.coerce(self, name, value) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 195, in coerce 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi return self._type.coerce(obj, attr, value) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 317, in coerce 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi raise ValueError(msg) 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi ValueError: Field value nova-cert is invalid 2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi 2019-07-24 15:21:42.131 4071 INFO nova.api.openstack.wsgi [req-fc30d427-db1a-4496-9d51-e45643395f04 65dd7dce219249f48a4fc92dc6173d52 502217386c5c4e60b03a6564e47d0e68 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.ValueError'> Version-Release number of selected component (if applicable): OSP 13 current
The nova-cert service was removed from the codebase during the Pike release: https://blueprints.launchpad.net/nova/+spec/remove-nova-cert and in Queens, the versioned notifications code began using an enumerated value for the possible notification source services: https://github.com/openstack/nova/blob/17.0.0/nova/objects/fields.py#L797-L816 and the nova-cert service was not included as it had already been removed. Because the nova-cert service is not in the expected enumerated values, the notification object code blows up.
Note: This is related to BZ1716025 where we work to disable nova-cert on the undercloud with a minor update before doing the ffu.
Needs a KBase to document manual database manipulation to remove traces of nova-cert. SME: Martin / Mel.
A way to remove the nova-cert service in OSP13 is documented in [1]. A fix to remove the service on a minor OSP10 update is tracked in BZ1716025. Closing this with CANTFIX and the documented workaround in [1] [1] https://access.redhat.com/solutions/4260191