Bug 1732955

Summary: Unable to delete nova-cert service on OSP13
Product: Red Hat OpenStack Reporter: Matt Flusche <mflusche>
Component: openstack-novaAssignee: melanie witt <mwitt>
Status: CLOSED CANTFIX QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: dasmith, eglynn, jhakimra, kchamart, mschuppe, mwitt, pamadio, sbauza, sgordon, vromanso
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-30 06:53:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matt Flusche 2019-07-24 20:10:18 UTC 
Description of problem:

After upgrading to osp13.

$ openstack compute service list
+----+----------------+---------------------+----------+---------+-------+----------------------------+
| ID | Binary         | Host                | Zone     | Status  | State | Updated At                 |
+----+----------------+---------------------+----------+---------+-------+----------------------------+
|  1 | nova-cert      | undercloud | internal | enabled | down  | 2019-06-17T21:45:59.000000 |
|  2 | nova-scheduler | undercloud | internal | enabled | up    | 2019-07-24T18:50:46.000000 |
|  3 | nova-conductor | undercloud | internal | enabled | up    | 2019-07-24T18:50:43.000000 |
|  6 | nova-compute   | undercloud | nova     | enabled | up    | 2019-07-24T18:50:45.000000 |

$ openstack compute service delete 1
Failed to delete compute service with ID '1': Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-fc30d427-db1a-4496-9d51-e45643395f04)
1 of 1 compute services failed to delete.

from nova-api.log

2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi [req-fc30d427-db1a-4496-9d51-e45643395f04 65dd7dce219249f48a4fc92dc6173d52 502217386c5c4e60b03a6564e47d0e68 - default default] Unexpected exception in API method: ValueError: Field value nova-cert is invalid
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 260, in delete
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     self.host_api.service_delete(context, id)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4925, in service_delete
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     self._service_delete(context, service_id)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4921, in _service_delete
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     service.destroy()
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     return fn(self, *args, **kwargs)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 409, in destroy
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     self._send_notification(fields.NotificationAction.DELETE)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 399, in _send_notification
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     self),
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 164, in from_service_obj
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     return cls(host=service.host, source=source)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line 159, in __init__
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     self.source = source
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     field_value = field.coerce(self, name, value)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 195, in coerce
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     return self._type.coerce(obj, attr, value)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 317, in coerce
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi     raise ValueError(msg)
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi ValueError: Field value nova-cert is invalid
2019-07-24 15:21:41.942 4071 ERROR nova.api.openstack.wsgi 
2019-07-24 15:21:42.131 4071 INFO nova.api.openstack.wsgi [req-fc30d427-db1a-4496-9d51-e45643395f04 65dd7dce219249f48a4fc92dc6173d52 502217386c5c4e60b03a6564e47d0e68 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'>





Version-Release number of selected component (if applicable):
OSP 13 current
Comment 2 melanie witt 2019-07-26 04:08:01 UTC 
The nova-cert service was removed from the codebase during the Pike release:

https://blueprints.launchpad.net/nova/+spec/remove-nova-cert

and in Queens, the versioned notifications code began using an enumerated value for the possible notification source services:

https://github.com/openstack/nova/blob/17.0.0/nova/objects/fields.py#L797-L816

and the nova-cert service was not included as it had already been removed. Because the nova-cert service is not in the expected enumerated values, the notification object code blows up.
Comment 3 Martin Schuppert 2019-07-26 08:52:09 UTC 
Note:
This is related to BZ1716025 where we work to disable nova-cert on the undercloud
with a minor update before doing the ffu.
Comment 4 Kashyap Chamarthy 2019-07-26 14:55:33 UTC 
Needs a KBase to document manual database manipulation to remove traces of nova-cert.

SME: Martin / Mel.
Comment 8 Martin Schuppert 2019-07-30 06:53:02 UTC 
A way to remove the nova-cert service in OSP13 is documented in [1].
A fix to remove the service on a minor OSP10 update is tracked in 
BZ1716025.

Closing this with CANTFIX and the documented workaround in [1]

[1] https://access.redhat.com/solutions/4260191