Description of problem: Cannot delete a Nova VM that is in vm_state error. Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20210506.n.1 How reproducible: 100% Steps to Reproduce: 1.Create stack that creates 2 nova VMs - http://paste.openstack.org/show/792445/ 2.Try to delete the stack running the "openstack stack delete stackname" command 3.One of the nova VMs is deleted successfully, the other VM enters ERROR state. 4.Try deleting the VM by running "openstack server delete vmname" 5.VM is not deleted Commands: [2021-05-24 06:54:33] (tester) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ | 34be3d57-2063-4a08-8a13-55864757085d | srvtree-server2-ectyxcy6bunb | ACTIVE | int_net_1=2001::f816:3eff:feed:cfc9, 10.0.1.81, 10.0.0.154 | rhel80 | rhel_flavor_1ram_1vpu_10disk | | 6b751039-a995-48ba-adcf-dccd03b4853f | srvtree-server1-rh5da6maeidh | ACTIVE | int_net_1=2001::f816:3eff:fe5f:24b0, 10.0.1.79, 10.0.0.246 | rhel80 | rhel_flavor_1ram_1vpu_10disk | +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ [2021-05-24 06:54:41] (tester) [stack@undercloud-0 ~]$ openstack stack list +--------------------------------------+------------+-----------------+----------------------+--------------+ | ID | Stack Name | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+-----------------+----------------------+--------------+ | 74a73f04-c22c-47ba-8c3e-f28427f2e029 | srvtree | CREATE_COMPLETE | 2021-05-23T12:22:22Z | None | +--------------------------------------+------------+-----------------+----------------------+--------------+ [2021-05-24 06:54:49] (tester) [stack@undercloud-0 ~]$ openstack stack delete srvtree -y [2021-05-24 06:55:22] (tester) [stack@undercloud-0 ~]$ [2021-05-24 06:55:22] (tester) [stack@undercloud-0 ~]$ openstack stack list +--------------------------------------+------------+---------------+----------------------+----------------------+ | ID | Stack Name | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+---------------+----------------------+----------------------+ | 74a73f04-c22c-47ba-8c3e-f28427f2e029 | srvtree | DELETE_FAILED | 2021-05-23T12:22:22Z | 2021-05-24T10:55:01Z | +--------------------------------------+------------+---------------+----------------------+----------------------+ [2021-05-24 06:55:27] (tester) [stack@undercloud-0 ~]$ [2021-05-24 06:55:28] (tester) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ | 6b751039-a995-48ba-adcf-dccd03b4853f | srvtree-server1-rh5da6maeidh | ERROR | int_net_1=2001::f816:3eff:fe5f:24b0, 10.0.1.79, 10.0.0.246 | rhel80 | rhel_flavor_1ram_1vpu_10disk | +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ [2021-05-24 06:55:35] (tester) [stack@undercloud-0 ~]$ openstack server delete srvtree-server1-rh5da6maeidh [2021-05-24 06:55:46] (tester) [stack@undercloud-0 ~]$ [2021-05-24 06:55:47] (tester) [stack@undercloud-0 ~]$ [2021-05-24 06:55:47] (tester) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ | 6b751039-a995-48ba-adcf-dccd03b4853f | srvtree-server1-rh5da6maeidh | ERROR | int_net_1=2001::f816:3eff:fe5f:24b0, 10.0.1.79, 10.0.0.246 | rhel80 | rhel_flavor_1ram_1vpu_10disk | +--------------------------------------+------------------------------+--------+------------------------------------------------------------+--------+------------------------------+ [2021-05-24 06:58:10] (tester) [stack@undercloud-0 ~]$ openstack server show srvtree-server1-rh5da6maeidh +-----------------------------+------------------------------------------------------------------------------------+ | Field | Value | +-----------------------------+------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | 2021-05-23T12:22:56.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | int_net_1=2001::f816:3eff:fe5f:24b0, 10.0.1.79, 10.0.0.246 | | config_drive | | | created | 2021-05-23T12:22:28Z | | fault | {'code': 500, 'created': '2021-05-24T10:55:47Z', 'message': 'InternalServerError'} | | flavor | rhel_flavor_1ram_1vpu_10disk (c14d9936-418e-46d7-9470-989eff2a2be6) | | hostId | 197ce344bd8552aae2631e452db3fd7f865c27a2d25113e4a7760527 | | id | 6b751039-a995-48ba-adcf-dccd03b4853f | | image | rhel80 (cb64f9d0-1532-4100-b941-064f9dada6b0) | | key_name | None | | name | srvtree-server1-rh5da6maeidh | | project_id | a3eae79beff8488882944e96046fe3c1 | | properties | | | security_groups | name='srvtree-sec_group-zafuoakmrmtv' | | status | ERROR | | updated | 2021-05-24T10:55:47Z | | user_id | 40ac9d36050c400ca9c64cd69a7778b4 | | volumes_attached | | +-----------------------------+------------------------------------------------------------------------------------+
Can you provide the following please: $ openstack server event list srvtree-server1-rh5da6maeidh $ openstack server event show srvtree-server1-rh5da6maeidh $create-request-id $ openstack server event show srvtree-server1-rh5da6maeidh $delete-request-id Additionally nova logs referencing the above request ids will be appreciated if they are not already available.
Of course: [2021-05-24 07:48:32] (tester) [stack@undercloud-0 ~]$ openstack server event list srvtree-server1-rh5da6maeidh +------------------------------------------+--------------------------------------+--------+----------------------------+ | Request ID | Server ID | Action | Start Time | +------------------------------------------+--------------------------------------+--------+----------------------------+ | req-5e65a304-0a45-4d55-bb7c-354c02e2753a | 6b751039-a995-48ba-adcf-dccd03b4853f | delete | 2021-05-24T11:08:40.000000 | | req-d18ea26e-0a7b-43b9-847e-6022cccd9d79 | 6b751039-a995-48ba-adcf-dccd03b4853f | delete | 2021-05-24T11:03:38.000000 | | req-ab371647-322b-4bfe-8f0c-a588c355e7b3 | 6b751039-a995-48ba-adcf-dccd03b4853f | stop | 2021-05-24T11:03:26.000000 | | req-41a28e57-c8c5-49e1-b55e-851db69ea58e | 6b751039-a995-48ba-adcf-dccd03b4853f | delete | 2021-05-24T10:55:46.000000 | | req-4e5ddfab-3103-4587-9e53-fae3c8bb601f | 6b751039-a995-48ba-adcf-dccd03b4853f | delete | 2021-05-24T10:55:02.000000 | | req-66b9dab8-b05e-4447-8f92-779ff568dd61 | 6b751039-a995-48ba-adcf-dccd03b4853f | create | 2021-05-23T12:22:25.000000 | +------------------------------------------+--------------------------------------+--------+----------------------------+ CREATE request id: [2021-05-24 07:50:44] (tester) [stack@undercloud-0 ~]$ openstack server event show srvtree-server1-rh5da6maeidh req-66b9dab8-b05e-4447-8f92-779ff568dd61 +---------------+------------------------------------------+ | Field | Value | +---------------+------------------------------------------+ | action | create | | instance_uuid | 6b751039-a995-48ba-adcf-dccd03b4853f | | message | None | | project_id | a3eae79beff8488882944e96046fe3c1 | | request_id | req-66b9dab8-b05e-4447-8f92-779ff568dd61 | | start_time | 2021-05-23T12:22:25.000000 | | user_id | 40ac9d36050c400ca9c64cd69a7778b4 | +---------------+------------------------------------------+ We have several DELETE request ids, from bottom to top: [2021-05-24 07:51:16] (tester) [stack@undercloud-0 ~]$ openstack server event show srvtree-server1-rh5da6maeidh req-4e5ddfab-3103-4587-9e53-fae3c8bb601f +---------------+------------------------------------------+ | Field | Value | +---------------+------------------------------------------+ | action | delete | | instance_uuid | 6b751039-a995-48ba-adcf-dccd03b4853f | | message | Error | | project_id | a3eae79beff8488882944e96046fe3c1 | | request_id | req-4e5ddfab-3103-4587-9e53-fae3c8bb601f | | start_time | 2021-05-24T10:55:02.000000 | | user_id | 40ac9d36050c400ca9c64cd69a7778b4 | +---------------+------------------------------------------+ [2021-05-24 07:51:50] (tester) [stack@undercloud-0 ~]$ openstack server event show srvtree-server1-rh5da6maeidh req-41a28e57-c8c5-49e1-b55e-851db69ea58e +---------------+------------------------------------------+ | Field | Value | +---------------+------------------------------------------+ | action | delete | | instance_uuid | 6b751039-a995-48ba-adcf-dccd03b4853f | | message | Error | | project_id | a3eae79beff8488882944e96046fe3c1 | | request_id | req-41a28e57-c8c5-49e1-b55e-851db69ea58e | | start_time | 2021-05-24T10:55:46.000000 | | user_id | 40ac9d36050c400ca9c64cd69a7778b4 | +---------------+------------------------------------------+ [2021-05-24 07:52:01] (tester) [stack@undercloud-0 ~]$ openstack server event show srvtree-server1-rh5da6maeidh req-d18ea26e-0a7b-43b9-847e-6022cccd9d79 +---------------+------------------------------------------+ | Field | Value | +---------------+------------------------------------------+ | action | delete | | instance_uuid | 6b751039-a995-48ba-adcf-dccd03b4853f | | message | Error | | project_id | a3eae79beff8488882944e96046fe3c1 | | request_id | req-d18ea26e-0a7b-43b9-847e-6022cccd9d79 | | start_time | 2021-05-24T11:03:38.000000 | | user_id | 40ac9d36050c400ca9c64cd69a7778b4 | +---------------+------------------------------------------+ [2021-05-24 07:52:16] (tester) [stack@undercloud-0 ~]$ openstack server event show srvtree-server1-rh5da6maeidh req-5e65a304-0a45-4d55-bb7c-354c02e2753a +---------------+------------------------------------------+ | Field | Value | +---------------+------------------------------------------+ | action | delete | | instance_uuid | 6b751039-a995-48ba-adcf-dccd03b4853f | | message | Error | | project_id | a3eae79beff8488882944e96046fe3c1 | | request_id | req-5e65a304-0a45-4d55-bb7c-354c02e2753a | | start_time | 2021-05-24T11:08:40.000000 | | user_id | 40ac9d36050c400ca9c64cd69a7778b4 | +---------------+------------------------------------------+ Full SOS Report is available here: http://rhos-release.virt.bos.redhat.com/log/bz1963903 Let me know if you need anything else, and thanks for the help!
(In reply to Bruna Bonguardo from comment #2) > > Full SOS Report is available here: > http://rhos-release.virt.bos.redhat.com/log/bz1963903 > > Let me know if you need anything else, and thanks for the help! These don't appear to be full sosreports and are missing /var/log/containers etc. Can you rerun or alternatively just gather logs referring to req-66b9dab8-b05e-4447-8f92-779ff568dd61 and req-4e5ddfab-3103-4587-9e53-fae3c8bb601f from /var/log/containers/nova from the controllers and computes?
Thanks for access to the env, looking at the events as an admin I can see that the creation request was successful: $ openstack server event show 6b751039-a995-48ba-adcf-dccd03b4853f req-66b9dab8-b05e-4447-8f92-779ff568dd61 -f json -c events | awk '{gsub("\\\\n","\n")};1' { "events": [ { "event": "compute__do_build_and_run_instance", "start_time": "2021-05-23T12:22:31.000000", "finish_time": "2021-05-23T12:22:56.000000", "result": "Success", "traceback": null, "host": "compute-0.redhat.local", "hostId": "197ce344bd8552aae2631e452db3fd7f865c27a2d25113e4a7760527" } ] } The first delete attempt however fails when we try to delete the port in Neutron: $ openstack server event show 6b751039-a995-48ba-adcf-dccd03b4853f req-4e5ddfab-3103-4587-9e53-fae3c8bb601f -f json -c events | awk '{gsub("\\\\n","\n")};1' { "events": [ { "event": "compute_terminate_instance", "start_time": "2021-05-24T10:55:02.000000", "finish_time": "2021-05-24T10:55:05.000000", "result": "Error", "traceback": " File \"/usr/lib/python3.6/site-packages/nova/compute/utils.py\", line 1372, in decorated_function return function(self, context, *args, **kwargs) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 219, in decorated_function kwargs['instance'], e, sys.exc_info()) File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__ self.force_reraise() File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File \"/usr/lib/python3.6/site-packages/six.py\", line 675, in reraise raise value File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 207, in decorated_function return function(self, context, *args, **kwargs) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3049, in terminate_instance do_terminate_instance(instance, bdms) File \"/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py\", line 328, in inner return f(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3047, in do_terminate_instance self._set_instance_obj_error_state(context, instance) File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__ self.force_reraise() File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File \"/usr/lib/python3.6/site-packages/six.py\", line 675, in reraise raise value File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3037, in do_terminate_instance self._delete_instance(context, instance, bdms) File \"/usr/lib/python3.6/site-packages/nova/hooks.py\", line 154, in inner rv = f(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 2974, in _delete_instance self._shutdown_instance(context, instance, bdms) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 2869, in _shutdown_instance self._try_deallocate_network(context, instance, requested_networks) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 2786, in _try_deallocate_network self._set_instance_obj_error_state(context, instance) File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__ self.force_reraise() File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File \"/usr/lib/python3.6/site-packages/six.py\", line 675, in reraise raise value File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 2781, in _try_deallocate_network _deallocate_network_with_retries() File \"/usr/lib/python3.6/site-packages/oslo_service/loopingcall.py\", line 436, in func return evt.wait() File \"/usr/lib/python3.6/site-packages/eventlet/event.py\", line 125, in wait result = hub.switch() File \"/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py\", line 298, in switch return self.greenlet.switch() File \"/usr/lib/python3.6/site-packages/oslo_service/loopingcall.py\", line 150, in _run_loop result = func(*self.args, **self.kw) File \"/usr/lib/python3.6/site-packages/oslo_service/loopingcall.py\", line 407, in _func result = f(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 2771, in _deallocate_network_with_retries context, instance, requested_networks) File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 2002, in _deallocate_network context, instance, requested_networks=requested_networks) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 1589, in deallocate_for_instance self._delete_ports(neutron, instance, ports, raise_if_fail=True) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 1563, in _delete_ports raise exceptions[0] File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 1553, in _delete_ports neutron.delete_port(port) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 121, in wrapper ret = obj(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py\", line 812, in delete_port return self.delete(self.port_path % (port)) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 121, in wrapper ret = obj(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py\", line 350, in delete headers=headers, params=params) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 121, in wrapper ret = obj(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py\", line 331, in retry_request headers=headers, params=params) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 121, in wrapper ret = obj(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py\", line 294, in do_request self._handle_fault_response(status_code, replybody, resp) File \"/usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py\", line 121, in wrapper ret = obj(*args, **kwargs) File \"/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py\", line 269, in _handle_fault_response exception_handler_v20(status_code, error_body) File \"/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py\", line 93, in exception_handler_v20 request_ids=request_ids) ", "host": "compute-0.redhat.local", "hostId": "197ce344bd8552aae2631e452db3fd7f865c27a2d25113e4a7760527" } ] } The trace doesn't capture the output from neutronclient that I've included below: compute-0 /var/log/containers/nova/nova-compute.log 2021-05-24 10:55:05.923 7 ERROR oslo_messaging.rpc.server neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. 2021-05-24 10:55:05.923 7 ERROR oslo_messaging.rpc.server Neutron server returns request_ids: ['req-1c3c2211-e9f1-4f60-a703-cfac853794d2'] 2021-05-24 10:55:05.923 7 ERROR oslo_messaging.rpc.server On the Neutron side I see the following trace suggesting some kind of deletion race? controller-0 /var/log/containers/neutron/server.log.1 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation [req-1c3c2211-e9f1-4f60-a703-cfac853794d2 40ac9d36050c400ca9c64cd69a7778b4 a3eae79beff8488882944e96046fe3c1 - default default] DELETE failed.: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ovs_neutron`.`floatingips`, CONSTRAINT `floatingips_ibfk_1` FOREIGN KEY (`fixed_port_id`) REFERENCES `ports` (`id`))') [SQL: DELETE FROM ports WHERE ports.id = %(id)s] [parameters: {'id': '87f95eda-34f5-47de-8c92-c37e39575ca3'}] (Background on this error at: http://sqlalche.me/e/gkpj) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation cursor, statement, parameters, context 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation cursor.execute(statement, parameters) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 165, in execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation result = self._query(query) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation conn.query(q) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 860, in query 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1061, in _read_query_result 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation result.read() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1349, in read 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation first_packet = self.connection._read_packet() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1018, in _read_packet 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation packet.check_error() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 384, in check_error 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation err.raise_mysql_exception(self._data) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise errorclass(errno, errval) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ovs_neutron`.`floatingips`, CONSTRAINT `floatingips_ibfk_1` FOREIGN KEY (`fixed_port_id`) REFERENCES `ports` (`id`))') 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation The above exception was the direct cause of the following exception: 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1967, in delete_port 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation super(Ml2Plugin, self).delete_port(context, id) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/db/db_base_plugin_v2.py", line 1513, in delete_port 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.ipam.delete_port(context, id) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/db/ipam_pluggable_backend.py", line 495, in delete_port 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation port['fixed_ips']) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/db/ipam_pluggable_backend.py", line 110, in _ipam_deallocate_ips 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation "external system for %s", addresses) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/db/ipam_pluggable_backend.py", line 86, in _ipam_deallocate_ips 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation ipam_subnet = ipam_driver.get_subnet(ip['subnet_id']) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 337, in get_subnet 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return NeutronDbSubnet.load(subnet_id, self._context) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 94, in load 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation ctx, neutron_subnet_id) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 30, in load_by_neutron_subnet_id 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation context, neutron_subnet_id=neutron_subnet_id) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 640, in get_objects 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation cls, context, _pager=_pager, **cls.modify_fields_to_db(kwargs)) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 52, in get_objects 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation **(_pager.to_kwargs(context, obj_cls) if _pager else {})) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 311, in get_collection 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation for c in query 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3316, in __iter__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.session._autoflush() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1576, in _autoflush 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self.flush() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2451, in flush 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self._flush(objects) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2589, in _flush 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation transaction.rollback(_capture_exception=True) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation compat.reraise(exc_type, exc_value, exc_tb) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2549, in _flush 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation flush_context.execute() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation rec.execute(self) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 624, in execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation uow, 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 348, in delete_obj 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation delete, 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1305, in _emit_delete_statements 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation c = connection.execute(statement, del_objects) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return meth(self, multiparams, params) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation return connection._execute_clauseelement(self, multiparams, params) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation distilled_params, 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation e, statement, parameters, cursor, context 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation util.raise_from_cause(newraise, exc_info) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation reraise(type(exception), exception, tb=exc_tb, cause=cause) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise value.with_traceback(tb) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation cursor, statement, parameters, context 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation cursor.execute(statement, parameters) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 165, in execute 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation result = self._query(query) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation conn.query(q) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 860, in query 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1061, in _read_query_result 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation result.read() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1349, in read 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation first_packet = self.connection._read_packet() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1018, in _read_packet 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation packet.check_error() 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 384, in check_error 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation err.raise_mysql_exception(self._data) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation raise errorclass(errno, errval) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`ovs_neutron`.`floatingips`, CONSTRAINT `floatingips_ibfk_1` FOREIGN KEY (`fixed_port_id`) REFERENCES `ports` (`id`))') 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation [SQL: DELETE FROM ports WHERE ports.id = %(id)s] 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation [parameters: {'id': '87f95eda-34f5-47de-8c92-c37e39575ca3'}] 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation (Background on this error at: http://sqlalche.me/e/gkpj) 2021-05-24 10:55:05.495 29 ERROR neutron.pecan_wsgi.hooks.translation 2021-05-24 10:55:05.508 29 INFO neutron.wsgi [req-1c3c2211-e9f1-4f60-a703-cfac853794d2 40ac9d36050c400ca9c64cd69a7778b4 a3eae79beff8488882944e96046fe3c1 - default default] 172.17.1.100 "DELETE /v2.0/ports/87f95eda-34f5-47de-8c92-c37e39575ca3 HTTP/1.1" status: 500 len: 344 time: 0.7178788 Moving to Neutron for more review.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenStack Platform 16.1 (openstack-neutron) security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0990