1823795 – [ML2/OVS] Cannot delete routers with IPv6 subnet, sometimes crashes L3 agent

Bug 1823795 - [ML2/OVS] Cannot delete routers with IPv6 subnet, sometimes crashes L3 agent

Summary: [ML2/OVS] Cannot delete routers with IPv6 subnet, sometimes crashes L3 agent

Keywords:
Status:	CLOSED EOL
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	16.0 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Slawek Kaplonski
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1795638 1827064
TreeView+	depends on / blocked

Reported:	2020-04-14 14:11 UTC by Candido Campos
Modified:	2020-11-04 08:21 UTC (History)
CC List:	6 users (show)
Fixed In Version:	openstack-neutron-15.0.3-0.20200418020110.476bb78.el8ost
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Clones:	1827064 1827072 (view as bug list)
Environment:
Last Closed:	2020-11-04 08:21:05 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1873240	0	None	None	None	2020-04-16 10:50:10 UTC
OpenStack gerrit	720431	0	None	MERGED	Add rootwrap filter rule for radvd-kill script	2020-11-03 18:00:56 UTC

Comment 1 Slawek Kaplonski 2020-04-15 12:36:45 UTC

Today I checked logs from this job which You linked to. I'm not 100% sure if that is our known issue with systemd scripts wrapper, which I think Brent fixed some time ago or if it's new bug.
But I also noticed in l3-agent logs errors like:

2020-04-13 11:07:43.064 31530 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['radvd-kill', '9', '43476']: multiprocessing.managers.RemoteError: 
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent [-] Error while deleting router f5735a8c-ede9-47a4-b2c8-2c93fc50ac07: multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
    send(msg)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
    s = self.dumps(obj)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
    return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
  File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
    return super(RpcJSONEncoder, self).default(o)
  File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ValueError' is not JSON serializable

---------------------------------------------------------------------------
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 506, in _safe_router_removed
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self._router_removed(ri, router_id)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 542, in _router_removed
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.router_info[router_id] = ri
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.force_reraise()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise value
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 539, in _router_removed
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     ri.delete()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 470, in delete
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.disable_radvd()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 587, in disable_radvd
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.radvd.disable()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ra.py", line 195, in disable
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     pm.disable()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/external_process.py", line 113, in disable
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     utils.execute(cmd, run_as_root=self.run_as_root)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 122, in execute
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 109, in execute_rootwrap_daemon
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     LOG.error("Rootwrap error running command: %s", cmd)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.force_reraise()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise value
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 106, in execute_rootwrap_daemon
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 154, in execute
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     res = self._run_one_command(proxy, cmd, stdin)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 139, in _run_one_command
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent multiprocessing.managers.RemoteError: 
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent Unserializable message: Traceback (most recent call last):
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     send(msg)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     s = self.dumps(obj)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     **kw).encode(obj)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     chunks = self.iterencode(o, _one_shot=True)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return _iterencode(o, 0)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return super(RpcJSONEncoder, self).default(o)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     o.__class__.__name__)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent TypeError: Object of type 'ValueError' is not JSON serializable
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent 
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------


Which seems like something new for me. Do You have env with such issues that I can use to investigate that? Or should I prepare one for my own?

Note You need to log in before you can comment on or make changes to this bug.