neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest.test_qos fails on several test configurations From log, tests seems to be going fine until some point where we get a socket exception: 2019-06-12 20:24:21,773 299289 INFO [tempest.lib.common.ssh] ssh connection to cirros.0.235 successfully created 2019-06-12 20:24:21,934 299289 DEBUG [neutron_tempest_plugin.scenario.test_qos] time_elapsed = 0.0180108547210693, total_bytes_read = 1048576, bytes_per_second = 58219113 [...] 2019-06-12 20:24:40,190 299289 DEBUG [neutron_tempest_plugin.scenario.test_qos] time_elapsed = 0.0212683677673340, total_bytes_read = 1048576, bytes_per_second = 49302137 2019-06-12 20:24:41,267 299289 DEBUG [neutron_tempest_plugin.scenario.test_qos] time_elapsed = 0.0225846767425537, total_bytes_read = 1048576, bytes_per_second = 46428647 2019-06-12 20:40:52,888 299289 ERROR [paramiko.transport] Socket exception: No route to host (113) Which generates this traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 382, in test_qos sleep=1) File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 68, in wait_until_true while not predicate(): File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 380, in <lambda> port=self.NC_PORT), File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_qos.py", line 103, in _check_bw ssh_client.exec_command(cmd) File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/ssh.py", line 187, in exec_command return super(Client, self).exec_command(cmd=cmd, encoding=encoding) File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 153, in exec_command with transport.open_session() as channel: File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 806, in open_session timeout=timeout, File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 933, in open_channel raise e File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 1982, in run ptype, m = self.packetizer.read_message() File "/usr/lib/python3.6/site-packages/paramiko/packet.py", line 441, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/usr/lib/python3.6/site-packages/paramiko/packet.py", line 288, in read_all x = self.__socket.recv(n) OSError: [Errno 113] No route to host
In L3 agent logs I found something like: 2019-06-12 20:21:45.554 130606 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-84ab79e6-0ac9-40d0-8605-ea518636c537', 'tc', 'qdisc', 'show', 'dev', 'qg-8d36b0e7-1b'] execute_rootwrap_daemon /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:103 2019-06-12 20:21:45.558 130606 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['ip', 'netns', 'exec', 'qrouter-84ab79e6-0ac9-40d0-8605-ea518636c537', 'tc', 'qdisc', 'show', 'dev', 'qg-8d36b0e7-1b']: multiprocessing.managers.RemoteError: 2019-06-12 20:21:45.559 130606 DEBUG oslo_concurrency.lockutils [-] Lock "qos-fip" released by "neutron.agent.l3.extensions.qos.fip.FipQosAgentExtension.update_router" :: held 0.026s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:339 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 84ab79e6-0ac9-40d0-8605-ea518636c537: multiprocessing.managers.RemoteError: --------------------------------------------------------------------------- Unserializable message: Traceback (most recent call last): File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client send(msg) File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send s = self.dumps(obj) File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8') File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default return super(RpcJSONEncoder, self).default(o) File "/usr/lib64/python3.6/json/encoder.py", line 180, in default o.__class__.__name__) TypeError: Object of type 'ValueError' is not JSON serializable --------------------------------------------------------------------------- 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 556, in _process_router_if_compatible 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent self._process_updated_router(router) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 600, in _process_updated_router 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent self.l3_ext_manager.update_router(self.context, router) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent extension.obj.update_router(context, data) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return f(*args, **kwargs) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/extensions/qos/fip.py", line 293, in update_router 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent self.process_floating_ip_addresses(context, router_info) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/extensions/qos/fip.py", line 278, in process_floating_ip_addresses 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent self._remove_fip_rate_limit(device, fip) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/extensions/qos/fip.py", line 164, in _remove_fip_rate_limit 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent tc_wrapper.clear_ip_rate_limit(direction, fip_ip) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 187, in clear_ip_rate_limit 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent qdisc_id = self._get_qdisc_id_for_filter(direction) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 45, in _get_qdisc_id_for_filter 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent qdisc_results = self._get_qdiscs().split('\n') 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 42, in _get_qdiscs 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return self._execute_tc_cmd(cmd) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 38, in _execute_tc_cmd 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return ip_wrapper.netns.execute(cmd, run_as_root=True, **kwargs) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 794, in execute 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent run_as_root=run_as_root) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 122, in execute 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent execute_rootwrap_daemon(cmd, process_input, addl_env)) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 109, in execute_rootwrap_daemon 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent LOG.error("Rootwrap error running command: %s", cmd) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent self.force_reraise() 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent raise value 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 106, in execute_rootwrap_daemon 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return client.execute(cmd, process_input) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 154, in execute 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent res = self._run_one_command(proxy, cmd, stdin) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 139, in _run_one_command 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent res = proxy.run_one_command(cmd, stdin) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "<string>", line 2, in run_one_command 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent raise convert_to_error(kind, result) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent multiprocessing.managers.RemoteError: 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent --------------------------------------------------------------------------- 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent Unserializable message: Traceback (most recent call last): 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent send(msg) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent s = self.dumps(obj) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8') 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent **kw).encode(obj) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent chunks = self.iterencode(o, _one_shot=True) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return _iterencode(o, 0) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent return super(RpcJSONEncoder, self).default(o) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.6/json/encoder.py", line 180, in default 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent o.__class__.__name__) 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent TypeError: Object of type 'ValueError' is not JSON serializable 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent 2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent --------------------------------------------------------------------------- So it looks like yet another rootwrap issue which I will investigate now.
I checked it on dev environment and after installing iprouter-tc package inside neutron_l3 container, this test is passing.
I pushed patch to RDO: https://review.rdoproject.org/r/#/c/21125/ to add this missing package as neutron's dependency.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811