Bug 1720134

Summary: [OSP15] FloatingIPQosTest.test_qos test fails in several test fails
Product: Red Hat OpenStack Reporter: Bernard Cafarelli <bcafarel>
Component: openstack-neutronAssignee: Slawek Kaplonski <skaplons>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: amuller, chrisw, scohen, skaplons
Target Milestone: betaKeywords: AutomationBlocker, Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-14.0.2-0.20190613150426.8c7c202.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-21 11:23:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bernard Cafarelli 2019-06-13 08:28:00 UTC 
neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest.test_qos fails on several test configurations

From log, tests seems to be going fine until some point where we get a socket exception:
2019-06-12 20:24:21,773 299289 INFO     [tempest.lib.common.ssh] ssh connection to cirros.0.235 successfully created
2019-06-12 20:24:21,934 299289 DEBUG    [neutron_tempest_plugin.scenario.test_qos] time_elapsed = 0.0180108547210693, total_bytes_read = 1048576, bytes_per_second = 58219113
[...]
2019-06-12 20:24:40,190 299289 DEBUG    [neutron_tempest_plugin.scenario.test_qos] time_elapsed = 0.0212683677673340, total_bytes_read = 1048576, bytes_per_second = 49302137
2019-06-12 20:24:41,267 299289 DEBUG    [neutron_tempest_plugin.scenario.test_qos] time_elapsed = 0.0225846767425537, total_bytes_read = 1048576, bytes_per_second = 46428647
2019-06-12 20:40:52,888 299289 ERROR    [paramiko.transport] Socket exception: No route to host (113)

Which generates this traceback:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 382, in test_qos
    sleep=1)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 68, in wait_until_true
    while not predicate():
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 380, in <lambda>
    port=self.NC_PORT),
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_qos.py", line 103, in _check_bw
    ssh_client.exec_command(cmd)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/ssh.py", line 187, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 153, in exec_command
    with transport.open_session() as channel:
  File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 806, in open_session
    timeout=timeout,
  File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 933, in open_channel
    raise e
  File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 1982, in run
    ptype, m = self.packetizer.read_message()
  File "/usr/lib/python3.6/site-packages/paramiko/packet.py", line 441, in read_message
    header = self.read_all(self.__block_size_in, check_rekey=True)
  File "/usr/lib/python3.6/site-packages/paramiko/packet.py", line 288, in read_all
    x = self.__socket.recv(n)
OSError: [Errno 113] No route to host
Comment 2 Slawek Kaplonski 2019-06-13 09:38:34 UTC 
In L3 agent logs I found something like:

2019-06-12 20:21:45.554 130606 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-84ab79e6-0ac9-40d0-8605-ea518636c537', 'tc', 'qdisc', 'show', 'dev', 'qg-8d36b0e7-1b'] execute_rootwrap_daemon /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:103
2019-06-12 20:21:45.558 130606 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['ip', 'netns', 'exec', 'qrouter-84ab79e6-0ac9-40d0-8605-ea518636c537', 'tc', 'qdisc', 'show', 'dev', 'qg-8d36b0e7-1b']: multiprocessing.managers.RemoteError: 
2019-06-12 20:21:45.559 130606 DEBUG oslo_concurrency.lockutils [-] Lock "qos-fip" released by "neutron.agent.l3.extensions.qos.fip.FipQosAgentExtension.update_router" :: held 0.026s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:339
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 84ab79e6-0ac9-40d0-8605-ea518636c537: multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
    send(msg)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
    s = self.dumps(obj)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
    return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
  File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
    return super(RpcJSONEncoder, self).default(o)
  File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ValueError' is not JSON serializable

---------------------------------------------------------------------------
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 556, in _process_router_if_compatible
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 600, in _process_updated_router
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     self.l3_ext_manager.update_router(self.context, router)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     extension.obj.update_router(context, data)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return f(*args, **kwargs)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/extensions/qos/fip.py", line 293, in update_router
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     self.process_floating_ip_addresses(context, router_info)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/extensions/qos/fip.py", line 278, in process_floating_ip_addresses
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     self._remove_fip_rate_limit(device, fip)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/extensions/qos/fip.py", line 164, in _remove_fip_rate_limit
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     tc_wrapper.clear_ip_rate_limit(direction, fip_ip)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 187, in clear_ip_rate_limit
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     qdisc_id = self._get_qdisc_id_for_filter(direction)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 45, in _get_qdisc_id_for_filter
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     qdisc_results = self._get_qdiscs().split('\n')
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 42, in _get_qdiscs
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return self._execute_tc_cmd(cmd)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/l3_tc_lib.py", line 38, in _execute_tc_cmd
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return ip_wrapper.netns.execute(cmd, run_as_root=True, **kwargs)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 794, in execute
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     run_as_root=run_as_root)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 122, in execute
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 109, in execute_rootwrap_daemon
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     LOG.error("Rootwrap error running command: %s", cmd)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     self.force_reraise()
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     raise value
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 106, in execute_rootwrap_daemon
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 154, in execute
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     res = self._run_one_command(proxy, cmd, stdin)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 139, in _run_one_command
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent multiprocessing.managers.RemoteError: 
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent Unserializable message: Traceback (most recent call last):
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     send(msg)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     s = self.dumps(obj)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     **kw).encode(obj)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     chunks = self.iterencode(o, _one_shot=True)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return _iterencode(o, 0)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     return super(RpcJSONEncoder, self).default(o)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent     o.__class__.__name__)
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent TypeError: Object of type 'ValueError' is not JSON serializable
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent 
2019-06-12 20:21:45.559 130606 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------

So it looks like yet another rootwrap issue which I will investigate now.
Comment 3 Slawek Kaplonski 2019-06-13 10:56:10 UTC 
I checked it on dev environment and after installing iprouter-tc package inside neutron_l3 container, this test is passing.
Comment 4 Slawek Kaplonski 2019-06-13 11:07:50 UTC 
I pushed patch to RDO: https://review.rdoproject.org/r/#/c/21125/ to add this missing package as neutron's dependency.
Comment 16 errata-xmlrpc 2019-09-21 11:23:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811