Bug 1713321

Summary: [OSP15] L3 agent reports "Failed sending gratuitous ARP" with "TypeError: Object of type 'ValueError' is not JSON serializable" backtrace
Product: Red Hat OpenStack Reporter: Bernard Cafarelli <bcafarel>
Component: openstack-containersAssignee: Dan Prince <dprince>
Status: CLOSED ERRATA QA Contact: Candido Campos <ccamposr>
Severity: medium Docs Contact:
Priority: medium    
Version: 15.0 (Stein)CC: amuller, chrisw, dprince, ekuris, m.andre, njohnston, scohen
Target Milestone: betaKeywords: Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-21 16:39:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bernard Cafarelli 2019-05-23 12:03:52 UTC
Spotted while running tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario test on 1 controller, 1 compute, vlxan setup

Test passes, but checking /var/log/containers/neutron/l3-agent.log I see gARP fails:
2019-05-23 09:18:11.010 103143 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 10.0.0.239 on qg-cf4b1490-bd in namespace qrouter-f62f7e76-34d0-48f5-99d8-64fd04c04b4b:
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
    send(msg)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
    s = self.dumps(obj)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
    return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
  File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
    return super(RpcJSONEncoder, self).default(o)
  File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ValueError' is not JSON serializable

Seen in log on every test run so looks reproducible

Comment 1 Slawek Kaplonski 2019-05-24 09:18:50 UTC
The problem IMO is due to missing arping binary in neutron-l3-agent container.
After installing iputils package in it, arping worked fine for me.

IMO problem may be due to how CommandFilter in Oslo.rootwrap works. See in https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/filters.py#L46 - it looks for binary (arping in our case) inside container. If there is no such binary in known paths, it returns None and that causes this failure mentioned in bz description.

Comment 3 Bernard Cafarelli 2019-06-03 10:01:12 UTC
*** Bug 1714146 has been marked as a duplicate of this bug. ***

Comment 7 Eran Kuris 2019-07-24 06:18:15 UTC
Hi Dan,
Please add Fixed In Version.

Comment 10 errata-xmlrpc 2019-09-21 16:39:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2853