Bug 1560872 – [Netvirt] ODL L2 Agent is dead after restarting a compute node

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1560872 - [Netvirt] ODL L2 Agent is dead after restarting a compute node

Summary:	[Netvirt] ODL L2 Agent is dead after restarting a compute node

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	opendaylight (Show other bugs)
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified Unspecified

Priority	high Severity high
Target Milestone:	rc
Target Release:	13.0 (Queens)
Assigned To:	Josh Hershberg
QA Contact:	Itzik Brown
Docs Contact:

URL:
Whiteboard:	odl_netvirt
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-03-27 03:45 EDT by Itzik Brown
Modified:	2018-10-18 03:23 EDT (History)
CC List:	4 users (show)

See Also:
Fixed In Version:	opendaylight-8.0.0-11.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:	N/A
Last Closed:	2018-06-27 09:48:49 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Karaf log (2.78 MB, text/plain) 2018-03-27 06:49 EDT, Itzik Brown	no flags	Details
Karaf log with OVSDB Trace (2.32 MB, text/plain) 2018-04-11 04:01 EDT, Itzik Brown	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
OpenDaylight Bug	NETVIRT-1178	None	None	None	2018-03-27 03:55 EDT
OpenDaylight gerrit	71203	None	None	None	2018-04-23 07:09 EDT
OpenDaylight gerrit	72188	None	None	None	2018-05-24 04:13 EDT
Red Hat Product Errata	RHEA-2018:2086	None	None	None	2018-06-27 09:49 EDT

Groups: None (edit)

Description Itzik Brown 2018-03-27 03:45:33 EDT

Description of problem:
After rebooting a compute node the OVS is connected to the all the controllers but the pseudo agent is down.

In Neutron log:
2018-03-27 07:31:09.202 34 WARNING neutron.db.agents_db [req-86b57593-85d6-4c20-bba1-d408151e94ef - - - - -] Agent healthcheck: found 1 dead agents out of 11:
                Type       Last heartbeat host
              ODL L2  2018-03-27 07:05:06 compute-0.localdomain

Version-Release number of selected component (if applicable):
opendaylight-8.0.0-3.el7ost.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Itzik Brown 2018-03-27 06:49 EDT

Created attachment 1413668 [details]
Karaf log

Comment 2 Josh Hershberg 2018-04-02 09:37:12 EDT

Please add these to the karaf logging configuration and post the resultant karaf.log.

log4j2.logger.itzik.name = org.opendaylight.neutron.hostconfig.ovs.NeutronHostconfigOvsListener
log4j2.logger.itzik.level = DEBUG

Comment 4 Josh Hershberg 2018-04-11 04:00:10 EDT

Itzik and I sat on this today. What we saw was that indeed, the rebooted host is missing from /operational/neutron:neutron/hostconfigs. We also saw that the node was missing from /operational/network-topology:network-topology/ which seems to indicate that ovsdb plugin is failing to write that node to operational. This requires some additional digging.

Comment 5 Itzik Brown 2018-04-11 04:01 EDT

Created attachment 1420206 [details]
Karaf log with OVSDB Trace

Comment 6 Itzik Brown 2018-04-11 05:01:31 EDT

Restarting the OVS on the compute node - no problem
Power down the compute , waiting for 10 minutes and powering it on - no problem.

Comment 7 Josh Hershberg 2018-04-23 07:09:12 EDT

Please see the upstream bug for details on the root cause

https://jira.opendaylight.org/browse/NETVIRT-1178

Patch here: https://git.opendaylight.org/gerrit/#/c/71203/

Comment 10 Mike Kolesnik 2018-05-21 04:42:26 EDT

Moving non blocker OSP 13 bugs to z1

Comment 12 Josh Hershberg 2018-05-24 04:14:20 EDT

Attached link to patch on u/s stable/oxygen above

https://git.opendaylight.org/gerrit/#/c/72188/

Comment 17 Itzik Brown 2018-05-31 09:00:31 EDT

Checked with:
opendaylight-8.0.0-11.el7ost.noarch

Comment 19 errata-xmlrpc 2018-06-27 09:48:49 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086

Note You need to log in before you can comment on or make changes to this bug.