Bug 1809738

Summary:	[ovn-kubernetes] When a node gets deleted, the Chassis record for that node is not deleted from the sbdb.
Product:	OpenShift Container Platform	Reporter:	Aniket Bhat <anbhat>
Component:	Networking	Assignee:	Aniket Bhat <anbhat>
Networking sub component:	ovn-kubernetes	QA Contact:	Ross Brattain <rbrattai>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	bbennett
Version:	4.4
Target Milestone:	---
Target Release:	4.4.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1809739 1809747 (view as bug list)		Environment:
Last Closed:	2020-05-18 13:35:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1809747
Bug Blocks:	1809739

Description Aniket Bhat 2020-03-03 19:08:14 UTC

Description of problem:

When a node is deprovisioned/deleted from the cluster, the southbound db's chassis record for this node doesn't get deleted. This results in stale geneve tunnels and vswitchd flows on the other nodes in the cluster. At scale, this can mean thousands of tunnels and unused stale flows.

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Always

Steps to Reproduce:
1. Create a ovn cluster
2. Add a few nodes
3. Delete one node
4. Note that the tunnels corresponding to the deleted node and the ovs flows for this remote ip endpoint stay in ovs' on the other nodes.

Actual results:

Flows and tunnels corresponding to the node being deleted stay as stale entries.

Expected results:

All flows and the corresponding tunnels for the node being deleted are cleaned up when the node goes away.

Additional info:

Upstream issue: https://github.com/ovn-org/ovn-kubernetes/issues/1105

Comment from Russell Bryant:

Just some more detail ... ovn-controller will delete its associated Chassis record if it shuts down gracefully. I'm not sure that's ever the case, though. The fallback is that something else needs to do the cleanup. ovn-kubernetes is already watching Nodes, so it can add this as another thing it does when syncing Nodes or when it sees a Node get deleted.

This will require knowing which Chassis record in the ovn southbound database corresponds to a Node. ovn-kubernetes already ensures that the hostname field of the Chassis is equal to the Node name.

Comment 5 Ross Brattain 2020-05-07 02:30:46 UTC

deleted several nodes and sbdb Chassis were deleted within ~5 minutes

Verified on 4.4.0-0.nightly-2020-05-01-231319

Comment 7 errata-xmlrpc 2020-05-18 13:35:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2133