Bug 1809747

Summary:	[ovn-kubernetes] When a node gets deleted, the Chassis record for that node is not deleted from the sbdb.
Product:	OpenShift Container Platform	Reporter:	Aniket Bhat <anbhat>
Component:	Networking	Assignee:	Aniket Bhat <anbhat>
Networking sub component:	ovn-kubernetes	QA Contact:	Ross Brattain <rbrattai>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	zzhao
Version:	4.4
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: On node deletion, the chassis record for the node would not get removed from the south-bound database Consequence: Stale chassis records will result in a lot of stale logical flows for that chassis which will never get removed. Fix: Added a node sync mechanism in ovnkube-master to purge chassis records of deleted nodes. Result: There are no more stale chassis records and therefore stale logical flows corresponding to deleted nodes in the south-bound database anymore.	Story Points:	---
Clone Of:	1809738	Environment:
Last Closed:	2020-07-13 17:17:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1809738

Description Aniket Bhat 2020-03-03 19:33:49 UTC

+++ This bug was initially created as a clone of Bug #1809738 +++

Description of problem:

When a node is deprovisioned/deleted from the cluster, the southbound db's chassis record for this node doesn't get deleted. This results in stale geneve tunnels and vswitchd flows on the other nodes in the cluster. At scale, this can mean thousands of tunnels and unused stale flows.

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Always

Steps to Reproduce:
1. Create a ovn cluster
2. Add a few nodes
3. Delete one node
4. Note that the tunnels corresponding to the deleted node and the ovs flows for this remote ip endpoint stay in ovs' on the other nodes.

Actual results:

Flows and tunnels corresponding to the node being deleted stay as stale entries.

Expected results:

All flows and the corresponding tunnels for the node being deleted are cleaned up when the node goes away.

Additional info:

Upstream issue: https://github.com/ovn-org/ovn-kubernetes/issues/1105

Comment from Russell Bryant:

Just some more detail ... ovn-controller will delete its associated Chassis record if it shuts down gracefully. I'm not sure that's ever the case, though. The fallback is that something else needs to do the cleanup. ovn-kubernetes is already watching Nodes, so it can add this as another thing it does when syncing Nodes or when it sees a Node get deleted.

This will require knowing which Chassis record in the ovn southbound database corresponds to a Node. ovn-kubernetes already ensures that the hostname field of the Chassis is equal to the Node name.

Comment 4 Ross Brattain 2020-05-07 02:15:58 UTC

delete node and sbdb Chassis was removed after ~5 minutes.

Verified on 4.5.0-0.nightly-2020-05-05-205255

Comment 6 errata-xmlrpc 2020-07-13 17:17:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409