Bug 2227236 - OVN DB was corrupted during scale-out procedure [NEEDINFO]
Summary: OVN DB was corrupted during scale-out procedure
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.1 (Train)
Hardware: All
OS: All
unspecified
medium
Target Milestone: ---
: ---
Assignee: Jakub Libosvar
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-28 13:17 UTC by Alex Stupnikov
Modified: 2023-08-16 14:50 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
jlibosva: needinfo? (anbs)

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-27013 0 None None None 2023-07-28 13:18:13 UTC

Description Alex Stupnikov 2023-07-28 13:17:01 UTC 
Description of problem:
Scale out procedure broken communication via FIPs and DHCP address allocation in RHOSP 16.1.9 deployment. In overcloud controller sosreports there is flood of following errors after connection to OVN DB flapped:

2023-07-25 19:54:39.506 26 ERROR ovsdbapp.event Stderr: 'ovsdb-client: failed to connect to "tcp:IP:6642" (Connection timed out)
...
2023-07-25 19:54:40.371 32 ERROR networking_ovn.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'UUID'". This should never happen in a normal situation, please check the status of your cluster: networking_ovn.common.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'UUID'". This should never happen in a normal situation, please check the status of your cluster

It looks like something happened with OVN DB, but there are no clear pointers in OVN logs. Communications were restored after running OVN DB sync tool, but it looks like there is an OVN-related problem in our scale-out workflow or some OVN bug.

sosreports from controller nodes are attached to support case.

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 16.1.9 (Train)

How reproducible:
We are not sure if there is reliable way to reproduce this problem.

Additional info:
Will be provided privately
Note You need to log in before you can comment on or make changes to this bug.