Bug 2213910 - neutron randomly stops working with "error: Hash Ring returned empty when hashing"
Summary: neutron randomly stops working with "error: Hash Ring returned empty when has...
Keywords:
Status: ON_DEV
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 16.2 (Train)
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
: ---
Assignee: Lucas Alvares Gomes
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-06-09 20:55 UTC by David Hill
Modified: 2023-06-26 12:24 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
fgadkano: needinfo? (rhos-maint)
dhill: needinfo-
dhill: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 886279 0 None NEW [OVN] Hash Ring: Set nodes as offline upon exit 2023-06-19 12:24:25 UTC
Red Hat Issue Tracker OSP-25724 0 None None None 2023-06-09 20:55:45 UTC

Description David Hill 2023-06-09 20:55:17 UTC 
Description of problem:
Since updating to 16.2.5, neutron randomly stops working with "error: Hash Ring returned empty when hashing".

We find nothing in the logs explaining why ovn/ovs or neutron is breaking like that so far.

controller00:
2023-06-09 14:19:19.747 27 ERROR networking_ovn.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'7baee1cb-75c4-4275-8ba6-ee6f33b015d6'". This should never happen in a normal situation, please check the status of your cluster: networking_ovn.common.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'7baee1cb-75c4-4275-8ba6-ee6f33b015d6'". This should never happen in a normal situation, please check the status of your cluster


controller01:
2023-06-09 14:19:19.732 33 INFO neutron.wsgi [req-cc773860-ef35-4a22-805a-c3b7f350173a ca0aa87bb5d247ae8a122230c4883414 364f0ba173634eebb7108a575d1d8a9e - default default] 10.100.151.7,10.100.151.5 "GET /v2.0/ports?device_id=a958085e-a114
-4e51-b52c-e395d11641a7 HTTP/1.1" status: 200  len: 186 time: 0.0281248
2023-06-09 14:19:19.746 26 ERROR networking_ovn.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'7baee1cb-75c4-4275-8ba6-ee6f33b015d6'". This should never happen in a normal situation, please check the status of your cluster: networking_ovn.common.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'7baee1cb-75c4-4275-8ba6-ee6f33b015d6'". This should never happen in a normal situation, please check the status of your cluster

Version-Release number of selected component (if applicable):


How reproducible:
Random, 2 environments

Steps to Reproduce:
1. Random
2.
3.

Actual results:
Neutron stops creating ports 

Expected results:
Neutron should not stop doing what's it's doing

Additional info:
2 environments so far were impacted by this issue, we rebooted the hosts and service came back.
Comment 20 Ihar Hrachyshka 2023-06-15 16:03:19 UTC 
This bz will be used to track Lucas work on neutron server resilience where it comes to cleaning up hash ring entries. The rest - e.g. guarding against invalid agent_down_time value - should be tracked elsewhere. Please create a bz for this.
Note You need to log in before you can comment on or make changes to this bug.