Bug 2168206

Summary:	[16.1] Nova DB archiving not working with thousands of records after update to 16.1.9
Product:	Red Hat OpenStack	Reporter:	Joaquín Veira <jveiraca>
Component:	openstack-nova	Assignee:	melanie witt <mwitt>
Status:	CLOSED WONTFIX	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	alifshit, dasmith, eglynn, gkadam, jhakimra, jmelvin, kchamart, mwitt, sapaul, sbauza, sgordon, vromanso
Target Milestone:	async	Keywords:	Patch, Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2170683 (view as bug list)		Environment:
Last Closed:	2023-08-15 18:49:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2170683
Bug Blocks:

Description Joaquín Veira 2023-02-08 13:02:33 UTC

Description of problem:
Since upgrade to 16.1.9 they're seeing these errors on mysql logs from DB nodes.
They also say the deleted instances are not being purged from the DB and rapidly increasing.
The load on the master DB node is increasing more than the average from time to time.

Version-Release number of selected component (if applicable):
RHOSP 16.1.9

Additional info:
In next updates and sosreports from all 3 DB and controller nodes are attached to the case.
Also DB dumps from nova and cinder DBs.

Comment 3 Artom Lifshitz 2023-02-08 15:00:43 UTC

Let's concentrate on Nova and ignore the Cinder error in this BZ (I assume you have filed, or will file, a BZ with Cinder for that).

Errors in the MySQL are indeed a problem, but could we please have a description in the form of "I did X, I expected Y, I got Z" (or just fill out the BZ template when you open a new bug, it's there for a reason ;)?

Based on my reading of your comment #1, it looks like you ran `nova-manage db archive_deleted_rows  --max_rows 10000  --all-cells --verbose` and expected more to get archived, because there are 104485 deleted instance record in the database, is that correct?

Comment 4 Joaquín Veira 2023-02-08 15:32:34 UTC

Yes, the Cinder bug will come after this one in case is not something somehow related.

Customer thinks that DB load in DB master is 4-5 times higher than usual after upgrade to 16.1.9 and also see repeating messages like the above.

And another issue seems to be that their nova_api_cron (pasted above in the private comment) is not fully clearing deleted instances.