Description of problem: Since upgrade to 16.1.9 they're seeing these errors on mysql logs from DB nodes. They also say the deleted instances are not being purged from the DB and rapidly increasing. The load on the master DB node is increasing more than the average from time to time. Version-Release number of selected component (if applicable): RHOSP 16.1.9 Additional info: In next updates and sosreports from all 3 DB and controller nodes are attached to the case. Also DB dumps from nova and cinder DBs.
Let's concentrate on Nova and ignore the Cinder error in this BZ (I assume you have filed, or will file, a BZ with Cinder for that). Errors in the MySQL are indeed a problem, but could we please have a description in the form of "I did X, I expected Y, I got Z" (or just fill out the BZ template when you open a new bug, it's there for a reason ;)? Based on my reading of your comment #1, it looks like you ran `nova-manage db archive_deleted_rows --max_rows 10000 --all-cells --verbose` and expected more to get archived, because there are 104485 deleted instance record in the database, is that correct?
Yes, the Cinder bug will come after this one in case is not something somehow related. Customer thinks that DB load in DB master is 4-5 times higher than usual after upgrade to 16.1.9 and also see repeating messages like the above. And another issue seems to be that their nova_api_cron (pasted above in the private comment) is not fully clearing deleted instances.