Bug 1703091

Summary: Since we're using nova_cell0 instead of nova database, nova-manage db archive_deleted_rows is no longer archiving deleted rows
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-novaAssignee: melanie witt <mwitt>
Status: CLOSED ERRATA QA Contact: Paras Babbar <pbabbar>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: amodi, chrisbro, cmedeiro, dasmith, jhakimra, jhardee, kchamart, lyarwood, mwitt, nlevinki, nwolf, sbauza, sgordon, vromanso
Target Milestone: Upstream M3Keywords: FutureFeature, Patch, Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-20.0.1-0.20191025043858.390db63.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1748990 1749087 1752711 (view as bug list) Environment:
Last Closed: 2020-02-06 14:40:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1748990, 1749087, 1752711, 1769458, 1778905    

Description David Hill 2019-04-25 13:39:53 UTC 
Description of problem:
Since we're using nova_cell0 instead of nova database, nova-manage db archive_deleted_rows is no longer archiving deleted rows

nova-manage db archive_deleted_rows --verbose --max_rows 1000000

Version-Release number of selected component (if applicable):
# rpm -qa|grep nova
python-nova-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-api-17.0.5-3.d7864fbgit.el7ost.noarch
puppet-nova-12.4.0-6.el7ost.noarch
openstack-nova-common-17.0.5-3.d7864fbgit.el7ost.noarch
python2-novaclient-10.1.0-1.el7ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Get 705k deleted instances
2. run nova-manage db archive_deleted_rows
3.

Actual results:
Nothing is archived

Expected results:
All rows should be achived up to --max_rows

Additional info:
Comment 1 David Hill 2019-04-25 13:54:45 UTC 
MariaDB [nova_cell0]> select count(*) from instances where deleted_at is not null;
+----------+
| count(*) |
+----------+
|   708405 |
+----------+
Comment 2 David Hill 2019-04-25 14:40:09 UTC 
We do have a workaround on this and it's to use --config-file nova_cell0.conf where we changed the connection string to use nova_cell0 instead of nova.   This is still a bug in my opinion if customers have a big failure rate, that will consume a lot of disk space that could be avoided with the cleanup cronjobs.
Comment 5 chrisbro@redhat.com 2019-07-03 22:13:19 UTC 
Hi Team,

Can I get an update with this as this impacting a very important customer?
Comment 6 David Hill 2019-07-04 12:55:49 UTC 
Hey Chris,
  
    I created another BZ for another issue but this helps cleaning up common tables but some of them are not cleaned up which I feel like it's another issue.
We do have a workaround which is to set the connection string to nova_cell0 instead of nova and call that configuration using nova-manage.

Thank you very much,
David Hill
Comment 8 melanie witt 2019-08-29 00:48:29 UTC 
Upstream patch has merged: https://review.opendev.org/507486
Comment 15 errata-xmlrpc 2020-02-06 14:40:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283