Bug 1770552 - mongo storage upgrade to wiredtiger fails and drops database if mongod is stopped
Summary: mongo storage upgrade to wiredtiger fails and drops database if mongod is sto...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installation
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: Unspecified
Assignee: Chris Roberts
QA Contact: Devendra Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-10 13:20 UTC by Francisco Garcia
Modified: 2020-05-01 13:05 UTC (History)
3 users (show)

Fixed In Version: foreman-installer-1.24.1.3-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-01 13:05:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 28712 0 Normal Closed mongo storage upgrade to wiredtiger fails and drops database if mongod is stopped 2020-04-28 12:24:44 UTC
Github theforeman foreman-installer pull 450 0 None closed Fixes #28712 - Verify MongoDB is on before upgrade 2020-04-28 12:24:43 UTC

Description Francisco Garcia 2019-11-10 13:20:35 UTC
Description of problem:

If "satellite-installer --upgrade-mongo-storage-engine" is launched while mongod is stopped, the installer fails to detect the export failed and ends up dropping the source database.


Version-Release number of selected component (if applicable):

foreman-installer-1.22.0.16-1.el7sat.noarch
foreman-installer-katello-1.22.0.16-1.el7sat.noarch
satellite-installer-6.6.0.21-1.el7sat.noarch


How reproducible:

Always 


Steps to Reproduce:
1. systemctl stop mongod
2. satellite-installer --upgrade-mongo-storage-engine

# systemctl stop mongod
# time satellite-installer --upgrade-mongo-storage-engine
Starting disk space check for upgrade
Package versions are locked. Continuing with unlock.
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, goferd, httpd
\ All services stopped                                                [OK]      
--------------------------------------------------------------------------------

foreman-maintain service stop --exclude "rh-mongodb34-mongod","postgresql","tomcat","dynflowd","foreman-proxy","puppetserver" finished successfully!
2019-11-06T10:14:34.070+0100    Failed: error connecting to db server: no reachable servers
mongodump --host localhost --out /var/lib/pulp/tmp/mongodb_engine_upgrade failed! Check the output for error!
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
| All services stopped                                                [OK]      
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
rm -rf /var/lib/mongodb/* finished successfully!
sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/lib/pulp/tmp/mongodb_engine_upgrade finished successfully!
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
/ All services started                                                [OK]      
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
2019-11-06T10:14:41.885+0100    the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
2019-11-06T10:14:41.885+0100    the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2019-11-06T10:14:41.885+0100    Failed: mongorestore target '/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database' invalid: stat /var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database: no such file or director
y
mongorestore --host localhost --db=pulp_database --drop --dir=/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
/ All services stopped                                                [OK]      
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
mv -f /var/lib/pulp/tmp/mongodb_engine_upgrade/mongod.conf.bak /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
rm -rf /var/lib/mongodb/* finished successfully!
2019-11-06T10:14:47.656+0100    Failed: error connecting to db server: no reachable servers
mongorestore --host localhost --db=pulp_database --drop --dir=/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
| All services started                                                [OK]      
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!

real    0m32.729s
user    0m16.383s
sys     0m7.381s



Expected results:

The migration should check that the mongodump process is successful prior to engaging into a potentially destructive migration.

Comment 3 Chris Roberts 2019-11-12 16:04:41 UTC
Will test this on upstream but I think this is fixed, if not will look into it more.

Comment 4 Chris Roberts 2020-01-09 22:34:18 UTC
I was able to reproduce this, and yeah that is bad with it blowing away the db, I will work on a pr for this:

foreman-maintain service stop --exclude "rh-mongodb34-mongod","postgresql","tomcat","dynflowd","foreman-proxy","puppetserver" finished successfully!
[ INFO 2020-01-09T17:30:51 verbose] Starting mongodump to /var/tmp/mongodb_engine_upgrade
2020-01-09T17:30:55.214-0500    Failed: error connecting to db server: no reachable servers
mongodump --host localhost --out /var/tmp/mongodb_engine_upgrade failed! Check the output for error!
[ERROR 2020-01-09T17:30:55 verbose] mongodump --host localhost --out /var/tmp/mongodb_engine_upgrade failed! Check the output for error!
[ INFO 2020-01-09T17:30:55 verbose] Export done, stopping MongoDB to remove old datafiles
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
| All services stopped                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
[ INFO 2020-01-09T17:30:57 verbose] Removing contents from /var/lib/mongodb
rm -rf /var/lib/mongodb/* finished successfully!
[ INFO 2020-01-09T17:30:58 verbose] Changing config to WiredTiger and starting restore.
sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/tmp/mongodb_engine_upgrade finished successfully!
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
/ All services started                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
2020-01-09T17:31:00.439-0500    the --db and --collection args should only be used when restoring from a BSON file.
lude instead
2020-01-09T17:31:00.439-0500    Failed: mongorestore target '/var/tmp/mongodb_engine_upgrade/pulp_database' invalid
rectory
mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed!
[ERROR 2020-01-09T17:31:00 verbose] mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_
[ERROR 2020-01-09T17:31:00 verbose] The restore could not be completed correctly, reverting actions.
[ INFO 2020-01-09T17:31:00 verbose] Stopping MongoDB
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
| All services stopped                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
[ INFO 2020-01-09T17:31:02 verbose] Restoring old config
mv -f /var/tmp/mongodb_engine_upgrade/mongod.conf.bak /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
[ INFO 2020-01-09T17:31:02 verbose] Removing contents in /var/lib/mongodb
rm -rf /var/lib/mongodb/* finished successfully!
[ INFO 2020-01-09T17:31:02 verbose] Restoring database under MMAPV1 storage engine
2020-01-09T17:31:06.357-0500    Failed: error connecting to db server: no reachable servers
mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed!
[ERROR 2020-01-09T17:31:06 verbose] mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_
[ INFO 2020-01-09T17:31:06 verbose] Starting MongoDB with old config and database files
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
- All services started                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
[ERROR 2020-01-09T17:31:08 verbose] Mongo started up in MMAPV1 mode, backup at /var/tmp/mongodb_engine_upgrade
[ INFO 2020-01-09T17:31:08 verbose] Installer finished in 131.457323212 seconds

Comment 5 Bryan Kearney 2020-01-15 17:06:07 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/28712 has been resolved.

Comment 12 Devendra Singh 2020-04-30 17:35:17 UTC
Verified on 6.7.

Verfication point:

# cat /etc/foreman-installer/custom-hiera.yaml
....
# Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage to upgrade to WiredTiger.
mongodb::server::storage_engine: 'mmapv1'

# systemctl stop mongod

# satellite-installer --upgrade-mongo-storage-engine
Ensuring MongoDB is running before upgrade.
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
....
...
mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database finished successfully!
rm -rf /var/tmp/mongodb_engine_upgrade finished successfully!
sed -i -e 's/Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage.* to upgrade to WiredTiger./Do not remove/g' /etc/foreman-installer/custom-hiera.yaml finished successfully!
sed -i -e 's/mmapv1/wiredTiger/g' /etc/foreman-installer/custom-hiera.yaml finished successfully!
sed -i -e 's/mmapv1/wiredTiger/g' /etc/foreman-installer/custom-hiera.yaml finished successfully!
Installing             Done                                               [100%] [................................................................................................................................]
  Success!

Comment 13 Bryan Kearney 2020-05-01 13:05:25 UTC
This was delivered with Satellite 6.7.0.


Note You need to log in before you can comment on or make changes to this bug.