Description of problem: If "satellite-installer --upgrade-mongo-storage-engine" is launched while mongod is stopped, the installer fails to detect the export failed and ends up dropping the source database. Version-Release number of selected component (if applicable): foreman-installer-1.22.0.16-1.el7sat.noarch foreman-installer-katello-1.22.0.16-1.el7sat.noarch satellite-installer-6.6.0.21-1.el7sat.noarch How reproducible: Always Steps to Reproduce: 1. systemctl stop mongod 2. satellite-installer --upgrade-mongo-storage-engine # systemctl stop mongod # time satellite-installer --upgrade-mongo-storage-engine Starting disk space check for upgrade Package versions are locked. Continuing with unlock. Running Stop Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, goferd, httpd \ All services stopped [OK] -------------------------------------------------------------------------------- foreman-maintain service stop --exclude "rh-mongodb34-mongod","postgresql","tomcat","dynflowd","foreman-proxy","puppetserver" finished successfully! 2019-11-06T10:14:34.070+0100 Failed: error connecting to db server: no reachable servers mongodump --host localhost --out /var/lib/pulp/tmp/mongodb_engine_upgrade failed! Check the output for error! Running Stop Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod | All services stopped [OK] -------------------------------------------------------------------------------- foreman-maintain service stop --only rh-mongodb34-mongod finished successfully! rm -rf /var/lib/mongodb/* finished successfully! sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully! mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/lib/pulp/tmp/mongodb_engine_upgrade finished successfully! Running Start Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-mongodb34-mongod / All services started [OK] -------------------------------------------------------------------------------- foreman-maintain service start --only rh-mongodb34-mongod finished successfully! 2019-11-06T10:14:41.885+0100 the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead foreman-maintain service start --only rh-mongodb34-mongod finished successfully! 2019-11-06T10:14:41.885+0100 the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead 2019-11-06T10:14:41.885+0100 Failed: mongorestore target '/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database' invalid: stat /var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database: no such file or director y mongorestore --host localhost --db=pulp_database --drop --dir=/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error! Running Stop Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod / All services stopped [OK] -------------------------------------------------------------------------------- foreman-maintain service stop --only rh-mongodb34-mongod finished successfully! mv -f /var/lib/pulp/tmp/mongodb_engine_upgrade/mongod.conf.bak /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully! rm -rf /var/lib/mongodb/* finished successfully! 2019-11-06T10:14:47.656+0100 Failed: error connecting to db server: no reachable servers mongorestore --host localhost --db=pulp_database --drop --dir=/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error! Running Start Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-mongodb34-mongod | All services started [OK] -------------------------------------------------------------------------------- foreman-maintain service start --only rh-mongodb34-mongod finished successfully! real 0m32.729s user 0m16.383s sys 0m7.381s Expected results: The migration should check that the mongodump process is successful prior to engaging into a potentially destructive migration.
Will test this on upstream but I think this is fixed, if not will look into it more.
I was able to reproduce this, and yeah that is bad with it blowing away the db, I will work on a pr for this: foreman-maintain service stop --exclude "rh-mongodb34-mongod","postgresql","tomcat","dynflowd","foreman-proxy","puppetserver" finished successfully! [ INFO 2020-01-09T17:30:51 verbose] Starting mongodump to /var/tmp/mongodb_engine_upgrade 2020-01-09T17:30:55.214-0500 Failed: error connecting to db server: no reachable servers mongodump --host localhost --out /var/tmp/mongodb_engine_upgrade failed! Check the output for error! [ERROR 2020-01-09T17:30:55 verbose] mongodump --host localhost --out /var/tmp/mongodb_engine_upgrade failed! Check the output for error! [ INFO 2020-01-09T17:30:55 verbose] Export done, stopping MongoDB to remove old datafiles Running Stop Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod | All services stopped [OK] -------------------------------------------------------------------------------- foreman-maintain service stop --only rh-mongodb34-mongod finished successfully! [ INFO 2020-01-09T17:30:57 verbose] Removing contents from /var/lib/mongodb rm -rf /var/lib/mongodb/* finished successfully! [ INFO 2020-01-09T17:30:58 verbose] Changing config to WiredTiger and starting restore. sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully! mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/tmp/mongodb_engine_upgrade finished successfully! Running Start Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-mongodb34-mongod / All services started [OK] -------------------------------------------------------------------------------- foreman-maintain service start --only rh-mongodb34-mongod finished successfully! 2020-01-09T17:31:00.439-0500 the --db and --collection args should only be used when restoring from a BSON file. lude instead 2020-01-09T17:31:00.439-0500 Failed: mongorestore target '/var/tmp/mongodb_engine_upgrade/pulp_database' invalid rectory mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed! [ERROR 2020-01-09T17:31:00 verbose] mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_ [ERROR 2020-01-09T17:31:00 verbose] The restore could not be completed correctly, reverting actions. [ INFO 2020-01-09T17:31:00 verbose] Stopping MongoDB Running Stop Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod | All services stopped [OK] -------------------------------------------------------------------------------- foreman-maintain service stop --only rh-mongodb34-mongod finished successfully! [ INFO 2020-01-09T17:31:02 verbose] Restoring old config mv -f /var/tmp/mongodb_engine_upgrade/mongod.conf.bak /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully! [ INFO 2020-01-09T17:31:02 verbose] Removing contents in /var/lib/mongodb rm -rf /var/lib/mongodb/* finished successfully! [ INFO 2020-01-09T17:31:02 verbose] Restoring database under MMAPV1 storage engine 2020-01-09T17:31:06.357-0500 Failed: error connecting to db server: no reachable servers mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed! [ERROR 2020-01-09T17:31:06 verbose] mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_ [ INFO 2020-01-09T17:31:06 verbose] Starting MongoDB with old config and database files Running Start Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-mongodb34-mongod - All services started [OK] -------------------------------------------------------------------------------- foreman-maintain service start --only rh-mongodb34-mongod finished successfully! [ERROR 2020-01-09T17:31:08 verbose] Mongo started up in MMAPV1 mode, backup at /var/tmp/mongodb_engine_upgrade [ INFO 2020-01-09T17:31:08 verbose] Installer finished in 131.457323212 seconds
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/28712 has been resolved.
Verified on 6.7. Verfication point: # cat /etc/foreman-installer/custom-hiera.yaml .... # Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage to upgrade to WiredTiger. mongodb::server::storage_engine: 'mmapv1' # systemctl stop mongod # satellite-installer --upgrade-mongo-storage-engine Ensuring MongoDB is running before upgrade. Running Start Services ================================================================================ Check if command is run as root user: [OK] .... ... mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database finished successfully! rm -rf /var/tmp/mongodb_engine_upgrade finished successfully! sed -i -e 's/Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage.* to upgrade to WiredTiger./Do not remove/g' /etc/foreman-installer/custom-hiera.yaml finished successfully! sed -i -e 's/mmapv1/wiredTiger/g' /etc/foreman-installer/custom-hiera.yaml finished successfully! sed -i -e 's/mmapv1/wiredTiger/g' /etc/foreman-installer/custom-hiera.yaml finished successfully! Installing Done [100%] [................................................................................................................................] Success!
This was delivered with Satellite 6.7.0.